Project:Perl/Dot-In-INC-Removal

From Gentoo Wiki
Jump to:navigation Jump to:search

Perl 5.26 makes some significant changes to the Perl Environment, which is expected to affect many users, still has (few) bugs open for various packages, and is expected to need care from Gentoo Maintainers and end users alike.

Background

Perl Module Loading

Perl has 4 main mechanisms for loading code from external libraries:

  1. use Module
  2. require Module
  3. require "Module.pm"
  4. do "Module.pm"

All of the above syntax imply looking in an array called @INC, which contains a list of paths to search for the module/file in question to load, in a similar mechanism as how calling commands from the bash shell traverses the contents of the $PATH environment variable searching for an applicable binary.

Prior to Perl 5.26, @INC had a default last entry of ., which behaved similar to how appending the string :. to $PATH might work: it allowed Perl to assume that, if the library in question was not available from any of the official installed paths, that the developer who wrote the code might intend to have their code in ./, so it falls back to check there also.

This technique was very useful, and was widespread in places where assumptions could be safely made about where the command was being invoked from.

For instance, it was incredibly common to have a project laid out like such: 

 /
 /Makefile.PL
 /inc/Module/Install.pm

Where Makefile.PL contained: 

 use inc::Module::Install;

And due to an established standard of what $CWD would be when calling Makefile.PL it mostly just worked.

It was also incredibly common to have a layout like 

 /t/
 /t/foo.t
 /t/lib/Bar.pm

Where foo.t contained: 

 use t::lib::Bar

And this was also reliable.

Similarly, lots of private projects likely utilize this convenience for similar reasons, and it is especially useful for any tool that wants to load extensions/configuration from Perl code by assuming a working directory.

The problem with do

Unfortunately, the documentation for do here wasn't entirely clear, and lead the interpretation that 

  do "File.pm"

Was logically equivalent to 

  eval qx[cat File.pm];

Wherein it only gave the impression that was the case due to the presence of . in @INC.

Subsequently, a substantial number of people used do $file under the assumption it had the same semantics as open my $fh, '<', $file, where the path can always assumed to be relative to $CWD

Motivation for change: CVE-2016-1238

04 April 2016, Debian Report a Security issue with perl, which became eventually known as CVE-2016-1238 Perl RT Ticketbug #589680

Naturally, assuming . in @INC has its downsides: Especially when the user executing the code is in a $CWD that is not under their control, ( or was under an attackers control at some previous time ).

/tmp is subsequently a great example of a location where . falls under an attackers control, and so any code that chdir's to /tmp as part of its build process, and then later proceeds to load modules, becomes an arbitrary-code execution attack surface.

And this attack surface is amplified by the presence of "Optional Dependencies", as an optional dependency is not required for the application to function normally ( exhibiting no failures in a non-attack location ), but can still be loaded in an attack location.

Pre-5.26 Mitigations for CVE

As this security issue is multi-layered, and naturally, affected code will be still vulnerable older Perls that lack language-level mitigation, the first line of defence was for all vulnerable uses of this feature to change their code to avoid the vulnerable condition.

End user applications and scripts that anticipate these kinds of problems being possible ( even if unlikely ) were encouraged to be fortified against this, by explicitly dropping . from @INC early in the applications life cycle.

Perl 5.22.3 and Perl 5.24.1 shipped fixes of this sort of nature in a few of its modules and bin/ scripts to avoid loading any dependencies that could not be conceivably expected to load from somewhere other than a proper Perl5 Install Path, but the language level mechanism was preserved in-tact, and the breakage fallout was effectively nil.

5.26 CVE Mitigation

Perl 5.26 introduced new changes to the long established mechanics:

  • Perl has a new configure tunable default_inc_excludes_dot, which when enabled (the default), omits the tailing . from the default @INC
  • Perl has a new environment variable PERL_USE_UNSAFE_INC that becomes active under -Ddefault_inc_excludes_dot

This means in practice with the stock configuration, that . is no longer in @INC, and any code that previously tried to load from . will start to fail, and become fatal conditions for use/require( unless trapped by eval)

do, due to its default mechanism of not failing upon module load, has special mechanisms added to induce warnings indicating that it would normally find a file, but no longer does.

Not Security, Just Improved Defaults

It should be stated at this point that these CVE Mitigations aren't really security solutions, they make specific classes of security defects harder to fall into by accident under default configuration, but by no means make Perl significantly more secure, and you should pay little attention to the word "safe" used in PERL_USE_UNSAFE_INC.

  1. It is still entirely possible for any Perl code to add literally any path into @INC
  2. Literally every path in @INC ( including those added by other code ) can be subject to the 3rd party unauthorized privileged arbitrary execution problem.
  3. . is by no means universally insecure, and as long as the person who controls . in entirety is the same person who runs the script in question, then there is no security threat.

For instance, if somebody were to do: 

  use lib "/tmp";

You would have the same security vulnerability, albeit more nefarious and not requiring anyone to chdir. And then a would-be attacker can place custom versions of modules use'd in "/tmp" and execute arbitrary code.

This is because the real vulnerability is the ability to load executable code from arbitrary paths, and if you have any of those paths under the control of an attacker, then they may take precedence over the intended ones.

More over, if the administrator of your computer is careless, and leaves any of the stock Perl @INC paths such that they are writeable by anyone ( e.g: attacker in the group and the directories/files are g+w, or the directories/files are o+w ), or, were at any time writeable by the same threat audience, then the same security threats apply.

Consequences

Unfortunately, a vast number of packages relied on the old mechanic, and without failure counter-balances, amount to a failure rate of between 10% and 30% of all Perl Software, both in the wild, and on CPAN.

It is expected this problem is much worse in private codebases, where there is no safety net of the CPAN Testers, and no way for dedicated individuals to sit down and find all the affected private code prior to an affected Perl being released. And the CPAN Testers infrastructure also doesn't test critical opensource tools like Autoconf and Automake, both of which have bugs now related to newer Perl releases.

The Majority of the failures seen on CPAN appear in the ./Makefile.PL or ./Build.PL paths, and in t/*.t as mentioned above, with the majority of Makefile.PL failures being due to the use of Module::Install.

Unfortunately, one of the leading users of Module::Install is ADAMK, who has many prolific and heavily used modules, of which, many are shipped in Gentoo. But ADAMK hasn't been active in recent years, leaving those modules in a state of desperate need of maintainership.

Counter Balance

As a result of these significant problems, Perl upstream has instituted a workaround to limit the fallout, by pushing that CPAN installers like CPAN.pm and cpanm set PERL_USE_UNSAFE_INC=1 as follows, to restore the legacy . long enough to get the package to install. 

 $ENV{PERL_USE_UNSAFE_INC} = 1 unless exists $ENV{PERL_USE_UNSAFE_INC}

This approach is also employed in Test::Harness, so that even outside a CPAN installer, performing make test will retain legacy environment semantics by default.

Hazards of Counter-Balance

Unfortunately, this counter-balance, while avoiding bugs in Makefile.PL and tests, also means that the runtime code that the tests target, is also regressed into legacy behaviour.

Which can mean in practice, that setting PERL_USE_UNSAFE_INC=1 will make tests pass, and then give a false sense of security, only to reveal broken code when used at runtime, where the workaround ceases to be active.

Maintainers have also frequently overlooked this mechanic, and tested their fixes for . problems, only to find they've published code which is in fact, still broken.

Workarounds

Fixing the problem at its source

The ideal solution is to fix the problem properly, and this typically means changing your use/require/do invocations.

Ideally, these changes should be made upstream, and Gentoo will replicate them once fixed. But depending on time, Gentoo maintainers may start applying some of these fixes themselves.

Convert implied relative paths to explicit paths

Where the required source is specified as a path already, the best approach is to use an explicit relative or absolute path, ie: one that starts with either a ./ or /, as follows: 

 do "./Foo.pm"
 do "/absolute/path/to/Foo.pm"
 require "./Foo.pm"
 require "/absolute/path/to/Foo.pm"

Such paths are special cased in both do and require to be loaded only from the given location, without traversing @INC at all. By taking advantage of that, you can avoid tweaking @INC, which minimises the risk of hiding other real errors.

Use a custom directory for your resources

In tests, and in some Makefile/Build.PL cases, it can make sense to relocate assets you know you need into a custom location. For instance, if you previously used: 

 use t::Foo;

It might be more sensible to create t/testlib, relocate t/Foo.pm to t/testlib/Foo.pm, and rename the packages inside from t::Foo to Foo, and then change the load code into: 

 use lib "t/testlib"; # add t/testlib to @INC
 use Foo;             # load Foo from t/testlib via @INC

Restore . from Perl ( Module::Install )

This option should be considered as a last choice, because reinserting . in either configuration scripts, or tests, is likely to mask real issues in the runtime code, and adding this code in any modules runtime source is likely to restore the vulnerability problem, so it should only really be used when desperate.

However, for some things, ( like Module::Install ) there's no real alternative, due to the code making strong assumptions and dependence on the code layout and load method. 

  use lib '.';               # restore '.' so inc::Module::Install loads from ./inc/Module/Install.pm
  use inc::Module::Install;

If you find its possible for you to temporarily restore . to load a single dependency and its children, you *might* get away with doing: 

   BEGIN {
      local @INC = @INC;
      unshift @INC, '.';
      require Foo;
   }

Which will reset @INC after the scope ends. But this comes with a caveat, in that you must know that none of the things you require are expected to augment @INC themselves, or those augments will also be lost at the end of the scope.

Applying Patches to Gentoo Packages

All Gentoo Perl Packages in dev-perl/ and similar support gentoo user-patches.

So if you can formulate one of the above fixes yourself, you can have it applied at install time by saving the relative fix in: 

 /etc/portage/patches/${CATEGORY}/${PN}-${PV}/0001-your-name-here.patch

And all entries in ${PN}-${PV} will be applied in lexicographic order during src_prepare

However, there may be plenty of packages affected by this problem outside the Gentoo Perl direct ecosystem, which may not yet have these provisions, and you may need to temporarily maintain your own variation of the ebuild, including: putting the relevant file in files/ and somehow getting it applied by manually tweaking your copies src_prepare function.

But this technique is probably too technical for most people, even though it gives the most resilient result.

Restoring . temporarily from ENV on a per-package basis

This may be slightly more convenient than fixing the package itself for many users, at the price of potentially installing broken packages.

  1. Create a file in /etc/portage/env.d/ called perl-dot-in-inc
  2. Put the following in it: 
    PERL_USE_UNSAFE_INC=1
  3. Create a file like /etc/portage/package.env/perl-5.26
  4. For each affected package, add a line like: 
    =dev-perl/foo-1.0-r1  perl-dot-in-inc

This will ensure the work-around is only applied for packages that expressly need it, so you can keep tabs on the failures and report them.

At build time, portage will read the environment file, and tell Perl to restore ., and that part of the build will work the same as it did on 5.24

Restoring . temporarily from ENV for all packages

This is the least desirable option from the standpoint of security/reliability, but the easiest.

Simply add the line: 

 PERL_USE_UNSAFE_INC=1

to /etc/portage/make.conf

And then everything that does anything with Perl will see . in @INC like it did on 5.24

Restoring . in Perl itself

This is suggested as an absolute last option for end users who are stuck in a situation where the number of failures pertaining to . in Gentoo Packages is too high and they have custom code and infrastructure outside Portage that relies on the old mechanic, and they have no time to fix the real problems, ... or they don't have the time to properly assess their production software for this issue, and want to avoid the risk of accidental failures.

This solution should be considered temporary at best as this option is likely to evaporate upstream in the near future, and this option globally reintroduces the security risk stated above.

  1. Create a file in /etc/portage/env.d/ called perl-5.26-always-dot
  2. Put the following in it: 
    EXTRA_ECONF="-Udefault_inc_excludes_dot"
  3. Create a file like /etc/portage/package.env/perl-5.26-always-dot
  4. Add a line like: 
    dev-lang/perl  perl-5.26-always-dot

And this will ensure that EXTRA_ECONF is set in the environment for every version of perl that gets built, and this tells perl configure to "unset" the default_inc_excludes_dot option, which means . will always be in @INC, just like it was in Perl 5.24.0

Links

  1. https://bugs.gentoo.org/showdependencytree.cgi?id=613764&hide_resolved=0 - Tree of bugs for Perl 5.26
  2. https://metacpan.org/pod/release/XSAWYERX/perl-5.26.0-RC2/pod/perldelta.pod#Removal-of-the-current-directory-%28%22.%22%29-from-@INC - Official upstream documentation on . in @INC with a lot more writing targeted at people using vanilla installations and upstream authors.
  3. https://rt.perl.org/Ticket/Display.html?id=127834#txn-1394453 - CVE 2016-1238 Perl Bug
Retrieved from "https://wiki.gentoo.org/index.php?title=Project:Perl/Dot-In-INC-Removal&oldid=758620"