Project:Quality Assurance/Automagic dependencies

This guide aim to describe the problem of "automagic dependencies", describing the reason why they are problematic, and how to handle them in most common cases.

Introduction

What are automagic dependencies?

So-called automagic dependencies are shallow dependencies of a software recognized at build or runtime and changes the way the software works. The name automagic is a pun referred to the use of GNU autotools (which produces most of the cases of automagic dependencies).

Software usually has two kinds of dependencies: mandatory dependencies and optional dependencies:

1. The first kind of dependencies are needed to use the software (that might be a library or a program), and cannot be missing in the system while building or running the program (depending whether they are build or runtime dependencies).
2. Optional dependencies are the ones that can be disabled, usually at buildtime (but sometimes at runtime, too).

Optional dependencies are usually up to the user (or the builder) to enable or disable, the classic example is brought by the --enable-foo or --with-bar options at ./configure call (those parameters are used to enable dependencies that are off by default, but there are cases when the dependencies are on by default so you have --disable-foo and --without-bar).

But with build systems that detect what is present in the system building a package, sometimes dependencies are automagic . This means that the build system doesn't give the builder a way to decide to enable usage of a dependency and they just enable it when they find it. This is the wrong behavior.

Why automagic dependencies are wrong

In the case of binary-based distributions, like RPM or DEB based ones, automagic dependencies do not change anything: if the user has something installed and is building by hand, it's usually what they want to enable, while if it's the maintainer, they just have to add a dependency on the packages required to run the binaries created.

It's different in source-based distributions like Gentoo Linux (and variants). As source-based distributions don't detach user and development packages, the build systems might find more libraries than the user required, and will try to greedily link to everything detected. This is a major cause of (dynamically-linked) binaries breaking after a depclean.

To simplify, when an automagic dependency is not stated as mandatory in an ebuild, but rather has a flag that just adds or remove the dependencies on a given package without a build system toggle, then if this package is present in the system building the software with automagic dependencies, then later it's removed, the software will break, requiring a revdep-rebuild run to fix the linking. It's also possible that an user really doesn't want some dependency enabled because they know it's likely to break occasionally, or because he's going to create a binary package for another machine where the dependency might not be present (or might not work at all).

Even without binary packages, possibly broken libraries (which a user may try to avoid), or inconvenience, there's also the issue of determinism. Users want to avoid having different behavior on different systems and developers want to avoid confusing bug reports where something behaved differently purely because of a package which was coincidentally installed. This is why build-time automagic dependencies are also problematic (even e.g. doc generation if a tool is installed) because it not only introduces variance into package's installed files, but it may lead to confusing errors which aren't easy to understand.

Fixing automagic dependencies

When a package has automagic dependencies, there are only two things that can be done:

1. the first is to state the dependency as mandatory, no matter what the users put in their USE variable, but that might mean that some support that people don't want is always enable and its dependencies pulled in;
2. the other is to fix the build system to be able to disable the dependency at build time even if it's present on the system

Most of the time upstream developers don't consider the need for disabling automagic dependencies as they don't see them as problems: they do have all of them installed, or the ones they need, and they usually build with all of them.

Luckily, most upstream developers don't mind adding options to disable them if patches are provided (sometimes also without patches, but patches are always received better). This isn't always the case, rarely. For example Wine's upstream don't want to add support for enabling or disabling features in ./configure call as they want the software to always use as many options as possible.

Autotools

Most of automagic dependencies, like the name suggests, are due to (bad) use of GNU Autotools (autoconf to be exact). There are two main cases where automagic dependencies are brought in:

1. the first is the "lazy devs" case, where the dependencies does not have a ./configure parameter at all, they are just checked with AC_CHECK_LIB or the pkg-config macro PKG_CHECK_MODULES , that allows to run specific code when a library (or a package) is present or not;
2. the other case is the "silly argument" case, where a --disable-foo or --without-bar parameter is actually accepted by ./configure , but it's not checked correctly.

The first case is actually simple to fix, it's just matter of adding a AC_ARG_WITH (or AC_ARG_ENABLE ) call and then check for the corresponding variable before doing the test. It's useful to know that the first parameter passed to the above macro actually names a variable that gets loaded by autoconf without having to add the extra parameters for action to execute when the parameter is present and when it's not, the variable is named $enable_foo or $with_bar , depending on which of the two macros are called.

Note
For the patches to be accepted by upstream, it's usually suggested not to change the default behavior, when ./configure is called without parameters; for this reason, here will be listed two methods to make non-automagic the dependencies, one for enabled-by-default deps and one for disabled-by-default deps.

CODE Adding an enabled-by-default check for an optional dependency

AC_ARG_WITH([foo], AS_HELP_STRING([--without-foo], [Build without foo library (default: test)]))

AS_IF([test "x$with_foo" != "xno"], [
  PKG_CHECK_MODULES([FOO], [foo >= 0.1])
])

CODE Adding a disabled-by-default check for an optional dependency

AC_ARG_WITH([foo], AS_HELP_STRING([--with-foo], [Build with foo library (default: disabled)]))

AS_IF([test "x$with_foo" = "xyes"], [
  PKG_CHECK_MODULES([FOO], [foo >= 0.1])
])

When the parameter is present but it's not honored, it might be simple or complex to fix the dependency:

1. it might just be a test that's not properly written, so it has to be changed in something alike to the tests above, or
2. it might be a complete screw-up in calls of AC_ARG_WITH macros. In those cases, it's better to check the code carefully and contact upstream if a screw-up seems likely.

Warning
Often (almost always when a package is using automake) automagic dependencies are coupled with AM_CONDITIONAL calls. It's very important that those calls are put outside the if/fi block, or ./configure call will bomb out.

While it is possible to work around the issue of automagic dependencies without patching configure.ac , by messing with the autoconf cache values, this method is not recommended. This does not fix the original issue, and cannot be sent upstream for integration in new versions, and can actually conflict when tests are slightly different between environments.

Meson

All dependency lookups in meson support the required: keyword argument and default to true. Automagic dependencies occur when a meson-based project specifies false instead. Fixing this is both simple and easy, while simultaneously satisfying upstream desires to probe for support in non-distro build environments:

1. Add an option to meson_options.txt: option('foo', type: 'feature')
2. Rewrite feature autodetection from dependency('mydep', required: false) to dependency('mydep', required: get_option('foo'))

The default value of a feature option is auto, but the builder can set it to enabled (error if not found) or disabled (do not search for it, pretend it is not found regardless).

CMake

Automagic dependencies may occur in CMake-based build systems where PKG_CHECK_MODULES is called without the REQUIRED parameter unconditionally. Fixing this "properly" is quite complicated, but disabling autodetection entirely is much easier, as it only involves introducing an option to the build system and executing PKG_CHECK_MODULES, depending on its value.

CODE Adding ENABLE_FOO option to avoid an automagic dependency

OPTION(ENABLE_FOO "Enable foo library" ON)
...
IF (ENABLE_FOO)
  PKG_CHECK_MODULES (FOO foo>=0.1)
ENDIF (ENABLE_FOO)
...
IF (ENABLE_FOO)
  IF (FOO_FOUND)
  ...
  ELSE (FOO_FOUND)
  ...
  ENDIF (FOO_FOUND)
ENDIF (ENABLE_FOO)

Note
Set the default in OPTION according to the original behavior.

Other build systems

Important
Please expand this guide ;) Notes about other non-custom build systems such as scons are welcome, if the build system has ways to define automagic dependencies, it should have a way to fix them, too.

Automagic dependencies can be created also with custom build systems that are used by some software. Unfortunately, being custom, those build systems are usually difficult to tweak, and there's no way to describe a general approach to fix them.

---

This page is based on a document formerly found on our main website gentoo.org.
The following people contributed to the original document: Diego Elio Pettenò, Serkan Kaba
They are listed here because wiki history does not allow for any external attribution. If you edit the wiki article, please do not add yourself here; your contributions are recorded on each article's associated history page.