Add QA check for unresolved soname dependencies (bug 704320)

Example output for maven-bin-3.6.2 (bug 704618):

 * QA Notice: Unresolved soname dependencies:
 *
 * 	/usr/share/maven-bin-3.6/lib/jansi-native/freebsd32/libjansi.so: libc.so.7 libutil.so.9
 * 	/usr/share/maven-bin-3.6/lib/jansi-native/freebsd64/libjansi.so: libc.so.7 libutil.so.9
 *

This warning comes up when a library or executable has one or
more soname dependencies (found in its NEEDED.ELF.2 metadata)
that could not be resolved by usual means. If you run ldd
on files like these then it will report a "not found" error
for each unresolved soname dependency. In order to correct
problems with soname dependency resolution, use one or more
of the approaches described in the following sections.

Content of the NEEDED.ELF.2 metadata file may be useful
for debugging purposes. Find the NEEDED.ELF.2 file in the
${D}/../build-info/ directory after the ebuild src_install
phase completes, or in the /var/db/pkg/*/*/ directory for an
installed package. Each line of the NEEDED.ELF.2 file contains
semicolon separated values for a single ELF file. The soname
dependencies are found in the DT_NEEDED column:

 E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category

External dependencies

For packages that install pre-built binaries, it may be possible
to resolve soname dependencies simply by adding dependencies
for one or more other packages that are known to provide the
needed sonames.

Removal of unecessary files

For packages that install pre-built binaries, it may be possible
to resolve soname dependencies simply by removing unnecessary
files which have unresolved soname dependencies. For example,
some pre-built binary packages include binaries intended for
irrelevant architectures or operating systems, and these files
can simply be removed because they are unnecessary.

Addition of DT_RUNPATH entries

If the relevant dependencies are installed in a location that
is not included in the dynamic linker search path, then it's
necessary for files to include a DT_RUNPATH entry which refers
to the appropriate directory. The special $ORIGIN value can
be used to create a relative path reference in DT_RUNPATH,
where $ORIGIN is a placeholder for the directory where the
file having the DT_RUNPATH entry is located.

For pre-built binaries, it may be necessary to fix up
DT_RUNPATH using patchelf --set-rpath. For example, use
patchelf --set-rpath '$ORIGIN' if a given binary should link
to libraries found in the same directory as the binary itself,
or use patchelf --set-rpath '$ORIGIN/libs' if a given binary
should link to libraries found in a subdirectory named libs
found in the same directory as the binary itself.

For binaries built from source, a flag like
-Wl,-rpath,/path/of/directory/containing/libs will create
binaries with the desired DT_RUNPATH entry.

Addition of DT_SONAME settings

If a package installs dynamic libraries which do not
set DT_SONAME, then this can lead to unresolved soname
dependencies. For dynamic libraries built from source, a flag
like -Wl,-soname=foo.so.1 will create a DT_SONAME setting. For
pre-built dynamic libraries, it may be necessary to fix up
DT_SONAME using patchelf --set-soname.

Adjustment to Portage soname resolution logic

It may be necessary to adjust Portage soname resolution logic
in order to account for special circumstances. For example,
Portage soname resolution tolerates missing DT_SONAME for
dynamic libraries that a package installs in a directory that
its binaries reference via DT_RUNPATH. This behavior is useful
for packages that have internal dynamic libraries stored in
a private directory. An example is ebtables, as discussed in
bug 646190.

Bug: https://bugs.gentoo.org/704320
Signed-off-by: Zac Medico <zmedico@gentoo.org>

4 files changed, 256 insertions, 15 deletions

diff --git a/doc/qa.docbook b/doc/qa.docbook
index 28ff6cf8e..fd5279b41 100644
--- a/doc/qa.docbook
+++ b/doc/qa.docbook
@@ -127,6 +127,104 @@
   </para>
  </sect1>
 
+ <sect1 id='qa-unresolved-soname-dependencies'>
+  <title>Unresolved soname dependencies</title>
+  <para>
+   <programlisting>
+    QA Notice: Unresolved soname dependencies
+   </programlisting>
+  </para>
+  <para>
+   This warning comes up when a library or executable has one or more
+   soname dependencies (found in its NEEDED.ELF.2 metadata) that could
+   not be resolved by usual means. If you run <command>ldd</command> on
+   files like these then it will report a "not found" error for each
+   unresolved soname dependency. In order to correct problems with
+   soname dependency resolution, use one or more of the approaches
+   described in the following sections.
+   </para>
+   <para>
+   Content of the NEEDED.ELF.2 metadata file may be useful for
+   debugging purposes. Find the NEEDED.ELF.2 file in the
+   ${D}/../build-info/ directory after the ebuild src_install phase
+   completes, or in the /var/db/pkg/*/*/ directory for an installed
+   package. Each line of the NEEDED.ELF.2 file contains semicolon
+   separated values for a single ELF file. The soname dependencies are
+   found in the DT_NEEDED column:
+   <programlisting>
+   E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category
+  </programlisting>
+  </para>
+  <sect2 id='qa-unresolved-soname-dependencies-resolved-bu-external-dependencies'>
+   <title>External dependencies</title>
+   <para>
+   For packages that install pre-built binaries, it may be possible to
+   resolve soname dependencies simply by adding dependencies for one
+   or more other packages that are known to provide the needed sonames.
+   </para>
+  </sect2>
+  <sect2 id='qa-unresolved-soname-dependencies-resolved-by-removal-of-unecessary-files'>
+   <title>Removal of unecessary files</title>
+   <para>
+   For packages that install pre-built binaries, it may be possible to
+   resolve soname dependencies simply by removing unnecessary files
+   which have unresolved soname dependencies. For example, some pre-built
+   binary packages include binaries intended for irrelevant architectures
+   or operating systems, and these files can simply be removed because
+   they are unnecessary.
+   </para>
+  </sect2>
+  <sect2 id='qa-unresolved-soname-dependencies-resolved-by-addition-of-dt-runpath-entries'>
+   <title>Addition of DT_RUNPATH entries</title>
+   <para>
+   If the relevant dependencies are installed in a location that is not
+   included in the dynamic linker search path, then it's necessary for
+   files to include a DT_RUNPATH entry which refers to the appropriate
+   directory. The special $ORIGIN value can be used to create a relative
+   path reference in DT_RUNPATH, where $ORIGIN is a placeholder for the
+   directory where the file having the DT_RUNPATH entry is located.
+   </para>
+   <para>
+   For pre-built binaries, it may be necessary to fix up DT_RUNPATH using
+   <command>patchelf --set-rpath</command>. For example, use
+   <command>patchelf --set-rpath '$ORIGIN'</command> if a given binary
+   should link to libraries found in the same directory as the binary
+   itself, or use <command>patchelf --set-rpath '$ORIGIN/libs'</command>
+   if a given binary should link to libraries found in a subdirectory
+   named libs found in the same directory as the binary itself.
+   </para>
+   <para>
+   For binaries built from source, a flag like
+   <option>-Wl,-rpath,/path/of/directory/containing/libs</option> will
+   create binaries with the desired DT_RUNPATH entry.
+   </para>
+  </sect2>
+  <sect2 id='qa-unresolved-soname-dependencies-resolved-by-addition-of-dt-soname-settings'>
+   <title>Addition of DT_SONAME settings</title>
+   <para>
+   If a package installs dynamic libraries which do not set DT_SONAME,
+   then this can lead to unresolved soname dependencies.
+   For dynamic libraries built from source, a flag like
+   <option>-Wl,-soname=foo.so.1</option> will create a DT_SONAME setting.
+   For pre-built dynamic libraries, it may be necessary to fix up
+   DT_SONAME using <command>patchelf --set-soname</command>.
+   </para>
+  </sect2>
+  <sect2 id='qa-unresolved-soname-dependencies-resolved-by-adjustment-to-portage-soname-resolution-logic'>
+   <title>Adjustment to Portage soname resolution logic</title>
+   <para>
+   It may be necessary to adjust Portage soname resolution logic in
+   order to account for special circumstances. For example, Portage
+   soname resolution tolerates missing DT_SONAME for dynamic libraries
+   that a package installs in a directory that its binaries reference
+   via DT_RUNPATH. This behavior is useful for packages that have
+   internal dynamic libraries stored in a private directory. An example
+   is ebtables, as discussed in
+   <ulink url="https://bugs.gentoo.org/646190">bug 646190</ulink>.
+   </para>
+  </sect2>
+ </sect1>
+
  <sect1 id='qa-abs-lib-link'>
   <title>Absolute Symlink In Library Directory</title>
   <para>
diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index 50e3dd1f4..828e6e126 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -1,6 +1,8 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
+from __future__ import unicode_literals
+
 import functools
 import gzip
 import io
@@ -14,10 +16,17 @@ from _emerge.EbuildProcess import EbuildProcess
 from _emerge.CompositeTask import CompositeTask
 from _emerge.PackagePhase import PackagePhase
 from _emerge.TaskSequence import TaskSequence
+from portage.package.ebuild._ipc.QueryCommand import QueryCommand
+from portage.util._dyn_libs.soname_deps_qa import (
+	_get_all_provides,
+	_get_unresolved_soname_deps,
+)
 from portage.package.ebuild.prepare_build_dirs import (_prepare_workdir,
 		_prepare_fake_distdir, _prepare_fake_filesdir)
+from portage.util.futures.compat_coroutine import coroutine
 from portage.util import writemsg
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
+from portage.util.futures.executor.fork import ForkExecutor
 
 try:
 	from portage.xml.metadata import MetaDataXML
@@ -281,7 +290,7 @@ class EbuildPhase(CompositeTask):
 				fd, logfile = tempfile.mkstemp()
 				os.close(fd)
 			post_phase = _PostPhaseCommands(background=self.background,
-				commands=post_phase_cmds, fd_pipes=self.fd_pipes,
+				commands=post_phase_cmds, elog=self._elog, fd_pipes=self.fd_pipes,
 				logfile=logfile, phase=self.phase, scheduler=self.scheduler,
 				settings=settings)
 			self._start_task(post_phase, self._post_phase_exit)
@@ -315,13 +324,6 @@ class EbuildPhase(CompositeTask):
 			self._die_hooks()
 			return
 
-		if self.phase == "install":
-			out = io.StringIO()
-			_post_src_install_soname_symlinks(self.settings, out)
-			msg = out.getvalue()
-			if msg:
-				self.scheduler.output(msg, log_path=log_path)
-
 		self._current_task = None
 		self.wait()
 		return
@@ -414,7 +416,7 @@ class EbuildPhase(CompositeTask):
 
 class _PostPhaseCommands(CompositeTask):
 
-	__slots__ = ("commands", "fd_pipes", "logfile", "phase", "settings")
+	__slots__ = ("commands", "elog", "fd_pipes", "logfile", "phase", "settings")
 
 	def _start(self):
 		if isinstance(self.commands, list):
@@ -436,4 +438,43 @@ class _PostPhaseCommands(CompositeTask):
 				logfile=self.logfile, phase=self.phase,
 				scheduler=self.scheduler, settings=self.settings, **kwargs))
 
-		self._start_task(tasks, self._default_final_exit)
+		self._start_task(tasks, self._commands_exit)
+
+	def _commands_exit(self, task):
+
+		if self._default_exit(task) != os.EX_OK:
+			self._async_wait()
+			return
+
+		if self.phase == 'install':
+			out = io.StringIO()
+			_post_src_install_soname_symlinks(self.settings, out)
+			msg = out.getvalue()
+			if msg:
+				self.scheduler.output(msg, log_path=self.settings.get("PORTAGE_LOG_FILE"))
+
+			# This operates on REQUIRES metadata generated by the above function call.
+			future = self._soname_deps_qa()
+			# If an unexpected exception occurs, then this will raise it.
+			future.add_done_callback(lambda future: future.result())
+			self._start_task(AsyncTaskFuture(future=future), self._default_final_exit)
+		else:
+			self._default_final_exit(task)
+
+	@coroutine
+	def _soname_deps_qa(self):
+
+		vardb = QueryCommand.get_db()[self.settings['EROOT']]['vartree'].dbapi
+
+		all_provides = (yield self.scheduler.run_in_executor(ForkExecutor(loop=self.scheduler), _get_all_provides, vardb))
+
+		unresolved = _get_unresolved_soname_deps(os.path.join(self.settings['PORTAGE_BUILDDIR'], 'build-info'), all_provides)
+
+		if unresolved:
+			unresolved.sort()
+			qa_msg = ["QA Notice: Unresolved soname dependencies:"]
+			qa_msg.append("")
+			qa_msg.extend("\t%s: %s" % (filename, " ".join(sorted(soname_deps)))
+				for filename, soname_deps in unresolved)
+			qa_msg.append("")
+			self.elog("eqawarn", qa_msg)
diff --git a/lib/portage/dep/soname/SonameAtom.py b/lib/portage/dep/soname/SonameAtom.py
index a7dad973d..f32082568 100644
--- a/lib/portage/dep/soname/SonameAtom.py
+++ b/lib/portage/dep/soname/SonameAtom.py
@@ -1,4 +1,4 @@
-# Copyright 2015 Gentoo Foundation
+# Copyright 2015-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -10,7 +10,7 @@ from portage import _encodings, _unicode_encode
 class SonameAtom(object):
 
 	__slots__ = ("multilib_category", "soname", "_hash_key",
-		"_hash_value")
+		"_hash_value", "_immutable")
 
 	# Distiguishes package atoms from other atom types
 	package = False
@@ -21,10 +21,14 @@ class SonameAtom(object):
 		object.__setattr__(self, "_hash_key",
 			(multilib_category, soname))
 		object.__setattr__(self, "_hash_value", hash(self._hash_key))
+		object.__setattr__(self, "_immutable", True)
 
 	def __setattr__(self, name, value):
-		raise AttributeError("SonameAtom instances are immutable",
-			self.__class__, name, value)
+		if getattr(self, '_immutable', False):
+			raise AttributeError("SonameAtom instances are immutable",
+				self.__class__, name, value)
+		# This is needed for unpickling.
+		object.__setattr__(self, name, value)
 
 	def __hash__(self):
 		return self._hash_value
diff --git a/lib/portage/util/_dyn_libs/soname_deps_qa.py b/lib/portage/util/_dyn_libs/soname_deps_qa.py
new file mode 100644
index 000000000..6840bb602
--- /dev/null
+++ b/lib/portage/util/_dyn_libs/soname_deps_qa.py
@@ -0,0 +1,98 @@
+# Copyright 2020 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+import io
+
+from portage import (
+	_encodings,
+	_unicode_encode,
+	os,
+)
+from portage.dep.soname.parse import parse_soname_deps
+from portage.util._dyn_libs.NeededEntry import NeededEntry
+
+
+def _get_all_provides(vardb):
+	"""
+	Get all of the sonames provided by all of the installed packages.
+	This does not bother to acquire a lock, since its pretty safe to
+	assume that any packages merged or unmerged while this function
+	is running must be irrelevant.
+
+	@param vardb: an installed package database
+	@type vardb: vardbapi
+	@rtype: frozenset
+	@return: a frozenset od SonameAtom instances provided by all
+		installed packages
+	"""
+
+	all_provides = []
+
+	for cpv in vardb.cpv_all():
+		try:
+			provides, = vardb.aux_get(cpv, ['PROVIDES'])
+		except KeyError:
+			# Since we don't hold a lock, assume this is due to a
+			# concurrent unmerge, and PROVIDES from the unmerged package
+			# are most likely negligible due to topologically sorted
+			# merge order. Also, note that it's possible for aux_get
+			# to succeed and return empty PROVIDES metadata if the file
+			# disappears (due to unmerge) before it can be read.
+			pass
+		else:
+			if provides:
+				all_provides.extend(parse_soname_deps(provides))
+
+	return frozenset(all_provides)
+
+
+def _get_unresolved_soname_deps(metadata_dir, all_provides):
+	"""
+	Get files with unresolved soname dependencies.
+
+	@param metadata_dir: directory containing package metadata files
+		named REQUIRES and NEEDED.ELF.2
+	@type metadata_dir: str
+	@param all_provides: a frozenset on SonameAtom instances provided by
+		all installed packages
+	@type all_provides: frozenset
+	@rtype: list
+	@return: list of tuple(filename, tuple(unresolved sonames))
+	"""
+	try:
+		with io.open(_unicode_encode(os.path.join(metadata_dir, 'REQUIRES'),
+			encoding=_encodings['fs'], errors='strict'),
+			mode='rt', encoding=_encodings['repo.content'], errors='strict') as f:
+			requires = frozenset(parse_soname_deps(f.read()))
+	except EnvironmentError:
+		return []
+
+	unresolved_by_category = {}
+	for atom in requires:
+		if atom not in all_provides:
+			unresolved_by_category.setdefault(atom.multilib_category, set()).add(atom.soname)
+
+	needed_filename = os.path.join(metadata_dir, "NEEDED.ELF.2")
+	with io.open(_unicode_encode(needed_filename, encoding=_encodings['fs'], errors='strict'),
+		mode='rt', encoding=_encodings['repo.content'], errors='strict') as f:
+		needed = f.readlines()
+
+	unresolved_by_file = []
+	for l in needed:
+		l = l.rstrip("\n")
+		if not l:
+			continue
+		entry = NeededEntry.parse(needed_filename, l)
+		missing = unresolved_by_category.get(entry.multilib_category)
+		if not missing:
+			continue
+		# NOTE: This can contain some false positives in the case of
+		# missing DT_RPATH settings, since it's possible that a subset
+		# package files have the desired DT_RPATH settings. However,
+		# since reported sonames are unresolved for at least some file(s),
+		# false positives or this sort should not be not too annoying.
+		missing = [soname for soname in entry.needed if soname in missing]
+		if missing:
+			unresolved_by_file.append((entry.filename, tuple(missing)))
+
+	return unresolved_by_file