Get Gentoo!
gentoo.org sites
gentoo.org Wiki Bugs Forums Packages
Planet Archives Sources
Infra Status
aboutsummaryrefslogtreecommitdiff
path: root/lib/_emerge
Commit message (Collapse)AuthorAgeFilesLines
* Revert "Support PORTAGE_LOG_FILTER_FILE (bug 709746)"Zac Medico2020-04-073-55/+15
| | | | | | | This reverts commit 73f72f526a66b9953a46868cc1390fde2820997f. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Revert "_BinpkgFetcherProcess: fix async_lock event loop recursion (bug 711178)"Zac Medico2020-04-071-17/+15
| | | | | | | This reverts commit 1681309f252a4e91d7256b895a9af26ef82a9b30. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Revert "SpawnProcess: cancel _main_task in _unregister (bug 711174)"Zac Medico2020-04-071-2/+0
| | | | | | | This reverts commit 97587a47fcd00d070c081ad3933e4036cf0f8e81. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Revert "Rename PORTAGE_LOG_FILTER_FILE_CMD from PORTAGE_LOG_FILTER_FILE"Zac Medico2020-04-072-2/+2
| | | | | | | This reverts commit e24859eaa03ec86e10d842296f5570dd98bed4b7. Bug: https://bugs.gentoo.org/716636 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* emerge --info: Filter variables for credentialsSam James (sam_c)2020-03-221-0/+4
| | | | | | | Bug: https://bugs.gentoo.org/713726 Closes: https://github.com/gentoo/portage/pull/536 Signed-off-by: Sam James (sam_c) <sam@cmpct.info> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: in _async_wait, call self.wait() immediatelyZac Medico2020-03-221-6/+6
| | | | | | | | | | In _async_wait, call self.wait() immediately so that the stack trace for an InvalidStateError will contain useful debugging information. Since commit 46903f3e5622, exit listeners are invoked via call_soon, so it's no longer unecessary for _async_wait to invoke self.wait() via call_soon. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: include self in InvalidStateError messageZac Medico2020-03-222-2/+2
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* depgraph: respect --deep=<depth> with --update (bug 712298)Zac Medico2020-03-141-2/+6
| | | | | | | | | | | Fix the _wrapped_select_pkg_highest_available_imp method to select an installed package when appropriate for the current --deep=<depth> setting, even with --update enabled. This prevents violation of the current --deep=<depth> setting in cases where an installed package which satisfies a dependency is masked for any reason. Bug: https://bugs.gentoo.org/712298 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SequentialTaskQueue: cancel unstarted tasks when appropriate (bug 711322)Zac Medico2020-03-072-0/+7
| | | | | | | | | | | | When the clear method is called, cancel any tasks which have not started yet, in order to ensure that their start/exit listeners are called. This fixes a case where emerge would hang after SIGINT. Also fix the CompositeTask _cancel method to react appropriately to the cancel event when the task has not started yet. Bug: https://bugs.gentoo.org/711322 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: handle addStartListener after exit (bug 711322)Zac Medico2020-03-071-0/+8
| | | | | | | | | When addStartListener is called after the task has already exited with a returncode, immediately schedule the listener to be invoked via call_soon. This behavior is similar to the Future add_done_callback method (and addExitListener since commit 5d476c4e5002). Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: simplify isAlive (bug 711688)Zac Medico2020-03-074-14/+1
| | | | | | | | | | | | | | Simplify all AsynchronousTask subclasses to use the default isAlive implementation, which returns True if self.returncode is not None. This fixes cases where the method would erroneously return False, leading to issues like bug 711688, where the CompositeTask isAlive implementation returned False for a BinpkgPrefetcher instance that was still in the process of starting via its async_start method. Fixes: d66e9ec0b105 ("AsynchronousTask: add coroutine async_start method") Bug: https://bugs.gentoo.org/711688 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: handle addExistListener after exitZac Medico2020-03-061-0/+2
| | | | | | | | | When addExistListener is called after the task has already exited with a returncode, immediately schedule the listener to be invoked via call_soon. This behavior is similar to the Future add_done_callback method. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Scheduler: replace add_done_callback with addExitListenerZac Medico2020-03-051-16/+11
| | | | | | | | | | | For simplicity, use addExitListener instead of add_done_callback, since addExitListener has been fixed to use call_soon in commit 46903f3e5622. Note that each addExitListener call occurs *after* a call to the SequentialTaskQueue add method, since the SequentialTaskQueue needs to be notified of task exit *first* (see commit 459b3535baa4). Bug: https://bugs.gentoo.org/711322 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: schedule exit listeners via call_soon (bug 711322)Zac Medico2020-03-051-32/+21
| | | | | | | | | | | Schedule exit listeners via call_soon, in order to avoid callback races like the SequentialTaskQueue exit listener race that triggered bug 711322. Callbacks scheduled via call_soon are placed in a fifo queue, ensuring that they execute in an order that is unsurprising relative to other callbacks. Bug: https://bugs.gentoo.org/711322 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: schedule start listeners via call_soon (bug 711322)Zac Medico2020-03-051-1/+1
| | | | | | | | | | | Schedule start listeners via call_soon, in order to avoid callback races like the SequentialTaskQueue exit listener race that triggered bug 711322. Callbacks scheduled via call_soon are placed in a fifo queue, ensuring that they execute in an order that is unsurprising relative to other callbacks. Bug: https://bugs.gentoo.org/711322 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Scheduler: use add_done_callback (bug 711322)Zac Medico2020-03-051-14/+22
| | | | | | | | | | | | | Use add_done_callback instead of addExistListener, in order to avoid callback races like the SequentialTaskQueue exit listener race that triggered bug 711322. The addExistListener method is prone to races because its listeners are executed in quick succession. In contrast, callbacks scheduled via add_done_callback are placed in a fifo queue, ensuring that they execute in an order that is unsurprising relative to other callbacks. Bug: https://bugs.gentoo.org/711322 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SequentialTaskQueue: update bool(self) sooner (bug 711322)Zac Medico2020-03-041-7/+9
| | | | | | | | | | | | | | | | Use addExitListener to add a _task_exit callback that will be invoked as soon as the task exits (before the future's done callback is called). This is required in order for bool(self) to have an updated value for Scheduler._schedule to base assumptions upon. Delayed updates to bool(self) is what caused Scheduler to hang as in bug 711322. This reverts changes in SequentialTaskQueue task queue exit listener behavior from commit c7e52d046621, so that only the changes necessary to support async_start remain. Fixes: c7e52d046621 ("EbuildPhase: add _async_start coroutine") Bug: https://bugs.gentoo.org/711322 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Rename PORTAGE_LOG_FILTER_FILE_CMD from PORTAGE_LOG_FILTER_FILEZac Medico2020-03-022-2/+2
| | | | | | Suggested-by: Michael 'veremitz' Everitt <gentoo@veremit.xyz> Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* EbuildMetadataPhase: make _unregister idempotent (bug 711362)Zac Medico2020-03-021-1/+2
| | | | | | | | | | Make the _unregister method check if self._files is None, as necessary for idempotency. This will prevent the AttributeError reported in bug 711362, which was triggered by recent changes in callback scheduling order. Closes: https://bugs.gentoo.org/711362 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Subprocess: delay unregister in _async_wait (bug 711174)Zac Medico2020-03-011-1/+10
| | | | | | | | | | | | | Since the super class AbstractPollTask _async_wait method calls _unregister, it can trigger premature _unregister before pid status is available. Therefore, only call the super class _async_wait method after pid status is available. This should help prevent premature _unregister events that trigger reading of build logs before they're closed as in bug 658806 and bug 711174. Bug: https://bugs.gentoo.org/711174 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _GeneratorTask: throw CancelledError in cancelled coroutine (bug 711174)Zac Medico2020-03-011-3/+9
| | | | | | | | | | | | | | | | Throw asyncio.CancelledError in a cancelled coroutine, ensuring that the coroutine can handle this exception in order to perform any necessary cleanup (like close the log file for bug 711174). Note that the asyncio.CancelledError will only be thrown in the coroutine if there's an opportunity (yield) before the generator raises StopIteration. Also fix the AsynchronousTask exit listener handling for compatibility with this new behavior. Fixes: 8074127bbc21 ("SpawnProcess: add _main coroutine") Bug: https://bugs.gentoo.org/711174 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: cancel _main_task in _unregister (bug 711174)Zac Medico2020-03-011-0/+2
| | | | | | | | | | | | | | | | Cancel _main_task in _unregister, in order to ensure that the build log is properly closed. Note that is analogous to the fix for bug 658806 in commit a9a62e57194c, but behavior related to coroutine cancellation now plays a role since the PipeLogger is now cancelled indirectly via _main_task. We still need to verify that coroutine cancellation will raise an appropriate exception in the coroutine (asyncio.CancelledError, GeneratorExit, or StopIteration) and also that the specific coroutine will react appropriately to the specific exception that will be raised. Fixes: 8074127bbc21 ("SpawnProcess: add _main coroutine") Bug: https://bugs.gentoo.org/711174 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _BinpkgFetcherProcess: fix async_lock event loop recursion (bug 711178)Zac Medico2020-03-011-15/+17
| | | | | | | | | Make the async_lock method use the AsynchronousLock async_start method in order to avoid event loop recursion. Fixes: 5c40c3e7e ("SpawnProcess: use async_start method (bug 709746)") Bug: https://bugs.gentoo.org/711178 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* fetch: drop privileges early for NFS root_squash (bug 601252)Zac Medico2020-02-291-1/+11
| | | | | | | | | Drop privileges prior to fetch function calls, so that all necessary operations can succeed when DISTDIR is on NFS with root_squash enabled. Bug: https://bugs.gentoo.org/601252 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Support PORTAGE_LOG_FILTER_FILE (bug 709746)Zac Medico2020-02-293-15/+55
| | | | | | | | | | | | | | This variable specifies a command that filters build log output to a log file. The plan is to extend this to support a separate filter for tty output in the future. In order to enable the EbuildPhase class to write elog messages to the build log with PORTAGE_LOG_FILTER_FILE support, convert its _elog method to a coroutine, and add a SchedulerInterface async_output method for it to use. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _PostPhaseCommands: avoid CancelledError in _soname_deps_qa done callbackZac Medico2020-02-291-1/+1
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* EbuildBuildDir: cancel current tasks for CancelledErrorZac Medico2020-02-291-16/+30
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* CompositeTask: call self.cancel() for _start_task CancelledErrorZac Medico2020-02-291-1/+1
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: cancel pipe_logger after async_start CancelledErrorZac Medico2020-02-291-1/+6
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: make _main cancel pipe_logger when appropriateZac Medico2020-02-291-2/+7
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: only cancel self._main_task if it's not doneZac Medico2020-02-291-1/+1
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: add _main coroutineZac Medico2020-02-291-11/+22
| | | | | | | Add a _main coroutine method and make the PipeLogger instance a local variable. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: use async_start method (bug 709746)Zac Medico2020-02-236-12/+40
| | | | | | | | | | | | | | Convert SpawnProcess to use the async_start method, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Also update AbstractEbuildProcess, AsynchronousLock, BinpkgExtractorAsync, BinpkgFetcher, EbuildFetcher, IterCompletedTestCase, AsyncFunction, and FileDigester for compatibility with SpawnProcess async_start. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* ForkExecutor: use async_start methodZac Medico2020-02-231-2/+13
| | | | | | | | Also, fix AsynchronousTask.async_start to handle cancellation of the _async_start coroutine, ensuring that start and exit listeners are notified in this case (otherwise RetryForkExecutorTestCase will hang). Signed-off-by: Zac Medico <zmedico@gentoo.org>
* EbuildBuildDir: use async_start methodZac Medico2020-02-231-78/+44
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* action_sync: display error message with --quietZac Medico2020-02-211-0/+3
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: Call _start_hook after _start_asyncZac Medico2020-02-171-2/+2
| | | | | | | | | The start listeners need to run after _start_async has completed in case they rely on any state that _start_async has created (same goes for _start). Fixes: d66e9ec0b105 ("AsynchronousTask: add coroutine async_start method") Signed-off-by: Zac Medico <zmedico@gentoo.org>
* EbuildPhase: add _async_start coroutineZac Medico2020-02-172-14/+27
| | | | | | | | | | | | Convert the _start method to an _async_start coroutine, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Also convert SequentialTaskQueue to use the async_start method, which is now required in order to start EbuildPhase instances. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AbstractEbuildProcess: add _async_start coroutineZac Medico2020-02-172-27/+14
| | | | | | | | | Convert the _start method to an _async_start coroutine, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: add coroutine async_start methodZac Medico2020-02-173-9/+37
| | | | | | | | | | | Add coroutine async_start coroutine method which calls an _async_start template method. Eventually, subclasses having _start implementations that need to write to a build log will be required to implement an _async_start coroutine method to replace the _start method as discussed in bug 709746. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _expand_set_args: un-reverse nested set traversalZac Medico2020-02-141-4/+2
| | | | | | | Un-reverse traversal order for @profile @selected @system. Bug: https://bugs.gentoo.org/649622 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* depclean: ensure consistency with update actions (bug 649622)Zac Medico2020-02-142-16/+38
| | | | | | | | | | | | | | Make depclean traverse dependencies in the same order as update actions, in order to ensure consistency in decisions which are dependent on the order of dependency evaluation due to inconsistent use of || preferences in different packages. In unit tests, update test_virtual_w3m_realistic to assert that the order of graph traversal is deterministic and consistent between update and removal (depclean) actions. Bug: https://bugs.gentoo.org/649622 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* depgraph: sort nested package set names for consistent resultsZac Medico2020-02-111-2/+4
| | | | | | | | | Traverse the unordered set returned from PackageSet.getNonAtoms() in sorted order, for more consistent results. Use a reverse sort since we pop these sets from a stack. Bug: https://bugs.gentoo.org/649622 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* depgraph: sort package set atoms for consistent resultsZac Medico2020-02-091-3/+3
| | | | | | | | Traverse the unordered set returned from PackageSet.getAtoms() in sorted order, for more consistent results. Bug: https://bugs.gentoo.org/649622 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Support FEATURES=qa-unresolved-soname-deps (bug 708448)Zac Medico2020-02-081-5/+8
| | | | | | | | | | | | Support FEATURES=qa-unresolved-soname-deps so that it's possible to disable the QA warning that was introduced for bug 704320. This is useful for systems that may have incomplete PROVIDES metadata due to alternative package managers or experimentation with a new architecture. Bug: https://bugs.gentoo.org/708448 Bug: https://bugs.gentoo.org/704320 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* create_world_atom: fix spelling of multiple in commentsZac Medico2020-02-031-2/+2
| | | | | Reported-by: Arfrever Frehtes Taifersar Arahesis <Arfrever@Apache.Org> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* create_world_atom: fix outdated comments regarding greedy atomsZac Medico2020-02-031-5/+6
| | | | | | Fixes: 1342b456c460 ("Remove --update greedy SLOTs code from depgraph.select_files()") Reported-by: Arfrever Frehtes Taifersar Arahesis <Arfrever@Apache.Org> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* cpv_expand: treat GLEP 81 acct-* categories like virtual (bug 691798)Zac Medico2020-02-011-1/+1
| | | | | | | | | | This solves common name collisions introduced by GLEP 81 user and group management packages. TODO: Provide user configuration so that users an ignore name collisions with selected categories like app-emacs or dev-haskell. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* dep_zapdeps: adjust || preference for slot upgrades (bug 706278)Zac Medico2020-01-261-2/+14
| | | | | | | | | | | | | | | | | | | | | Prefer choices that include a slot upgrade when appropriate, like for the || ( llvm:10 ... llvm:7 ) case reported in bug 706278. In order to avoid pulling in inappropriate slot upgrades, like those which should only be pulled in with --update and --deep, add a want_update flag to each choice which is True for choices that pull in a new slot for which an update is desirable. Mark the test case for bug 480736 as todo, since the "undesirable" slot upgrade which triggers a blocker conflict in this test case is practically indistinguishable from a desirable slot upgrade. This particular blocker conflict is no longer relevant, since current versions of media-libs/libpostproc are no longer compatible with any available media-video/ffmpeg slot. In order to solve this test case, some fancy backtracking (like for bug 382421) will be required. Bug: https://bugs.gentoo.org/706278 Bug: https://bugs.gentoo.org/480736 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Add QA check for unresolved soname dependencies (bug 704320)Zac Medico2020-01-221-11/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Example output for maven-bin-3.6.2 (bug 704618): * QA Notice: Unresolved soname dependencies: * * /usr/share/maven-bin-3.6/lib/jansi-native/freebsd32/libjansi.so: libc.so.7 libutil.so.9 * /usr/share/maven-bin-3.6/lib/jansi-native/freebsd64/libjansi.so: libc.so.7 libutil.so.9 * This warning comes up when a library or executable has one or more soname dependencies (found in its NEEDED.ELF.2 metadata) that could not be resolved by usual means. If you run ldd on files like these then it will report a "not found" error for each unresolved soname dependency. In order to correct problems with soname dependency resolution, use one or more of the approaches described in the following sections. Content of the NEEDED.ELF.2 metadata file may be useful for debugging purposes. Find the NEEDED.ELF.2 file in the ${D}/../build-info/ directory after the ebuild src_install phase completes, or in the /var/db/pkg/*/*/ directory for an installed package. Each line of the NEEDED.ELF.2 file contains semicolon separated values for a single ELF file. The soname dependencies are found in the DT_NEEDED column: E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category External dependencies For packages that install pre-built binaries, it may be possible to resolve soname dependencies simply by adding dependencies for one or more other packages that are known to provide the needed sonames. Removal of unecessary files For packages that install pre-built binaries, it may be possible to resolve soname dependencies simply by removing unnecessary files which have unresolved soname dependencies. For example, some pre-built binary packages include binaries intended for irrelevant architectures or operating systems, and these files can simply be removed because they are unnecessary. Addition of DT_RUNPATH entries If the relevant dependencies are installed in a location that is not included in the dynamic linker search path, then it's necessary for files to include a DT_RUNPATH entry which refers to the appropriate directory. The special $ORIGIN value can be used to create a relative path reference in DT_RUNPATH, where $ORIGIN is a placeholder for the directory where the file having the DT_RUNPATH entry is located. For pre-built binaries, it may be necessary to fix up DT_RUNPATH using patchelf --set-rpath. For example, use patchelf --set-rpath '$ORIGIN' if a given binary should link to libraries found in the same directory as the binary itself, or use patchelf --set-rpath '$ORIGIN/libs' if a given binary should link to libraries found in a subdirectory named libs found in the same directory as the binary itself. For binaries built from source, a flag like -Wl,-rpath,/path/of/directory/containing/libs will create binaries with the desired DT_RUNPATH entry. Addition of DT_SONAME settings If a package installs dynamic libraries which do not set DT_SONAME, then this can lead to unresolved soname dependencies. For dynamic libraries built from source, a flag like -Wl,-soname=foo.so.1 will create a DT_SONAME setting. For pre-built dynamic libraries, it may be necessary to fix up DT_SONAME using patchelf --set-soname. Adjustment to Portage soname resolution logic It may be necessary to adjust Portage soname resolution logic in order to account for special circumstances. For example, Portage soname resolution tolerates missing DT_SONAME for dynamic libraries that a package installs in a directory that its binaries reference via DT_RUNPATH. This behavior is useful for packages that have internal dynamic libraries stored in a private directory. An example is ebtables, as discussed in bug 646190. Bug: https://bugs.gentoo.org/704320 Signed-off-by: Zac Medico <zmedico@gentoo.org>