aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Scheduler: use add_done_callback (bug 711322)Zac Medico2020-03-051-14/+22
| | | | | | | | | | | | | Use add_done_callback instead of addExistListener, in order to avoid callback races like the SequentialTaskQueue exit listener race that triggered bug 711322. The addExistListener method is prone to races because its listeners are executed in quick succession. In contrast, callbacks scheduled via add_done_callback are placed in a fifo queue, ensuring that they execute in an order that is unsurprising relative to other callbacks. Bug: https://bugs.gentoo.org/711322 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Updates for portage-2.3.92 releaseportage-2.3.92Zac Medico2020-03-042-1/+14
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SequentialTaskQueue: update bool(self) sooner (bug 711322)Zac Medico2020-03-041-7/+9
| | | | | | | | | | | | | | | | Use addExitListener to add a _task_exit callback that will be invoked as soon as the task exits (before the future's done callback is called). This is required in order for bool(self) to have an updated value for Scheduler._schedule to base assumptions upon. Delayed updates to bool(self) is what caused Scheduler to hang as in bug 711322. This reverts changes in SequentialTaskQueue task queue exit listener behavior from commit c7e52d046621, so that only the changes necessary to support async_start remain. Fixes: c7e52d046621 ("EbuildPhase: add _async_start coroutine") Bug: https://bugs.gentoo.org/711322 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Rename PORTAGE_LOG_FILTER_FILE_CMD from PORTAGE_LOG_FILTER_FILEZac Medico2020-03-027-8/+8
| | | | | | Suggested-by: Michael 'veremitz' Everitt <gentoo@veremit.xyz> Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* dep_zapdeps: skip want_update_pkg if parent is None (bug 711400)Zac Medico2020-03-021-1/+1
| | | | | | | | | | | | | | | Skip the want_update_pkg call when parent is None, since passing a None parent to want_update_pkg results in an error like this: File "libdep/dep_check.py", line 513, in dep_zapdeps if graph_interface.want_update_pkg(parent, avail_pkg): File "lib/_emerge/depgraph.py", line 5850, in _want_update_pkg depth = parent.depth or 0 AttributeError: 'NoneType' object has no attribute 'depth' Fixes: f7d83d75c6b0 ("dep_zapdeps: adjust || preference for slot upgrades (bug 706278)") Bug: https://bugs.gentoo.org/711400 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* EbuildMetadataPhase: make _unregister idempotent (bug 711362)Zac Medico2020-03-021-1/+2
| | | | | | | | | | Make the _unregister method check if self._files is None, as necessary for idempotency. This will prevent the AttributeError reported in bug 711362, which was triggered by recent changes in callback scheduling order. Closes: https://bugs.gentoo.org/711362 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* doebuild: avoid emerge --pretend --fetchonly event loop recursionZac Medico2020-03-021-2/+10
| | | | | | Fixes: ec654122c0eb ("fetch: drop privileges early for NFS root_squash (bug 601252)") Bug: https://bugs.gentoo.org/601252 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* bin/socks5-server.py: PEP 492 coroutines with async and await syntaxZac Medico2020-03-011-19/+17
| | | | | https://bugs.gentoo.org/709334 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Updates for portage-2.3.91 releaseportage-2.3.91Zac Medico2020-03-012-1/+8
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsyncioEventLoop: always die with SIGTERM in exception handler (bug 705910)Zac Medico2020-03-011-22/+9
| | | | | | | | | Remove call to pdb.set_trace() in exception handler, since it's not very useful, and always die with a SIGTERM for unexpected exceptions here. Bug: https://bugs.gentoo.org/705910 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _check_build_log: convert zlib EOFError to eerror message (bug 711174)Zac Medico2020-03-011-1/+1
| | | | | | | | Display a non-fatal eerror message when EOFError is encountered, since this need not be fatal. Bug: https://bugs.gentoo.org/711174 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Subprocess: delay unregister in _async_wait (bug 711174)Zac Medico2020-03-011-1/+10
| | | | | | | | | | | | | Since the super class AbstractPollTask _async_wait method calls _unregister, it can trigger premature _unregister before pid status is available. Therefore, only call the super class _async_wait method after pid status is available. This should help prevent premature _unregister events that trigger reading of build logs before they're closed as in bug 658806 and bug 711174. Bug: https://bugs.gentoo.org/711174 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _GeneratorTask: throw CancelledError in cancelled coroutine (bug 711174)Zac Medico2020-03-013-11/+49
| | | | | | | | | | | | | | | | Throw asyncio.CancelledError in a cancelled coroutine, ensuring that the coroutine can handle this exception in order to perform any necessary cleanup (like close the log file for bug 711174). Note that the asyncio.CancelledError will only be thrown in the coroutine if there's an opportunity (yield) before the generator raises StopIteration. Also fix the AsynchronousTask exit listener handling for compatibility with this new behavior. Fixes: 8074127bbc21 ("SpawnProcess: add _main coroutine") Bug: https://bugs.gentoo.org/711174 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: cancel _main_task in _unregister (bug 711174)Zac Medico2020-03-011-0/+2
| | | | | | | | | | | | | | | | Cancel _main_task in _unregister, in order to ensure that the build log is properly closed. Note that is analogous to the fix for bug 658806 in commit a9a62e57194c, but behavior related to coroutine cancellation now plays a role since the PipeLogger is now cancelled indirectly via _main_task. We still need to verify that coroutine cancellation will raise an appropriate exception in the coroutine (asyncio.CancelledError, GeneratorExit, or StopIteration) and also that the specific coroutine will react appropriately to the specific exception that will be raised. Fixes: 8074127bbc21 ("SpawnProcess: add _main coroutine") Bug: https://bugs.gentoo.org/711174 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Add python3.9 supportManuel Rüger2020-03-014-11/+12
| | | | | Closes: https://github.com/gentoo/portage/pull/515 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _BinpkgFetcherProcess: fix async_lock event loop recursion (bug 711178)Zac Medico2020-03-011-15/+17
| | | | | | | | | Make the async_lock method use the AsynchronousLock async_start method in order to avoid event loop recursion. Fixes: 5c40c3e7e ("SpawnProcess: use async_start method (bug 709746)") Bug: https://bugs.gentoo.org/711178 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Updates for portage-2.3.90 releaseportage-2.3.90Zac Medico2020-02-293-1/+16
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* fetch: drop privileges early for NFS root_squash (bug 601252)Zac Medico2020-02-293-6/+59
| | | | | | | | | Drop privileges prior to fetch function calls, so that all necessary operations can succeed when DISTDIR is on NFS with root_squash enabled. Bug: https://bugs.gentoo.org/601252 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Support PORTAGE_LOG_FILTER_FILE (bug 709746)Zac Medico2020-02-297-20/+213
| | | | | | | | | | | | | | This variable specifies a command that filters build log output to a log file. The plan is to extend this to support a separate filter for tty output in the future. In order to enable the EbuildPhase class to write elog messages to the build log with PORTAGE_LOG_FILTER_FILE support, convert its _elog method to a coroutine, and add a SchedulerInterface async_output method for it to use. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _PostPhaseCommands: avoid CancelledError in _soname_deps_qa done callbackZac Medico2020-02-291-1/+1
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* EbuildBuildDir: cancel current tasks for CancelledErrorZac Medico2020-02-291-16/+30
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsyncScheduler: cancel task after _task_coroutine CancelledErrorZac Medico2020-02-291-0/+1
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* CompositeTask: call self.cancel() for _start_task CancelledErrorZac Medico2020-02-291-1/+1
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: cancel pipe_logger after async_start CancelledErrorZac Medico2020-02-291-1/+6
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: make _main cancel pipe_logger when appropriateZac Medico2020-02-291-2/+7
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: only cancel self._main_task if it's not doneZac Medico2020-02-291-1/+1
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: add _main coroutineZac Medico2020-02-291-11/+22
| | | | | | | Add a _main coroutine method and make the PipeLogger instance a local variable. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* FileDigester: eliminate _pipe_logger_exit with _async_waitpidZac Medico2020-02-281-2/+3
| | | | | | | Eliminate the _pipe_logger_exit method by overriding the _async_waitpid method. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsyncFunction: eliminate _pipe_logger_exit with _async_waitpidZac Medico2020-02-281-2/+3
| | | | | | | Eliminate the _pipe_logger_exit method by overriding the _async_waitpid method. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* asyncio: improve _AsyncioEventLoop isinstance logicZac Medico2020-02-281-4/+4
| | | | | | | | Since _AsyncioEventLoop can be wrapped, use the _asyncio_wrapper attributre for isinstance checks (_wrap_loop guarantees that this attribute exists). Signed-off-by: Zac Medico <zmedico@gentoo.org>
* repoman: Restore eclass deprecations lost in rewriteMichał Górny2020-02-261-0/+5
| | | | | Reviewed-by: Zac Medico <zmedico@gentoo.org> Signed-off-by: Michał Górny <mgorny@gentoo.org>
* PipeLogger: non-blocking write to pipe (bug 709746)Zac Medico2020-02-242-14/+94
| | | | | | | | | | | Add support to write to a non-blocking pipe instead of a log file. This is needed for the purposes of bug 709746, where PipeLogger will write to a pipe that is drained by anoher PipeLogger instance which is running in the same process. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SpawnProcess: use async_start method (bug 709746)Zac Medico2020-02-2310-19/+61
| | | | | | | | | | | | | | Convert SpawnProcess to use the async_start method, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Also update AbstractEbuildProcess, AsynchronousLock, BinpkgExtractorAsync, BinpkgFetcher, EbuildFetcher, IterCompletedTestCase, AsyncFunction, and FileDigester for compatibility with SpawnProcess async_start. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* ForkExecutor: use async_start methodZac Medico2020-02-232-5/+31
| | | | | | | | Also, fix AsynchronousTask.async_start to handle cancellation of the _async_start coroutine, ensuring that start and exit listeners are notified in this case (otherwise RetryForkExecutorTestCase will hang). Signed-off-by: Zac Medico <zmedico@gentoo.org>
* EbuildBuildDir: use async_start methodZac Medico2020-02-231-78/+44
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* action_sync: display error message with --quietZac Medico2020-02-211-0/+3
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* einstalldocs: Fix test for DOCS being unset.Ulrich Müller2020-02-201-2/+2
| | | | | | | | | | | | | | | | | | | The current test does not exactly test for unset DOCS, because it also evaluates as true if the variable has attributes. Such attributes can be defined even for an unset variable. Therefore test the output of declare -p for presence of an = sign instead, which indicates that a value has been assigned to the variable (bug 710076 comment #2). PMS reference: Algorithm 12.4, line 7: https://projects.gentoo.org/pms/7/pms.html#x1-135011r183 See also bash upstream discussion: https://lists.gnu.org/archive/html/bug-bash/2020-02/msg00045.html Closes: https://bugs.gentoo.org/710076 Signed-off-by: Ulrich Müller <ulm@gentoo.org>
* async_fetch_map: handle _parse_uri_map exception (bug 710130)Zac Medico2020-02-201-3/+6
| | | | | | | | If _parse_uri_map raises an exception then raise it to the caller via the returned future. Bug: https://bugs.gentoo.org/710130 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsyncScheduler: use async_start methodZac Medico2020-02-204-15/+53
| | | | | | | | | | | | | Convert AsyncScheduler to use the async_start method, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Also fix async_iter_completed to be compatible with callback scheduling differences introduced by migration to the async_start method. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: Call _start_hook after _start_asyncZac Medico2020-02-171-2/+2
| | | | | | | | | The start listeners need to run after _start_async has completed in case they rely on any state that _start_async has created (same goes for _start). Fixes: d66e9ec0b105 ("AsynchronousTask: add coroutine async_start method") Signed-off-by: Zac Medico <zmedico@gentoo.org>
* EbuildPhase: add _async_start coroutineZac Medico2020-02-172-14/+27
| | | | | | | | | | | | Convert the _start method to an _async_start coroutine, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Also convert SequentialTaskQueue to use the async_start method, which is now required in order to start EbuildPhase instances. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AbstractEbuildProcess: add _async_start coroutineZac Medico2020-02-172-27/+14
| | | | | | | | | Convert the _start method to an _async_start coroutine, since eventually this method will need to be a coroutine in order to write messages to the build log as discussed in bug 709746. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsynchronousTask: add coroutine async_start methodZac Medico2020-02-1715-48/+87
| | | | | | | | | | | Add coroutine async_start coroutine method which calls an _async_start template method. Eventually, subclasses having _start implementations that need to write to a build log will be required to implement an _async_start coroutine method to replace the _start method as discussed in bug 709746. Bug: https://bugs.gentoo.org/709746 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* fetch: remove secpass import (bug 601252)Zac Medico2020-02-151-3/+3
| | | | | | | | | | Refer to the portage.secpass.data attribute directly rather than via import value, since the value will change after fork and os.setuid calls when the fetch function is called with dropped privileges as discussed in bug 601252. Bug: https://bugs.gentoo.org/601252 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* fetch: split out _ensure_distdir function (bug 601252)Zac Medico2020-02-151-47/+55
| | | | | | | | | Split out an _ensure_distdir function which will have to be called earlier if the fetch function is called with dropped privileges as discussed in bug 601252. Bug: https://bugs.gentoo.org/601252 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* RELEASE-NOTES: fix portage-2.3.89 typoZac Medico2020-02-151-1/+1
| | | | | Reported-by: josef.95 <josef.95@freenet.de> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Updates for portage-2.3.89 releaseportage-2.3.89Zac Medico2020-02-142-1/+7
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _expand_set_args: un-reverse nested set traversalZac Medico2020-02-142-6/+4
| | | | | | | Un-reverse traversal order for @profile @selected @system. Bug: https://bugs.gentoo.org/649622 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* depclean: ensure consistency with update actions (bug 649622)Zac Medico2020-02-144-47/+96
| | | | | | | | | | | | | | Make depclean traverse dependencies in the same order as update actions, in order to ensure consistency in decisions which are dependent on the order of dependency evaluation due to inconsistent use of || preferences in different packages. In unit tests, update test_virtual_w3m_realistic to assert that the order of graph traversal is deterministic and consistent between update and removal (depclean) actions. Bug: https://bugs.gentoo.org/649622 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* depgraph: sort nested package set names for consistent resultsZac Medico2020-02-111-2/+4
| | | | | | | | | Traverse the unordered set returned from PackageSet.getNonAtoms() in sorted order, for more consistent results. Use a reverse sort since we pop these sets from a stack. Bug: https://bugs.gentoo.org/649622 Signed-off-by: Zac Medico <zmedico@gentoo.org>