Get Gentoo!
gentoo.org sites
gentoo.org Wiki Bugs Forums Packages
Planet Archives Sources
Infra Status
aboutsummaryrefslogtreecommitdiff
path: root/lib
Commit message (Collapse)AuthorAgeFilesLines
* emaint --fix merges: add -y, --yes optionZac Medico2021-02-072-6/+19
| | | | | | | | | Since the emaint --fix merges uses emerge --ask, add a -y, --yes option to use --ask=n instead. Bug: https://bugs.gentoo.org/766767 Reviewed-by: Brian Dolbec <dolsen@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* emirrordist: prevent distfiles_db _pkg_str pickle problemsZac Medico2021-01-211-1/+2
| | | | | | | | When storing a cpv as a shelve value, convert _pkg_str to str in order to prevent pickle problems. Bug: https://bugs.gentoo.org/766459 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* binarytree.move_ent: copy on write for package moveZac Medico2021-01-193-16/+48
| | | | | | | | | | | | | Copy on write when applying package moves, and silently skip package moves when the same move has already been applied to the same build of the package. Since the old package instance is preserved, it avoids the problem of having entries for deleted packages remain in the package index. We can simply assume that the package will be deleted by eclean-pkg when its time comes. Bug: https://bugs.gentoo.org/766012 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* bin: misc pylint fixesZac Medico2021-01-181-1/+2
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* ProxyManager: remove unused loop parameterZac Medico2021-01-181-1/+1
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* ProxyManager: Use async and await syntaxZac Medico2021-01-182-8/+5
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* RepoStorageInterface: Use async and await syntaxZac Medico2021-01-185-96/+56
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SubprocessExecTestCase: Use async and await syntaxZac Medico2021-01-181-18/+14
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* ChildWatcherTestCase: Use async and await syntaxZac Medico2021-01-181-6/+4
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsyncFunctionTestCase: Use async and await syntaxZac Medico2021-01-181-6/+4
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* SimpleEmergeTestCase: Use async and await syntaxZac Medico2021-01-181-9/+7
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AuxdbTestCase: Use async and await syntaxZac Medico2021-01-181-7/+5
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* RetryTestCase: Use async and await syntaxZac Medico2021-01-181-19/+9
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* PipeLoggerTestCase: Use async and await syntaxZac Medico2021-01-181-12/+10
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* coroutine: do not require loop parameterZac Medico2021-01-181-5/+1
| | | | | | | | | The loop parameter is not needed since global_event_loop now returns the running event loop for the current thread. Bug: https://bugs.gentoo.org/737698 Bug: https://bugs.gentoo.org/763339 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* bindbapi.aux_update: fix spurious package file renamesZac Medico2021-01-181-1/+1
| | | | | | | | | | | Omit the filename argument for the binarytree.inject method in aux_update, since an in-place update is desired, and the filename argument causes the file to be renamed when binpkg-multi-instance is enabled. Fixes: 328dd4712f88 ("binpkg-multi-instance 3 of 7") Bug: https://bugs.gentoo.org/765847 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* bindbapi.move_ent: use cpv_exists instead of getnameZac Medico2021-01-171-1/+1
| | | | | | | | The getname method is complicated by binpkg-multi-instance and soon BINPKG_FORMAT, so it's much simpler to use cpv_exists. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* doebuild: use NamedTemporaryFile for PORTAGE_BINPKG_TMPFILEZac Medico2021-01-171-5/+8
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Add @changed-subslot package setMatt Turner2021-01-171-2/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This set is the upgradable packages for which the highest visible version has a different subslot than the currently installed version. The primary purpose of this feature is for use in catalyst builds. We update the "seed" stage3 before using it to build a new stage1. Updating the entire stage is expensive and unnecessary (since we're going to build the latest packages in stage1 and then rebuild everything in stage3). What we definitely do need to update in the original stage3 however, is any package that would trigger a subslot rebuild. For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc will link with libmpfr.so.4, but the latest version of dev-libs/mpfr will be built and libmpfr.so.6 included into the stage1. Since the old libmpfr.so.4 is not included in the stage1, gcc will not work, breaking subsequent stage builds. Our current options to update the seed are too large a hammer (e.g., "--update --deep --newuse @world" or "--update --deep --newuse --complete-graph --rebuild-if-new-ver gcc") and spend too much time updating seed stages for no gain beyond updating only packages for whom the subslot has changed. With this set, catalyst will likely use emerge @changed-subslot --ignore-built-slot-operator-deps y to update the seed stage. Thank you to Zac Medico for showing me how to do this. Bug: https://bugs.gentoo.org/739004 Signed-off-by: Matt Turner <mattst88@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* bindbapi.move_ent: remove redundant self._pkg_str callsZac Medico2021-01-171-7/+3
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* vardbapi.move_ent: remove redundant self._pkg_str callsZac Medico2021-01-171-6/+2
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* global_event_loop: return running loop for current threadZac Medico2021-01-112-34/+24
| | | | | | | | | | | | | | | | Like asyncio.get_event_loop(), return the running loop for the current thread if there is one, and otherwise construct a new one if needed. This allows the _safe_loop function to become synonymous with the global_event_loop function. For the case of "loop running in non-main thread" of API consumer, this change makes portage compatible with PEP 492 coroutines with async and await syntax. Portage internals can safely begin using async / await syntax instead of compat_coroutine. Bug: https://bugs.gentoo.org/763339 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* AsyncioEventLoop: wrap child watcher for thread safety (bug 764905)Zac Medico2021-01-101-1/+29
| | | | | | | | | | Use a child watcher wrapper to deliver the callbacks via the call_soon_threadsafe method, since documentation for the asycio AbstractChildWatcher class says that callbacks must be thread safe. Bug: https://bugs.gentoo.org/764905 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* check_reverse_dependencies: dereference virtual expansions (bug 764764)Zac Medico2021-01-102-1/+23
| | | | | | | | | | | | | If an atom is the result of virtual expansion, then derefrence it to _orig_atom in check_reverse_dependencies so that it will be correctly handled as a built slot operator dependency when appropriate. This solves a case triggered in bug 764764 where a virtual expansion from virtual/dist-kernel:0/5.10.5= to =virtual/dist-kernel-5.10.5 prevented the atom from being handled as a built slot operator dependency, which prevented rebuilds from being triggered. Bug: https://bugs.gentoo.org/764764 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Add unit test demonstrating bug 764764Zac Medico2021-01-101-1/+97
| | | | | Bug: https://bugs.gentoo.org/764764 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Make digests available to fetch commandDaniel M. Weeks2021-01-091-0/+6
| | | | | | | | | This makes it possible for the fetch command to utilize a protocol/service that identifies files by their hash. Closes: https://github.com/gentoo/portage/pull/655 Signed-off-by: Daniel M. Weeks <dan@danweeks.net> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Fix varexpand, splitting in fetchDaniel M. Weeks2021-01-091-2/+3
| | | | | | | | | | | | | | | | | | Performing variable expansion after splitting prevents expanding a variable to an unquoted form: Given VAR="a b c": "echo \${VAR}" produces ['echo', 'a b c'] It should be possible for a user to control quoting such that: "echo \"\${VAR}\"" produces ['echo', 'a b c'] "echo \${VAR}" produces ['echo', 'a', 'b', 'c'] Reordering splitting and variable expansion fixes this and mirrors the order used in _start_gpg_proc of ManifestTask. Signed-off-by: Daniel M. Weeks <dan@danweeks.net> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _Retry: Use ensure_future for self._current_taskZac Medico2021-01-041-2/+2
| | | | | | | Use ensure_future for compatibility with PEP 492 coroutines with async and await syntax. Signed-off-by: Zac Medico <zmedico@gentoo.org>
* test_retry: use context manager to cancel pending futuresZac Medico2021-01-041-75/+106
| | | | | | | | | | | | | | | | | | | Cancel pending futures in order to avoid this "Task was destroyed but it is pending!" warning message following migration to PEP 492 coroutines with async and await syntax: testHangForever (portage.tests.util.futures.test_retry.RetryForkExecutorTestCase) ... ok testHangForever (portage.tests.util.futures.test_retry.RetryTestCase) ... ok testHangForever (portage.tests.util.futures.test_retry.RetryThreadExecutorTestCase) ... ok ---------------------------------------------------------------------- Ran 3 tests in 0.839s OK Task was destroyed but it is pending! task: <Task cancelling name='Task-4' coro=<HangForever.__call__() running at portage/tests/util/futures/test_retry.py:58> wait_for=<Future cancelled> cb=[RetryForkExecutorTestCase._wrap_coroutine_func.<locals>.wrapper.<locals>.done_callback() at portage/tests/util/futures/test_retry.py:192]> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* ci: enable repoman testsZac Medico2021-01-021-1/+2
| | | | | Bug: https://bugs.gentoo.org/763045 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* lib/portage/util/__init__.py: Fix useless-returnZac Medico2021-01-011-1/+0
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* env_update: use "with statement" on atomic_ofstreamFlorian Schmaus2020-12-301-18/+12
| | | | | Signed-off-by: Florian Schmaus <flo@geekplace.eu> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Make atomic_ofstream a Context ManagerFlorian Schmaus2020-12-301-1/+9
| | | | | | | | | This allows using a "with statement" together with instances of atomic_ofstream. Allowing for more readable, less error prone and shorter code. Signed-off-by: Florian Schmaus <flo@geekplace.eu> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* _safe_loop: instantiate asyncio loop for API consumer threadZac Medico2020-12-061-9/+40
| | | | | | | | | | | | | In order to maintain compatibility with an API consumer thread which has not instantiated an asyncio loop for the current thread prior to calling the portage API, instantiate a loop on its behalf. Since a ResourceWarning will be triggered if the loop has not been closed before the process exits, add the loop to a WeakValueDictionary, and close it if it still exists during exit for the current pid. Fixes: cecd2f8a259c ("Use default asyncio event loop implementation in API consumer threads") Bug: https://bugs.gentoo.org/758755 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Remove unused EventLoop lazy importsZac Medico2020-12-062-2/+0
| | | | Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Use default asyncio event loop implementation in API consumer threadsZac Medico2020-12-062-8/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Make the _safe_loop function return an AsyncioEventLoop instance, so that the default asyncio event loop implementation will be used in API consumer threads. This is possible because the underlying asyncio.get_event_loop() function returns a separate event loop for each thread. The AsyncioEventLoop _run_until_complete method will now appropriately handle a ValueError from signal.set_wakeup_fd(-1) if it is not called in the main thread. For external API consumers calling from a non-main thread, an asyncio loop must be registered for the current thread, or else an error will be raised like this: RuntimeError: There is no current event loop in thread 'Thread-1'. In order to avoid this RuntimeError, the external API consumer is responsible for setting an event loop and managing its lifecycle. For example, this code will set an event loop for the current thread: asyncio.set_event_loop(asyncio.new_event_loop()) In order to avoid a ResourceWarning, the caller should also close the corresponding loop before the current thread terminates. Bug: https://bugs.gentoo.org/758755 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Use default asyncio event loop implementation in child processesZac Medico2020-12-062-7/+4
| | | | | | | | | | | Use the default asyncio event loop implementation in child processes, instead of portage's internal EventLoop. After fork, force instantiation of a new event loop policy as a workaround for https://bugs.python.org/issue22087, which is necessary for RetryTestCase to succeed. Bug: https://bugs.gentoo.org/758740 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* depgraph._add_pkg: fix AttributeError when myparent is not a PackageZac Medico2020-12-031-1/+1
| | | | | | Fixes: b991f23ad915 ("Allow a package to replace its own buildtime dependency") Bug: https://bugs.gentoo.org/757741 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* MergeOrderTestCase: fix x11-drivers/xf86-video-fbdev slot operator depsZac Medico2020-12-021-4/+4
| | | | | | | | The real ebuild has the x11-base/xorg-server:= dependency in RDEPEND rather than DEPEND. Fixes: 07a604537e74 ("find_smallest_cycle: don't merge satisfied PDEPEND too early") Signed-off-by: Zac Medico <zmedico@gentoo.org>
* find_smallest_cycle: don't merge satisfied PDEPEND too earlyZac Medico2020-12-023-6/+30
| | | | | | | | | | | | | | After PDEPENDs have been neglected by the find_smallest_cycle function, do not try to merge them too early if they are already satisfied by an installed package. This fixes incorrect merge order for PDEPEND cycles involving xorg-server and xorg-drivers, which was triggered by commit 5095c2023595a75e2848f1ad3dbe25b5fb451a44 because it gave PDEPEND higher priority than satisfied buildtime dependencies. Fixes: 5095c2023595 ("find_smallest_cycle: enhance search prioritization") Reported-by: josef64 in #gentoo-portage Bug: https://bugs.gentoo.org/754903 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Allow a package to replace its own buildtime dependencyZac Medico2020-11-283-22/+164
| | | | | | | | | | | | | | | | | If a package has a buildtime dependency on a previous version that it will replace, then do not treat it as a slot conflict. This solves inappropriate behavior for dev-lang/rust[system-bootstrap]. This requires adjustments to package selection logic in several locations, in order to ensure that an installed package instance will be selected to satisfy a buildtime dependency when appropriate. Dependencies of the installed package will be entirely ignored, but that has already been the case when using installed package to break cycles, as discussed in bug 199856. Bug: https://bugs.gentoo.org/756793 Bug: https://bugs.gentoo.org/756961 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* backtracking: fix virtual choices for circular deps (bug 757306)Zac Medico2020-11-271-2/+5
| | | | | | | | | Fix virtual choices to be consistent with circular dependency backtracking choices. Fixes: f78a91e44e3e ("backtracking: adjust || preference to break dependency cycles") Bug: https://bugs.gentoo.org/757306 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* compare_files: fix missing xattr handlingZac Medico2020-11-221-2/+2
| | | | | | Fixes: a2e7bf7d1c7d ("compare_files: handle missing xattr support") Bug: https://bugs.gentoo.org/755950 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* compare_files: handle missing xattr supportZac Medico2020-11-211-1/+2
| | | | | Bug: https://bugs.gentoo.org/755950 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* find_smallest_cycle: enhance search prioritizationZac Medico2020-11-214-38/+70
| | | | | | | | | | | | | | | | | | | | | | | | | Enhance the find_smallest_cycle function to prioritize its search so that it will minimize the use of installed packages to break cycles. When installed packages must be used to break cycles, it will now prefer to do this for runtime dependencies over buildtime dependencies, since it's preferable to build against latest versions of buildtime dependencies whenever possible. This should solve some cases of bug 199856 which have been triggered by unsafe reliance on installed packages to break cycles. The included unit test case demonstrates correct merge order for a dependency calculation involving 6 independent cycles. This test case fails with the master branch, due to a buildtime dependency cycle of 3 packages being merged earlier than cycles of 2 packages. We can generalize this to say that the master branch may use an installed package to break an arbitrarily sized cycle in a somewhat random location, even though that cycle may be composed of smaller independent cycles which would be safer to break individually. Bug: https://bugs.gentoo.org/754903 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* emerge: Disable profile deprecation warning inheritance (bug 753497)Zac Medico2020-11-212-12/+27
| | | | | | | | | | | | According to PMS, a deprecated profile warning is not inherited. Since the current profile node may have been inherited by a user profile node, the deprecation warning may be relevant even if it is not a top-level profile node. Therefore, consider the deprecated warning to be irrelevant when the current profile node belongs to the same repo as the previous profile node. Bug: https://bugs.gentoo.org/753497 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Typofix: Use just one definite articlePetr Šabata2020-11-216-7/+7
| | | | | | | | | I first noticed this in make.conf(5) but it turned out there were many more occurences of these; fixed with find & sed. Closes: https://github.com/gentoo/portage/pull/634 Signed-off-by: Petr Šabata <contyk@redhat.com> Signed-off-by: Zac Medico <zmedico@gentoo.org>
* Add slot operator := BDEPEND unit tests for bug 752153Zac Medico2020-11-141-0/+209
| | | | | | | | | | | | | | Test these two distinct cases: 1) Regular dev-lang/go upgrade, with rebuild of packages that have dev-lang/go:= in BDEPEND. 2) Rebuild of packages that have dev-lang/go:= in BDEPEND, after the built slot operator deps have already been broken by an earlier dev-lang/go upgrade. Bug: https://bugs.gentoo.org/752153 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* make.conf: expand special *ROOT variables (bug 752147)Zac Medico2020-11-011-1/+6
| | | | | Bug: https://bugs.gentoo.org/752147 Signed-off-by: Zac Medico <zmedico@gentoo.org>
* emerge: add --quickpkg-direct-root optionZac Medico2020-11-014-6/+32
| | | | | | | | | Specify the root to use as the --quickpkg-direct package source. This root is assumed to be immutable during the entire emerge operation. The default is set to "/". Bug: https://bugs.gentoo.org/752066 Signed-off-by: Zac Medico <zmedico@gentoo.org>