Get Gentoo!
gentoo.org sites
gentoo.org Wiki Bugs Forums Packages
Planet Archives Sources
Infra Status
summaryrefslogtreecommitdiff
path: root/scripts/rsync-generation
Commit message (Collapse)AuthorAgeFilesLines
* pts/rsync-generation/update-rsync-master: fix PYTHONPATHFabian Groffen2024-07-101-1/+1
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: don't download a crap 404 messageFabian Groffen2024-06-141-1/+1
| | | | | | | Fail instead of downloading a 404/403/500 whatever message. Bug: https://bugs.gentoo.org/934308 Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: disable all but bz2 snapshotsFabian Groffen2024-03-311-2/+8
| | | | | | | | | | | who am I kidding, the bootstrap and the mirror retaining is all hardwired to bz2, and other compression formats don't really make a big enough dent to warrant lots of changes to allow picking a different compresion format disable all but bz2, that's the only thing we use afterall Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: xz can read lzip archives, so drop xzFabian Groffen2024-03-311-1/+0
| | | | | | | lzip (lzma) compresses better than xz (lzma2) but xz can read/decompress lzip archives, so drop the xz archive in favour of lzip Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: drop gzip snapshotFabian Groffen2024-03-311-1/+0
| | | | | | | | | | gzip compresses to around 75MB which is much higher than bzip2's 57MB The gzip snapshot was historically never used ever by the bootstrap script, it always used and hardcoded bz2. So don't bother creating a gzip variant. Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: increase zstd compression levelFabian Groffen2024-03-311-1/+1
| | | | | | | | | Using -9 we end up at 58MB, which is more than gzip2's 57MB. So up the level to -19 (the max, basically like how we configure the other compressors too) to get 52MB, which is still much more than xz (48MB) and lzip (47MB). Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: fix zstd file extensionFabian Groffen2024-03-311-1/+1
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: also compress using lzipFabian Groffen2024-03-301-0/+1
| | | | | | | lzip seems to reach a better compression ratio than xz, let's see if we can use it Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/bootstrap-prefix: shellcheckFabian Groffen2024-03-301-1/+1
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: start generating Zstd snapshotsFabian Groffen2024-03-301-22/+33
| | | | | | Prepare for Zstandard becoming a standard, replacing another compressor. Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/update-rsync-master: disable shellcheck fpsFabian Groffen2024-03-291-0/+3
| | | | | | | Disable false positives, unfortunately including the most helpful check SC2086. Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: shellcheckFabian Groffen2024-03-281-3/+3
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/update-rsync-master: shellcheckFabian Groffen2024-03-281-15/+19
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/refresh-mirror: shellcheckFabian Groffen2024-03-281-6/+10
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: shellcheckFabian Groffen2024-03-281-49/+39
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/push-rsync1: update to current usageFabian Groffen2024-03-281-17/+15
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/mksnapshot: only *add* snapshotsFabian Groffen2023-09-111-1/+3
| | | | | | | don't overwrite existing snapshots, only add so as to avoid changing generated snapshots Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts: make bootstrap snapshot available on rsync slavesFabian Groffen2023-09-031-12/+24
| | | | | | | | | | | | | | In bug #913415 a bunch of confusing and wrong things came out, and in an attempt to fix this, the historical "prefix-overlay" fetched from dev was replaced with ordinary "portage" from rsync snapshots dir. However, this relies on the prefix mirror to provide the file, which can easily be overridden by the user. So retrieve the snapshot in use and retain it on the rsync slaves to make this in the very least less confusing for those following the locations, and ensure it works for people with custom mirror setups. Bug: https://bugs.gentoo.org/913415 Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* s/r-g/update-rsync-master.sh: put an alias gentoo repo name.Benda Xu2023-08-011-2/+5
| | | | | Closes: https://bugs.gentoo.org/911543 Signed-off-by: Benda Xu <heroxbd@gentoo.org>
* scripts/rsync-generation/update-rsync-master.sh: use stronger git resetFabian Groffen2023-04-091-15/+11
| | | | | | | use git reset --hard HEAD to bring the tree out of any conflict or pollution state Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts: Resolve shellcheck errorsPeter Foley2023-02-241-1/+1
| | | | | | | | | | | | | | | | | | The action is failing at head, which makes it less than useful for evaluating new changes. e.g. Error: ./scripts/rsync-generation/refresh-mirror.sh:44:2: error: Shells disambiguate (( differently or not at all. For subshell, add spaces around ( . For ((, fix parsing errors. [SC1105] Error: ./scripts/rsync-generation/refresh-mirror.sh:44:3: error: Shells disambiguate (( differently or not at all. For subshell, add spaces around ( . For ((, fix parsing errors. [SC1105] Error: ./scripts/bootstrap-prefix.sh:5:20: error: SIGKILL/SIGSTOP can not be trapped. [SC2173] Error: ./scripts/bootstrap-prefix.sh:1669:19: error: Double quote array expansions to avoid re-splitting elements. [SC2068] Error: ./scripts/bootstrap-prefix.sh:1964:19: error: Double quote array expansions to avoid re-splitting elements. [SC2068] Error: ./scripts/bootstrap-prefix.sh:2286:19: error: Double quote array expansions to avoid re-splitting elements. [SC2068] Error: ./scripts/bootstrap-prefix.sh:3194:11: error: -e doesn't work with globs. Use a for loop. [SC2144] Signed-off-by: Peter Foley <pefoley2@pefoley.com> Closes: https://github.com/gentoo/prefix/pull/19 Signed-off-by: Sam James <sam@gentoo.org>
* scripts/rsync-generation/update-rsync-master: zap any changes before pullFabian Groffen2022-08-171-0/+5
| | | | | | | Somehow the git trees can get dirty, ensure they are completely clean each time. Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/update-rsync-master: blast if you canFabian Groffen2022-07-241-1/+1
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/update-rsync-master: hashgen -> qmanifestFabian Groffen2019-06-071-14/+3
| | | | | | | Now hashgen is integrated in portage-utils as qmanifest, switch to using it. Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation: update READMEFabian Groffen2019-06-071-6/+2
| | | | Signed-off-by: Fabian Groffen <grobian@gentoo.org>
* scripts/rsync-generation/update-rsync-master: include timestamp.commit filesFabian Groffen2018-05-141-0/+4
|
* scripts/rsync-generation/update-rsync-master: deal with nested files ↵Fabian Groffen2018-03-291-3/+8
| | | | (filesdir) in apply_git_mtimes
* scripts/rsync-generation/update-rsync-master: touch cat metadata.xml upon ↵Fabian Groffen2018-03-271-2/+6
| | | | package removal
* scripts/rsync-generation/hashgen: moved to its own repositoryFabian Groffen2018-03-171-1510/+0
| | | | hashgen will live on in github.com/grobian/hashgen
* scripts/rsync-generation/update-rsync-master: update package metadata.xmlFabian Groffen2018-03-171-1/+2
|
* scripts/rsync-generation/hashgen: report no garbage when key is missingFabian Groffen2018-03-121-17/+24
|
* scripts/rsync-generation/update-rsync-master: ensure deletes bump mtimeFabian Groffen2018-03-101-0/+5
|
* scripts/rsync-generation/hashgen: count missing and stray files as failedFabian Groffen2018-03-071-0/+2
|
* scripts/rsync-generation/hashgen: rewrite hashgen to be simplerFabian Groffen2018-03-031-168/+256
| | | | | a logic rewrite was necessary to be able to properly generate the right type of Manifest for the level at hand
* scripts/rsync-generation/hashgen: report some statsFabian Groffen2018-03-011-0/+20
|
* scripts/rsync-generation/hashgen: make error messages more uniformFabian Groffen2018-03-011-3/+3
|
* scripts/rsync-generation: remove tests again, they fail the treeFabian Groffen2018-03-0110-74/+9
| | | | | Tests were designed to expose problems, so it's kind of nasty when they are included in the tree. Not a good idea.
* scripts/rsync-generation/hashgen: always use sorted directory listingFabian Groffen2018-03-011-92/+122
| | | | | | To get reproducible outputs, ensure we sort the directory listings, such that the order and thus optional compression ratio is the same. This is in particular nice when multiple generators are in use.
* scripts/rsync-generation: add initial tests for hashverifyFabian Groffen2018-03-019-0/+69
|
* scripts/rsync-generation/hashgen: return when write_hashes would produce garbageFabian Groffen2018-03-011-7/+10
| | | | | | If we can't stat the file we're writing hashes for, we're not going to produce anything useful and something up in the stack went wrong already.
* scripts/rsync-generation/update-rsync-master: ensure mtime always moves forwardsFabian Groffen2018-02-281-1/+12
| | | | | relying on times stored in git is never going to work well, so pick a time that we reasonably sync between multiple generators
* script/rsync-generation/hashgen: allow compilation on DarwinFabian Groffen2018-02-281-0/+4
|
* scripts/rsync-generation/hashgen: allow verification to be massively parallelFabian Groffen2018-02-281-30/+79
| | | | | save up subdirs to handle so we can nicely loop over them in a for-loop which OpenMP can cheaply parallelise for us (for free)
* scripts/rsync-generation/hashgen: properly free members of listsFabian Groffen2018-02-281-8/+13
|
* scripts/rsync-generation/hashgen: try to print verification issues prettierFabian Groffen2018-02-281-28/+67
| | | | | Still not very happy about this, need a clearer way to deal with issues to make a readable conclusion.
* scripts/rsync-generation/update-rsync-master: use -d with git-cleanFabian Groffen2018-02-281-5/+5
| | | | | also remove stray directories that for some reason where left behind, as they cause verification problems
* hashgen: first shot at hashverifyFabian Groffen2018-02-281-32/+716
| | | | This variant can verify the gx86 tree, or so it seems.
* scripts/rsync-generation/update-rsync-master: don't bother egencache with ↵Fabian Groffen2018-02-221-2/+3
| | | | missing digests
* scripts/rsync-generation/update-rsync-master: set thin-manifests to falseFabian Groffen2018-02-221-0/+2
| | | | | | When using Git, we use thin-manifests, but on the rsync tree, we transformed the manifests into fat ones (hashgen), so we need to ensure Portage knows that and performs its checks.
* scripts/rsync-generation/update-rsync-master: use CommitDate iso AuthorDateFabian Groffen2018-02-211-1/+1
| | | | | AuthorDate can be anything, as I just found out, so don't rely on that. Use CommitDate instead, which is hopefully closer to reality.