Hi, I skipped the report last month for lack of time, so the delta below is for two months. I managed to do some patches and bug-reports. Mostly mundane stuff. The most noteworthy thing is the fallout from https://github.com/rpm-software-management/rpm/pull/2047 that adds the number of build machine cores to many rpms, so it gets harder to bit-reproduce. And the auto-generated report: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/GFJP5HGSAPHJ4S63D3PYQJ237EDCMBXQ/ Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2022-12/ I uploaded https://rb.zq1.de/compare.factory-20230109/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 14678 (+100) build-tried: 14664 (+99) build-failed: 33 (-12) build-n-a: 225 (-2) build-succeeded: 14405 (+111) build-official-failed+na: 17 (+0) build-compare-failed: 562 (-13) build-compare-succeeded: 13843 (+124) verify-failed: 401 (-18) verified-semi-reproducible: 11672 (-5) bit-by-bit-identical: 13241 (-249) not-bit-by-bit-identical: 1165 (+365) not-bit-by-bit-identicalcheck: 1164 (+360) https://rb.zq1.de/compare.factory-20230109/graph.png shows the change over time https://rb.zq1.de/compare.factory-20230109/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 5 were in ring0 78 were in ring1 That makes it 83/3316 => 2.50 % which is below the overall average of 562/14405 => 3.90 % 1165/14405 => 8.09 % of packages are not perfectly reproducible package notes: ==> ./ansible/.rb.notes <== python3 toolchain pyc ASLR ==> ./apache-commons-dbcp/.rb.notes <== javadoc toolchain ==> ./apache-commons-vfs2/.rb.notes <== javadoc toolchain /usr/share/javadoc/apache-commons-vfs2/deprecated-list.html differs (HTML document, ASCII text) ==> ./aqute-bnd/.rb.notes <== javadoc toolchain html order issues /usr/share/javadoc/aqute-bnd/biz.aQute.bndlib/index-all.html differs (HTML document, ASCII text, with very long lines) ==> ./asymptote/.rb.notes <== => https://github.com/vectorgraphics/asymptote/pull/351 date ; remaining pdf issue ==> ./b4/.rb.notes <== python3 toolchain .pyc ASLR old: parallelism in tar.xz from _service recompress at buildtime ==> ./bin86/.rb.notes <== date 2038-overflow ==> ./bmap-tools/.rb.notes <== FTBFS-2024-06-12 ==> ./borgbackup/.rb.notes <== tar/gz? = https://github.com/borgbackup/borg/issues/6996 FTBFS-2038 ==> ./checkstyle/.rb.notes <== date+time ; xmvn random uuid +++ new//usr/share/java/checkstylecompilation.properties 2020-03-09 23:52:37.968994653 +0000 ==> ./dwz/.rb.notes <== => 1036953 PGO+parallelism only in ./dwz PGO training call even with taskset 1 ==> ./dxvk/.rb.notes <== ?timestamp? in dlls ==> ./elixir/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1205134 FTBFS-j1 stuck osc build --vm-type=kvm --clean --noservice -j1 standard ==> ./endeavour/.rb.notes <== = https://gitlab.gnome.org/World/Endeavour/-/issues/503 report FTBFS-j1 osc build --vm-type=kvm -j1 --clean --noservice standard ==> ./form/.rb.notes <== FTBFS-j1 osc build --vm-type=kvm -j1 --noservice standard ==> ./himalaya/.rb.notes <== random | rust +++ new /usr/bin/himalaya (objdump) ==> ./icmake/.rb.notes <== => https://gitlab.com/fbb-git/icmake/-/merge_requests/5 sort find readdir order from ==> ./indriya/.rb.notes <== java date+time +++ new//usr/share/java/indriya/META-INF/MANIFEST.MF 2020-03-09 19:59:46.201491946 +0000 ==> ./java-17-openj9/.rb.notes <== Java - many issues /usr/lib64/jvm/java-17-openj9-17/lib/OMRTraceFormat.dat differs ==> ./java-18-openjdk/.rb.notes <== Java - many issues /usr/lib64/jvm/java-18-openjdk-18/lib/server/classes_nocoops.jsa differs at offset '5' (data) ==> ./ldap-sdk/.rb.notes <== javadoc toolchain ==> ./libarchive/.rb.notes <== = https://github.com/libarchive/libarchive/issues/1837 FTBFS-2038 => https://github.com/libarchive/libarchive/pull/1836 FTBFS-2038 ==> ./libnvme/.rb.notes <== => SR 1032570 date +++ new//usr/share/man/man2/nvmf_update_config.2 2022-09-01 12:23:50.953451512 +0000 ==> ./libspf2/.rb.notes <== doxygen toolchain ==> ./llmnrd/.rb.notes <== => SR 1037221 minor filesys or toolchain ==> ./morfologik-stemming/.rb.notes <== xmvn random uuid, date+time +++ new//usr/share/java/morfologik-stemming/morfologik-fsa-builders.jar/META-INF/maven/org.carrot2/morfologik-fsa-builders/pom.properties 2022-12-10 14:42:44.247061412 +0000 ==> ./muparserx/.rb.notes <== ASLR into /usr/share/doc/packages/muparserx-devel/html/inherit_graph_3.png ==> ./mybatis/.rb.notes <== java/maven filesys strip-nd, date+time /usr/share/java/mybatis.jar/META-INF/MANIFEST.MF differs (ASCII text, with CRLF line terminators) ==> ./nim/.rb.notes <== -j1 failed ==> ./opennlp/.rb.notes <== maven jar strip-nd , date+time /usr/share/java/opennlp/opennlp-tools.jar/META-INF/maven/org.apache.opennlp/opennlp-tools/pom.properties differs (ASCII text) ==> ./OpenRGB/.rb.notes <== => https://gitlab.com/CalcProgrammer1/OpenRGB/-/merge_requests/1567 filesys readdir order ==> ./opensaml/.rb.notes <== doxygen-1.9.5 toolchain readdir order ==> ./openscap/.rb.notes <== = https://github.com/OpenSCAP/openscap/issues/1909 FTBFS => https://github.com/OpenSCAP/openscap/commit/8dec1bb5e9546e75ae6e7b7cf94cf00197ce3e5d ==> ./ouch/.rb.notes <== rust libzstd_sys read_dir order => https://github.com/gyscos/zstd-rs/commit/6fe67d6a ==> ./pdsh/.rb.notes <== minor rpm metadata filterdiff printrpmtags RPMS*/*src.rpm|grep ^- ==> ./perl-HTTP-Cookies/.rb.notes <== => https://github.com/libwww-perl/HTTP-Cookies/pull/72 FTBFS-2038 ==> ./python-aiosmtplib/.rb.notes <== FTBFS-2038 SSL found range good=2145916633 bad=2145918558 (2038-01-01T00:29:18) ==> ./python-arrow/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-asyncpg/.rb.notes <== found range good=1956864895 bad=1956866820 (2032-01-04T22:07:00) osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-05-26T18:00:00" standard # Fails ==> ./python-azure-storage-file-share/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-bandit/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-devpi-server/.rb.notes <== = https://github.com/devpi/devpi/issues/948 FTBFS -j1 failed osc build --clean --vm-type=kvm -j1 --noservice standard ==> ./python-enaml/.rb.notes <== => https://github.com/nucleic/enaml/pull/516 FTBFS-2038 osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-02-01T18:00:00" standard ==> ./python-Faker/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-fsspec/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-gspread/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-gwpy/.rb.notes <== = failing from astropy leap second expiry warning https://github.com/astropy/astropy/issues/10228 --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed ==> ./python-holoviews/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-maturin/.rb.notes <== #= https://bugzilla.opensuse.org/show_bug.cgi?id=1206342 report rust random 8 random bytes from vendor/ahash-0.3.8/src/random_state.rs into /usr/bin/maturin-3.8 differs in ELF sections ==> ./python-mautrix/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-mitmproxy/.rb.notes <== => https://github.com/mitmproxy/mitmproxy/pull/5852 FTBFS-2037 ==> ./python-notebook/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-onionshare/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-pip/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-pre-commit/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-psutil/.rb.notes <== ASLR python ; stuck -j4? ==> ./python-pycryptodomex/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-pycsw/.rb.notes <== --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed found range good=1673068535 bad=1673070459 (2023-01-07T05:47:39) ==> ./python-PyInstaller/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-pypandoc/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-pyprimes/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-pytzdata/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-tablib/.rb.notes <== python3 .pyc ASLR ==> ./python-tasklib/.rb.notes <== --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed ==> ./python-Telethon/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-toolz/.rb.notes <== python3 pyc toolchain ASLR ==> ./python-trio/.rb.notes <== python3 pyc toolchain ASLR ==> ./qca/.rb.notes <== FTBFS-2031 found range good=1672719473 bad=1672721397 (2023-01-03T04:49:57) ==> ./qemu/.rb.notes <== /usr/share/qemu/slof.bin differs at offset '108' (data) ==> ./qt6-positioning/.rb.notes <== filesystem+ASLR+parallelism /usr/lib64/qt6/examples/positioning/weatherinfo/weatherinfo differs in assembler output ==> ./relaxngcc/.rb.notes <== minor strip-nondeterminism -/usr/share/javadoc/relaxngcc/relaxngcc/util/ConcatIterator.html ==> ./rnp/.rb.notes <== = https://github.com/rnpgp/rnp/issues/1961 FTBFS-2024-01-19 from src/lib/sec_profile.cpp SHA-1 depecation #found range good=1705629262 bad=1705631160 (2024-01-19T02:26:00) ==> ./rpm-config-SUSE/.rb.notes <== date+time ; acceptable variation? /usr/lib/rpm/macros.d/macros.opensuse differs (ASCII text) ==> ./rss2email/.rb.notes <== python3 toolchain .pyc ASLR ==> ./sbsigntools/.rb.notes <== filesys readdir ordering from automake in /usr/bin/sbvarsign ==> ./sccache/.rb.notes <== => https://github.com/gyscos/zstd-rs/commit/6fe67d6a toolchain filesys readdir order ; needs release ==> ./shibboleth-sp/.rb.notes <== doxygen-1.9.5 toolchain filesys readdir order ==> ./skaffold/.rb.notes <== parallelism issue solved 20221108 with go 1.19.2 ==> ./spotifyd/.rb.notes <== random ==> ./spyder/.rb.notes <== python3 toolchain pyc ASLR ==> ./syft/.rb.notes <== go1.18 parallelism build-id still problematic with go1.19.4 ==> ./telegraf/.rb.notes <== go parallelism .note.go.buildid: ==> ./teleport/.rb.notes <== go parallelism .note.go.buildid: ==> ./texlab/.rb.notes <== ASLR rust ==> ./thonny/.rb.notes <== python3 toolchain .pyc ASLR ==> ./trivy/.rb.notes <== parallelism go1.18 - solved with go1.19.3? ==> ./uimaj/.rb.notes <== xmvn random uuid ; date+time +++ new//usr/lib64/java/uimaj/uimaj-core.jar/META-INF/MANIFEST.MF 2022-12-10 16:52:06.762983827 +0000 ==> ./unit-api/.rb.notes <== javadoc jar html filesys, strip-nd, date+time +++ new//usr/share/java/unit-api.jar/META-INF/maven/javax.measure/unit-api/pom.properties 2022-12-10 17:06:16.577151499 +0000 ==> ./uom-lib/.rb.notes <== xmvn uuid, date+time /usr/share/java/uom-lib/uom-lib-common.jar/META-INF/maven/tech.uom.lib/uom-lib-common/pom.properties differs (ASCII text) ==> ./vectorscan/.rb.notes <== => SR 1032506 CPU -march= /usr/bin/unit-hyperscan differs in assembler output ==> ./waf/.rb.notes <== time-based .pyc ==> ./warzone2100/.rb.notes <== = https://github.com/Warzone2100/warzone2100/issues/2991 report parallelism-dependent output - Compression succeeded to file "/home/abuild/rpmbuild/BUILD/warzone2100/build/data/base/texpages/page-102-bottom-rockies.ktx2" size 1375248 bytes in 144.393 secs ==> ./weblate/.rb.notes <== = https://github.com/WeblateOrg/weblate/issues/8556 FTBFS-2038 toolchain mercurial ERROR: test_upstream_changes_rename (weblate.vcs.tests.test_vcs.VCSHgTest) ==> ./xemacs-packages/.rb.notes <== FTBFS-2038 stuck osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-12-10T00:59:10" standard ==> ./xemacs/.rb.notes <== FTBFS-2038 stuck osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-12-10T00:59:10" standard ==> ./xmltooling/.rb.notes <== doxygen toolchain filesys readdir order