Hi, I did some more patches and kept the build clean. Also sent some existing fixes towards SLFO, so that they can still reach Leap 16.0 and later 16.1 Finally found the problem with cosmic-related packages. It was from cargo-i18n / i18n-embed-fl that already had a fix, that is still missing in our vendor.tar. Additionally I did another test of the games repo and submitted fixes for 4 of the 9 issues found. If we keep improving at this pace, we could be done in 2029. That was what I thought mid-November. And then something happened in ghc that fixed a lot of packages at the same time. And https://qt-project.atlassian.net/browse/QTBUG-137440 also saw some movement, though we already applied workarounds to evade that problem. Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/HGMTV66UILWPYJ56ULYF3GOFV6A4KH5C/ Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2025-10/ I uploaded https://rb.zq1.de/compare.factory-20251129/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 16183 (+46) build-tried: 16111 (+47) build-failed: 51 (-20) build-n-a: 162 (+0) build-succeeded: 15898 (+67) build-official-failed+na: 46 (+1) build-compare-failed: 150 (-4) build-compare-succeeded: 15748 (+71) verify-failed: 636 (-38) verified-semi-reproducible: 15204 (+85) verified-bit-identical: 15008 (+85) bit-by-bit-identical: 15706 (+96) not-bit-by-bit-identical: 191 (-30) not-bit-by-bit-identicalcheck: 192 (-29) https://rb.zq1.de/compare.factory-20251129/graph.png shows the change over time https://rb.zq1.de/compare.factory-20251129/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 1 were in ring0 19 were in ring1 That makes it 20/4043 => 0.49 % which is below the overall average of 150/15898 => 0.94 % 191/15898 => 1.20 % of packages are not perfectly reproducible package notes: ==> ./Fragments/.rb.notes <== #=> SR 1315505 got glib-macros rust random HashMap => see identity ==> ./aws-lc/.rb.notes <== FTBFS-j1 ==> ./bat-extras/.rb.notes <== [ 25s] | Test Suite: batgrep | [ 25s] -------------------------------------------------------------------------------- ==> ./clamav/.rb.notes <== => SR 1317951 = https://bugzilla.opensuse.org/show_bug.cgi?id=1249404 rust1.88 toolchain? ==> ./cockpit/.rb.notes <== FTBFS-date => https://github.com/cockpit-project/cockpit/pull/21460 readdir ==> ./contrast/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1253488 => See identity ; https://github.com/gtk-rs/gtk-rs-core/pull/1840 ==> ./cosmic-edit/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1254147 => https://github.com/kellpossible/cargo-i18n/pull/151 i18n-embed-fl HashMap ? = https://github.com/pop-os/cosmic-edit/issues/221 hash order issue, rust ; ?from GCC_except_table0 ==> ./cosmic-ext-color-picker/.rb.notes <== cosmic order issue => cargo-i18n / i18n-embed-fl ==> ./cosmic-term/.rb.notes <== cosmic order issue => cargo-i18n / i18n-embed-fl ==> ./dealers-choice/.rb.notes <== => SR 1317313 dealers-choice nocheck ==> ./deno/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1252955 deno rust order -ext:deno_node/_fs/_fs_lutimes.ts ==> ./erlang27/.rb.notes <== see erlang ==> ./eww/.rb.notes <== = https://github.com/elkowar/eww/issues/1334 rust1.76/llvm ==> ./fractal/.rb.notes <== #=> SR 1315008 => See identity ; rust HashMap symbol order variation in glib-macros ==> ./freeciv/.rb.notes <== regression from https://build.opensuse.org/requests/1299973 from system lua? ==> ./fritzing/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1239967 fritzing random parts.db ==> ./ghc-bootstrap/.rb.notes <== FTBFS-CPU [ 83s] /home/abuild/rpmbuild/BUILD/ghc-bootstrap-9.10.1-build/BUILDROOT/opt/lib/ghc-9.10.1/bin/ghc-pkg: CPU ISA level is lower than required ==> ./gnutls/.rb.notes <== FTBFS-CPU building with cpu=Nehalem failed ; SandyBridge works ==> ./gstreamer-plugins-rs/.rb.notes <== => SR 1314943 = https://bugzilla.opensuse.org/show_bug.cgi?id=1237097 = https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/599 date ; rust/llvm asm diff [build] jobs = 1 doesn't help ==> ./helm/.rb.notes <== FTBFS-2028-04-05 years=1 multibuildrbkall works ==> ./helm3/.rb.notes <== FTBFS-20xx - see helm ==> ./ibus/.rb.notes <== => https://github.com/ibus/ibus/pull/2821 by raboof => SR 868399 ==> ./identity/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1253488 => https://github.com/gtk-rs/gtk-rs-core/pull/1840 rust HashMap ==> ./intel-ipsec-mb/.rb.notes <== FTBFS-CPU cpu=Conroe ==> ./intel-opencl/.rb.notes <== ASLR order variations: ==> ./java-25-openj9/.rb.notes <== date+mtime + other? ; similar to previous versions date in /usr/lib64/jvm/java-25-openj9-25/lib/java_ca.properties ; other in /usr/lib64/jvm/java-25-openj9-25/jmods/java.base.jmod ==> ./kwctl/.rb.notes <== minor rust order debuginfo only .note.gnu.build-id ==> ./lalburst/.rb.notes <== FTBFS-j1 ==> ./libplasma6/.rb.notes <== => SR 1319234 1318951 qmlcachegen parallelism/race variation in /usr/lib64/qt6/qml/org/kde/plasma/extras/libplasmaextracomponentsplugin.so from /home/abuild/rpmbuild/BUILD/libplasma-6.2.2/build/src/declarativeimports/plasmaextracomponents/.rcc/qmlcache/plasmaextracomponentsplugin_qml/PlasmoidHeading_qml.cpp ==> ./loupe/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1253488 glib-macros rust HashMap => see identity ?low-entropy? or not a problem anymore? ==> ./messagelib/.rb.notes <== parallelism race in /usr/share/doc/qt6/KPim6MessageViewer.qch - toolchain? ==> ./mousai/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1253488 => see `identity` ==> ./nautilus/.rb.notes <== FTBFS-j1 ==> ./octave-forge-tisean/.rb.notes <== FTBFS ==> ./pkcs11-provider/.rb.notes <== FTBFS-CPU? [ 274s] 66/78 pkcs11-provider:softhsm / tls FAIL 24.97s exit status 1 ==> ./plasma6-nm/.rb.notes <== => SR 1313806 plasma6-nm qml rcc cachegen ==> ./plasma6-print-manager/.rb.notes <== => SR 1313805 plasma6-print-manager qml cachegen ==> ./python-PyMuPDF/.rb.notes <== unknown issue in .a file / RECORD ==> ./python-ZEO/.rb.notes <== FTBFS-j1 + FTBFS-2038 found range 2147460804 -> 2147462729 ==> ./python-cfn-lint/.rb.notes <== #FTBFS-2025-11-01 found range good=1761954100 bad=1761956025 - fixed in version 1.40.4 #= https://github.com/aws-cloudformation/cfn-lint/issues/4125 FTBFS-2025-09-01 ==> ./python-falcon/.rb.notes <== => SR 1317294 => https://github.com/falconry/falcon/pull/2568 by Vytautas Liuolia = https://github.com/falconry/falcon/issues/2567 date date in /usr/share/doc/packages/python-falcon-doc/html/community/releases.html ==> ./python-orjson/.rb.notes <== #solved in 3.10.7 machine code diffs from rustc+gcc13 ==> ./python-oslo.privsep/.rb.notes <== FTBFS ==> ./python-oslo.versionedobjects/.rb.notes <== FTBFS sphinx date in html ==> ./python-proton-vpn-network-manager-openvpn/.rb.notes <== FTBFS 20250605 ==> ./python-pyrage/.rb.notes <== rust/llvm random / LTO same problem as in rage-encryption => https://github.com/kellpossible/cargo-i18n/pull/151 ==> ./python-spyder-notebook/.rb.notes <== FTBFS: spyder_notebook/tests/test_plugin.py::test_on_mainwindow_visible_with_opened_notebooks PASSED [ 1%]Fatal Python error: Segmentation fault #= https://bugzilla.opensuse.org/show_bug.cgi?id=1228441 report FTBFS ==> ./python-uproot/.rb.notes <== FTBFS-CPU checks --build-opt=--vm-custom-opt=-cpu=Conroe failed ==> ./python312/.rb.notes <== FTBFS-2028-08-26 test_ssl found range good=1850910434 bad=1850912359 ==> ./python315/.rb.notes <== nogil ==> ./qgis/.rb.notes <== parallelism in .note.gnu.build-id ==> ./qt6-base/.rb.notes <== CPU+ASLR or parallelism? and date? 0 0 0 0 0 0 0 1 1 0 0 1 ==> ./quick-webapps/.rb.notes <== see cosmic issue ==> ./resources/.rb.notes <== unknown rust/llvm LTO does not become reproducible with lto="off" ==> ./rnp/.rb.notes <== = https://github.com/rnpgp/rnp/issues/2371 FTBFS-date 2041-05-13? found range good=2252068601 bad=2252070526 ==> ./rocclr/.rb.notes <== => https://src.opensuse.org/ROCmWork/rocclr/pulls/1 PID ==> ./rocfft/.rb.notes <== ASLR order ==> ./rocsolver/.rb.notes <== => https://src.opensuse.org/ROCm/rocsolver/pulls/1 FTBFS-j1 unreproducible ==> ./switcheroo/.rb.notes <== = https://gitlab.com/adhami3310/Switcheroo/-/issues/86 FTBFS-j1 ==> ./xfishtank/.rb.notes <== => SR 1314685 1314682 xfishtank date, regression ==> ./zola/.rb.notes <== rust/llvm random order .rs generation from HashMap in vendor/minify-html-common-0.0.2/build.rs