# verification builds done:
grep "verified\" : [0-9]" reproducible-newdeps.json | wc -l
12235

# some were skipped for good reasons:
grep "verified\" : null" reproducible-newdeps.json | wc -l
53

# How many had major diffs:
wc -l verificationlist.txt 
801
grep "verified\" : [0]" reproducible-newdeps.json | wc -l
796

# After subtracting those known to be bad in Factory:
diff ~/rb/compare.factory/build-compare-differed-builds.txt verificationlist.txt |grep "^> "|wc -l
522

# Still plenty java stuff in there

# Some varied from a checkin of linux-glibc-devel shortly before release without rebuilding everything with it.
# Rebuilding them with linux-glibc-devel-4.15 made them verifiable - e.g. systemd
91 of them

jsonresultmerge $(cat .verificationlist) > .reproducible-verification.json
grep -3 '"status" : "reproducible"' .reproducible-verification.json|grep '"package"'|wc -l
230
# 230 packages could in theory build reproducibly, but didnt verify.
grep -7 '_status" : "reproducible"' .reproducible-verification.json|grep '"package"'|wc -l
116
# and some more that should pass build-compare

# 15 kmps like xtables-addons varied from obs-pesign and kernel updates
grep -l "^can't open binaries.*lp152.18" */binaries.nachbau/*out | cut -d/ -f1|wc
15
bbswitch crash dpdk drbd hdjmod lttng-modules mhvtl openafs pcfclock rtl8812au sysdig v4l2loopback vhba-kmp virtualbox xtables-addons
# some others differed because they were built with different versions of BuildRequires
# especially those 18 that caputure the kernel version:
grep -l 5.3.18-lp152.1[78] */binaries.nachbau/*out | tee .kernel-uname
cut -d/ -f1 .kernel-uname|sort -u
hdf jboss-logging jboss-logmanager jboss-marshalling jboss-modules jboss-websocket-1.0-api katacontainers-image-initrd libpt2 minidlna paraview wxWidgets-3_0 yast2-sound

rbfindverifiable | tee .verifiable
wc -l .verifiable 
239

most of these 239 probably would be verifiable if we used the older 15.2 RC binaries that were originally used to build them, but they are not kept on the servers.



Summary:
For openSUSE:Leap:15.2
12235 package verification builds were done,
of those, 796 failed initial verification with build-compare [1].
I did local double-builds of these 796 and found that only 346 of them
could produce the same build results twice.
15 were kmps that suffered from an issue with our OBS pesigning integration.
91 became reproducible when building with the older linux-glibc-devel-4.15
that was used for building the official binaries.

That left 239 verifiable packages that could not be verified.
At least 18 contained a previous kernel version string
and we usually dont do automatic rebuilds for kernel updates
and disabled full rebuilds during the last stage of 15.2 development.


[1] background reading for why verification does not give bit-identical results (yet):
https://www.suse.com/c/extending-trust-in-our-binaries-no-backdoors-have-been-found/