Hello from me, who did 500+ patches for reproducible builds since 2016.
We collected many good reasons for r-b in https://reproducible-builds.org/docs/buy-in/ and you might notice that several are not related to security. For example in openSUSE we save significant build power and network bandwidth of mirrors and users by build-tree-pruning. That is impossible if two builds from one source always differ.
I also found many bugs with r-b and fixing those certainly improved quality. Common bugs were build-time race conditions such as in intltool , libxslt/libxml2 , usage of -march=native , segmentation faults that were silently ignored and such. Also found dozens instances of software that stops to build or work in the future (1, 2, 3, 4)Even if non-deterministic software behaviour is not a bug, don't we all prefer software that does its work in a deterministic way as a computer is supposed to? We just want the same inputs to produce the same outputs.
Another point you make is that there are other attack vectors and this is true. Source-level issues are just out-of-scope for r-b. Other methods have to be applied to fix these other issues.IMHO, r-b is just the next logical step for open source software, where you do not have to worry about leaking your company secrets and can let as many parties rebuild as they like. Users do not even have to trust all of them.
Even if there is one trusted main build infrastructure, having those independent rebuilders helps to detect and deter intruders by shrinking their reward for an attack on build machines.These are my 10c