Coverage fails on linux with large java libraries (many classes)

[bendavini]

Description of the bug:

For large java libraries (those with a large number of classes - unclear if jar size also relevant), bazel coverage fails with FileAlreadyExistsException. See below. The tests themselves run fine:

bazel coverage :petstore-tests
...
ERROR: /home/vmware/code/git/bd007141-sandbox/bazel/issues/coverage-FileAlreadyExistsException/BUILD.bazel:24:13: Building libpetstore-library.jar (1 source file, 1 source jar) failed: (Exit 1): java failed: error executing Javac command (from target //:petstore-library)
  (cd /home/vmware/.cache/bazel/_bazel_vmware/1a134ebb0e3eb1ab84f7e5bc8cbdaac7/execroot/_main && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
    PATH=/bin:/usr/bin:/usr/local/bin \
  external/rules_java~~toolchains~remotejdk21_linux/bin/java '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.resources=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' '-Dsun.io.useCanonCaches=false' -XX:-CompactStrings -Xlog:disable '-Xlog:all=warning:stderr:uptime,level,tags' -jar external/rules_java~~toolchains~remote_java_tools/java_tools/JavaBuilder_deploy.jar @bazel-out/k8-fastbuild/bin/libpetstore-library.jar-0.params @bazel-out/k8-fastbuild/bin/libpetstore-library.jar-1.params)
# Configuration: 25c6b61d2b490f64bacfcbd8e3b48657aada7aa131f7b07e2420d917097d6690
# Execution platform: @@platforms//host:host
java.nio.file.FileAlreadyExistsException: bazel-out/k8-fastbuild/bin/bazel-out/k8-fastbuild/bin/libpetstore-library.jar-coverage-metadata/com/foo/api/Pets0Api$1.class.uninstrumented
        at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:906)
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
        at java.base/java.nio.file.Files.move(Files.java:1430)
        at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$1.visitFile(JacocoInstrumentationProcessor.java:136)
        at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$1.visitFile(JacocoInstrumentationProcessor.java:113)
        at java.base/java.nio.file.Files.walkFileTree(Files.java:2786)
        at java.base/java.nio.file.Files.walkFileTree(Files.java:2857)
        at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.instrumentRecursively(JacocoInstrumentationProcessor.java:111)
        at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.processRequest(JacocoInstrumentationProcessor.java:85)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:151)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:120)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.build(BazelJavaBuilder.java:111)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:91)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.lambda$main$0(BazelJavaBuilder.java:52)
        at com.google.devtools.build.lib.worker.WorkRequestHandler$WorkRequestCallback.apply(WorkRequestHandler.java:252)
        at com.google.devtools.build.lib.worker.WorkRequestHandler.respondToRequest(WorkRequestHandler.java:480)
        at com.google.devtools.build.lib.worker.WorkRequestHandler.lambda$startResponseThread$1(WorkRequestHandler.java:433)
        at java.base/java.lang.Thread.run(Thread.java:1583)Target //:petstore-tests failed to build
INFO: Elapsed time: 2.005s, Critical Path: 1.94s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully
//:petstore-tests                                               FAILED TO BUILD

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

• Linux (tested on Rocky 9.4 x86) (This does not repro on mac x86)

• extract attached archive
  bazel-coverage-FileAlreadyExistsException.tar.gz

• generate a swagger spec with 250 unique tags/paths: python spec_generator.py 250

• run bazel coverage :petstore-tests (this will fail)

• generate a swagger spec with 200 unique tags/paths: python spec_generator.py 200

• run bazel coverage :petstore-tests (this will succeed)

Which operating system are you running Bazel on?

Rocky 9.4 x86

What is the output of bazel info release?

release 7.4.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

This was reported by someone else on slack

Any other information, logs, or outputs that you want to share?

Reproduced with 7.4.1, 7.4.0, 6.5.0, 6.4.0

[fmeum]

I went up to 10000, but couldn't reproduce the failure on Linux. I also can't reproduce it on macOS.

@cushon I don't fully understand who is responsible for cleaning up the coverage metadata dir, especially with non-sandboxed execution. Can you shed some light on this? Is the old coverage implementation still used?

[cushon]

cc @hvadehra

@cushon I don't fully understand who is responsible for cleaning up the coverage metadata dir, especially with non-sandboxed execution. Can you shed some light on this?

JavaBuilder generally tries to clean up its own temp directories, like the logic in JacocoInstrumentationProcessor#cleanup. I could believe that there was an issue where that wasn't happening that was getting papered over by remote execution, though.

Is the old coverage implementation still used?

I'm not sure that it is, let's find out.

[bendavini]

I have NOT been able to reproduce this in an ubuntu:24.04 and rockylinux/rockylinux:9.4 container (x86 on Rancher Desktop on Mac). The file systems in this case appear to be ext4. This can be reproduced with a rockylinux 9.4 x86 VM on XFS as well as an ubuntu container with an overlay backed by XFS. I'm sure there are other differences; I'll continue to try to identify a (ideally) container-based reproduction.

[bendavini]

This is reproducible in a Rocky 9.5 VM (running in VirtualBox via vagrant on an x86 Mac). Since upstream rocky vagrant links are broken, I manually downloaded a rocky 9.5 box, added the box: vagrant box add rocky-95 Rocky-9-Vagrant-Vbox-9.5-20241118.0.x86_64.box

then, vagrant up with the following Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "rocky-95"
  config.vm.provider "virtualbox" do |v|
    v.memory = 8192
    v.cpus = 4
  end
end

• login to the box with vagrant ssh
• Download the dependencies:

sudo dnf install wget python3 gcc java-11-openjdk -y
wget https://github.com/bazelbuild/bazelisk/releases/download/v1.19.0/bazelisk-linux-amd64 && chmod +x bazelisk-linux-amd64
wget https://github.com/user-attachments/files/18325184/bazel-coverage-FileAlreadyExistsException.tar.gz
tar xvf bazel-coverage-FileAlreadyExistsException.tar.gz
cd coverage-FileAlreadyExistsException/

• Run the failing test: python3 spec_generator.py 250 && ../bazelisk-linux-amd64 coverage :petstore-tests

Same error as above is seen