Remote caching unstable after upgrading to Bazel 8.0.0

[armandomontanez]

Description of the bug:

When moving to Bazel 8.0.0, remote caching for Pigweed became unreliable, frequently emitting errors like the following:

ERROR: /usr/local/google/home/amontanez/development/projects/pigweed/pigweed/targets/rp2040/BUILD.bazel:247:14: Creating runfiles tree bazel-out/k8-fastbuild/bin/targets/rp2040/rp2350_system_async_example.runfiles failed: java.io.FileNotFoundException: /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/06cb1b6ef37c7adbbc0068a64c52919c/execroot/_main/bazel-out/k8-fastbuild/bin/targets/rp2040/rp2350_system_async_example.runfiles/_main/external/pico-sdk+/src/rp2350/boot_stage2 (No such file or directory)

Turning off --experimental_inprocess_symlink_creation works around this.

This seems to occur the most frequently around the boundaries of transitions that propagate runfiles, though that may be anecdotal. I was able to prevent some occurrences by excluding runfiles from being passed across the boundary of the transition.

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I can reproduce locally by:

1. Removing common:remote_cache --experimental_inprocess_symlink_creation=false from Pigweed's .bazelrc.
2. Run bazel test --config=remote_cache //...
3. Run bazel test --config=remote_cache --config=googletest //...

Note that for whatever reason the first bazel test does not reproduce, but switching configurations slightly does.

Which operating system are you running Bazel on?

linux

What is the output of bazel info release?

release 8.0.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@bazel-io flag 8.0.1

[fmeum]

@tjgq

[iancha1992]

@bazel-io fork 8.0.1

[UebelAndre]

I believe this is the root cause for this failed test in rules_rust when migrating to Bazel 8.0.0 (bazelbuild/rules_rust#3077)
https://buildkite.com/bazel/rules-rust-rustlang/builds/13675#0194496d-7d1d-4ac5-8559-5eb8b1038aca

(04:59:40) ERROR: /workdir/test/unpretty/BUILD.bazel:45:10: Creating runfiles tree bazel-out/k8-fastbuild/bin/test/unpretty/proc_macro_test_unpretty_diff_test-test.sh.runfiles failed: java.io.FileNotFoundException: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/execroot/_main/bazel-out/k8-fastbuild/bin/test/unpretty/proc_macro_test_unpretty_diff_test-test.sh.runfiles/_main/test/unpretty (No such file or directory)
(04:59:40) ERROR: /workdir/test/unpretty/BUILD.bazel:18:10: Creating runfiles tree bazel-out/k8-fastbuild/bin/test/unpretty/proc_macro_unpretty_diff_test-test.sh.runfiles failed: java.io.FileNotFoundException: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/execroot/_main/bazel-out/k8-fastbuild/bin/test/unpretty/proc_macro_unpretty_diff_test-test.sh.runfiles/_main/test/unpretty (No such file or directory)

[fmeum]

Could you rerun with --verbose_failures, which should give us a stack trace?

Nvm, that also doesn't show the extension. I can reproduce on pigweed though.

[fmeum]

Stack trace for pigweed:

java.io.FileNotFoundException: /private/var/tmp/_bazel_fmeum/c9f560a675e76a01d47db885533bcde8/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/pw_system/rp2350_system_example.runfiles/_main/external/pico-sdk+/src/rp2350/boot_stage2 (No such file or directory)
	at com.google.devtools.build.lib.vfs.inmemoryfs.InMemoryFileSystem$Errno.exception(InMemoryFileSystem.java:145)
	at com.google.devtools.build.lib.vfs.inmemoryfs.InMemoryFileSystem$Errno.inodeOrThrow(InMemoryFileSystem.java:126)
	at com.google.devtools.build.lib.vfs.inmemoryfs.InMemoryFileSystem.getDirectory(InMemoryFileSystem.java:340)
	at com.google.devtools.build.lib.vfs.inmemoryfs.InMemoryFileSystem.createSymbolicLink(InMemoryFileSystem.java:531)
	at com.google.devtools.build.lib.vfs.Path.createSymbolicLink(Path.java:481)
	at com.google.devtools.build.lib.remote.RemoteActionFileSystem.createSymbolicLink(RemoteActionFileSystem.java:559)
	at com.google.devtools.build.lib.vfs.Path.createSymbolicLink(Path.java:481)
	at com.google.devtools.build.lib.vfs.FileSystemUtils.ensureSymbolicLink(FileSystemUtils.java:345)
	at com.google.devtools.build.lib.exec.SymlinkTreeHelper$Directory.syncTreeRecursively(SymlinkTreeHelper.java:310)
	at com.google.devtools.build.lib.exec.SymlinkTreeHelper$Directory.syncTreeRecursively(SymlinkTreeHelper.java:317)
	at com.google.devtools.build.lib.exec.SymlinkTreeHelper$Directory.syncTreeRecursively(SymlinkTreeHelper.java:317)
	at com.google.devtools.build.lib.exec.SymlinkTreeHelper$Directory.syncTreeRecursively(SymlinkTreeHelper.java:317)
	at com.google.devtools.build.lib.exec.SymlinkTreeHelper$Directory.syncTreeRecursively(SymlinkTreeHelper.java:317)
	at com.google.devtools.build.lib.exec.SymlinkTreeHelper$Directory.syncTreeRecursively(SymlinkTreeHelper.java:317)
	at com.google.devtools.build.lib.exec.SymlinkTreeHelper$Directory.syncTreeRecursively(SymlinkTreeHelper.java:317)
	at com.google.devtools.build.lib.exec.SymlinkTreeHelper.createSymlinksDirectly(SymlinkTreeHelper.java:139)
	at com.google.devtools.build.lib.exec.SymlinkTreeStrategy.createSymlinks(SymlinkTreeStrategy.java:114)
	at com.google.devtools.build.lib.analysis.actions.SymlinkTreeAction.execute(SymlinkTreeAction.java:230)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1170)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1075)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:166)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:95)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:559)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:928)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:375)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:216)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:467)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

I wasn't expecting to see an in-memory FS in there. Does it make sense to run SymlinkTreeAction against the RemoteActionFileSystem given that the symlinks it creates aren't even tracked as outputs?

Edit: To reproduce, I only had to build //pw_system:rp2350_system_example with and without the config.

[fmeum]

I suspect that this is caused by a fundamental limitation of the RemoteActionFileSystem, which makes it unfit for this purpose: When the runfiles tree logic traverses the runfiles directory, which only exists locally and not in the in-memory file system, it will assume that the parent directories already exist since they do so locally. When it attempts to recreate a symlink, this operation is mirrored to the in-memory file system, which then fails because in that filesystem the parent directory doesn't exist.

[tjgq]

Yeah, I think it's a mistake to use the action filesystem for this purpose. I will write a fix.