So now we know that we have two Java actions running consecutively, and whatever the former does impacts the behavior of the latter. And, sure enough, for each incremental build, Bazel was only able to run two Java actions: the first successfully and the second crashing.Īlright. To confirm this, I ran the build with -jobs=1, waited until it failed, and then ran it again a few times. Thinking more about the problem, the only logical conclusion was that a preceding action was at fault, and that its effects were causing the JVM to fail later. Disabling workers made no difference, which makes sense because they are not sandboxed by default anyway… due to workers)… but that theory did not hold either. Then I thought that we might have a race between the sandbox destruction and the JVM process staying behind (e.g. But this was easy to rule out: running with a previous sandboxfs release didn’t make a difference. Maybe the unmap operations of an action were affecting the sandbox of another action, making files disappear from under them. My first thought was that my recent changes to the reconfiguration protocol in sandboxfs were at fault. When we pass the debugging flag, however, Bazel leaves all files and the sandboxfs instance behind so we can navigate into the directory ( /private/var/tmp/_bazel_jmmv/0db9681485d75d94d76b2f9336cd9468/sandbox/sandboxfs/2 from the logs above) and run the failing command by hand (everything after exec-but not including exec itself to avoid exiting the shell).īut, surprise! Passing -sandbox_debug makes this specific problem disappear and allows the build to complete. By default, Bazel deletes the sandbox directories immediately after executing an action and also unmounts the sandboxfs instance when the build finishes. The first step in troubleshooting any sandboxing-related failure, be it with sandboxfs or not, is to pass -sandbox_debug to the Bazel invocation. Combined with the lack of sources for many of the pieces involved and the parallelism introduced by Bazel, we face an interesting reverse-engineering process. We start with a crash of an arbitrary program and we have to figure out how sandboxfs, or who knows what else in the macOS black box, might have caused it. Initial debugging stepsĭebugging sandboxfs-related problems is always “fun”. Wait, what? The JVM crashed? And there are no core dumps under /cores even when I have startup -unlimit_coredumps in my ~/.bazelrc? Yikes.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |