fix: prevent buildkit corruption on sticky disk commit

[adityamaru]

Summary

• Adds graceful shutdown validation to prevent committing corrupted buildkit state
• Implements multiple sync operations to ensure database writes are flushed
• Adds state validation before committing sticky disk

Problem

We've been seeing buildkit database corruption issues manifesting as:

1. BoltDB panic: "assertion failed: Page expected to be: 188, but self identifies as 0"
2. Overlayfs errors: "failed to rename: file exists"

These indicate buildkitd isn't properly flushing its database writes before the Ceph volume is committed.

Solution

1. Graceful Shutdown Enforcement: Fail if buildkitd doesn't shutdown cleanly within timeout
2. Critical Sync After Shutdown: Add sync immediately after buildkitd terminates to flush all database writes
3. State Validation: Validate buildkit state before committing (no processes, no lock files, non-zero db files)
4. No Commit on Failure: Don't commit sticky disk if build fails or validation fails
5. Multiple Syncs: Add multiple sync operations before unmounting

Changes

• Modified shutdownBuildkitd() to track graceful shutdown and fail if timeout
• Added validateBuildkitState() to check for corruption indicators
• Added sync operations at critical points
• Prevention of sticky disk commit on any failure condition

Testing

• Test with normal successful builds
• Test with builds that fail
• Test with buildkitd that doesn't shutdown gracefully
• Monitor for corruption issues in staging

🤖 Generated with Claude Code