Fix your Bootsnap

Recently I came across a bunch of nice agent skills from the great folks over at Evil Martians. One of them, rails-boot-profiling, has your coding agent profile a Rails app’s boot with require-profiler and then fix what it finds. It comes with a long write-up on reading the profiles. I ran it on NeetoChat, and it flagged a few gems we load at boot but don’t need there, like telnyx and rubocop-minitest.

Near the end of its list of fixes, the skill says to enable Bootsnap and keep it updated, and to mind the cache precompilation if you build production images. Bootsnap was already in our Gemfile and required from config/boot.rb, so we assumed it was doing its job everywhere. Nobody had checked production. The bit about production images made me go and look. So take this as your reminder to actually read these kinds of skills. There’s a lot of useful knowledge in them, and you’ll learn more from reading one than from just letting your agent do its thing.

In production, Bootsnap was barely doing anything. Every new pod started with an empty cache, so Bootsnap missed on almost every file and compiled it from scratch. A second boot inside the same container could use what the first boot wrote, but the next pod started empty again.

Once I fixed that, our production web pods went from 10–12 seconds between container start and readiness to 5–6 seconds.

All of this got me curious about what Bootsnap actually does, so I started digging into its codebase. In this post I explain how Bootsnap works, and then what it took to make it work for us in production.

Everything about Bootsnap’s internals here refers to Bootsnap 1.26.1.

What happens during boot?

A full production boot of our app made roughly 12,000 Ruby and YAML compile-cache lookups. They come from three places:

  • config/boot.rb runs require "bundler/setup", which puts the lib/ directory of every gem in your bundle on $LOAD_PATH, the ordered list of directories Ruby searches when you require a name.
  • config/application.rb runs Bundler.require(*Rails.groups), which requires every gem in your default and current-environment groups, except those marked require: false.
  • In production, config.eager_load = true. Rails calls Zeitwerk::Loader.eager_load_all and loads every file in app/, plus every gem whose code is managed by Zeitwerk (Rails guides). In development, constants load lazily, so a development boot loads far fewer files.

What happens when Ruby loads a file?

Consider a single call:

require "active_support"

If Ruby hasn’t loaded that feature yet, it has three things to do.

Find the file. For a bare name like this, MRI, the standard Ruby implementation, walks $LOAD_PATH looking for a match. It first tries active_support.rb in each directory. If none has it, it tries active_support.so (.bundle on macOS) in each directory. Every attempt is one open() system call. So a file in the 200th directory takes 200 attempts, and a name that isn’t anywhere takes two per directory.

Compile it. Ruby parses the source and compiles it into an instruction sequence (ISeq), the bytecode of Ruby’s virtual machine.

Execute it. Loading a file can define classes, require more files, register callbacks, or run other top-level code.

A new process starts from scratch. Once a feature is loaded, requiring it again in that process is a hash lookup.

Two separate requires walking $LOAD_PATH: require "active_support", which exists, and require "nonexistent_file", which doesn't Six load path directories and two separate examples. require "active_support" tries active_support.rb in each directory in order: /app/lib, /gems/rack/lib and /gems/actionpack/lib miss, and /gems/activesupport/lib has it, so it stops after 4 open() calls. require "nonexistent_file" tries nonexistent_file.rb in all six directories, then nonexistent_file.bundle in all six, and raises LoadError after 12 open() calls. open("/app/lib/active_support.rb") open("/gems/rack/lib/active_support.rb") open("/gems/actionpack/lib/active_support.rb") open("/gems/activesupport/lib/active_support.rb") ✓ open("…/nonexistent_file.rb") in every directory ✕ .bundle in every directory too, then LoadError require "active_support" require "nonexistent_file" .rb .rb .bundle /app/lib /gems/rack/lib /gems/actionpack/lib /gems/activesupport/lib /gems/i18n/lib /gems/zeitwerk/lib ✕ ✕ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ open() calls: 1 open() calls: 2 open() calls: 3 open() calls: 4 open() calls: 6 open() calls: 12
Two separate requires, each probe one open() system call. require "active_support" stops at the first .rb it finds. require "nonexistent_file" checks every directory for .rb, then every directory for .bundle, before raising LoadError.

What is Bootsnap?

Bootsnap is a gem from Shopify that caches the expensive parts of loading files. Every new Rails app gets it in its default Gemfile. It has two independent parts:

  • The load-path cache answers “where is active_support.rb?” without walking $LOAD_PATH. It keeps its index in tmp/cache/bootsnap/load-path-cache.
  • The compile cache keeps what Ruby gets out of parsing a file. compile-cache-iseq/ holds compiled Ruby bytecode, and compile-cache-yaml/ holds parsed YAML data, stored as MessagePack (a compact binary format, think JSON but encoded as bytes rather than readable text).
Bootsnap at a glance: the load-path cache and the compile cache require "foo" first reaches Bootsnap's load-path cache, a patched Kernel#require whose index is saved in tmp/cache/bootsnap/load-path-cache. It turns "foo" into the absolute path /gems/foo/lib/foo.rb and hands that to Ruby's own require, which opens one file and asks the compile cache, RubyVM::InstructionSequence.load_iseq, for bytecode stored in compile-cache-iseq/. YAML.load_file also goes through the compile cache, stored in compile-cache-yaml/. The bytecode is then executed, and execution is never cached. require "foo" Load-path cache patched Kernel#require "/gems/foo/lib/foo.rb" Ruby's require opens one file YAML.load_file Compile cache InstructionSequence.load_iseq bytecode execute tmp/cache/bootsnap/ load-path-cache compile-cache-iseq/ compile-cache-yaml/
Bootsnap's two caches (solid) sit in front of Ruby's own loader (dashed). One finds the file, the other skips compiling it. Running the code is never cached.

Turning it on is one line, as early as possible:

# config/boot.rb
require "bundler/setup"
require "bootsnap/setup"

bootsnap/setup reads its settings from environment variables. I’ll explain each one where it comes up.

Inside the load-path cache

Bootsnap puts a wrapper around Ruby’s normal require. It keeps the original around under the name require_without_bootsnap and defines its own require in front of it. So when your app calls:

require "active_support"

Bootsnap gets that call first. Simplified, this is what it does:

def require(path)
  return false if loaded_features_index.key?(path)

  resolved = load_path_cache.find(path)
  if resolved.is_a?(String)
    require_without_bootsnap(resolved) # an absolute path
  else
    require_without_bootsnap(path)     # let Ruby search as usual
  end
end

Step by step:

  1. Check whether it’s already loaded. return false if loaded_features_index.key?(path). Bootsnap keeps its own index of the features this process has already loaded. If active_support is in it, there’s no need to load it again.
  2. Look up its location in Bootsnap’s index. resolved = load_path_cache.find(path) tries to turn "active_support" into an absolute filename, such as /gems/activesupport/lib/active_support.rb.
  3. Give that filename to Ruby’s original require. require_without_bootsnap(resolved). Ruby now knows exactly which file to load, so it doesn’t search $LOAD_PATH. It still has to compile and run the file. That compile step is where the compile cache, covered below, comes in.
  4. Fall back to normal Ruby behavior if needed. require_without_bootsnap(path). If the lookup doesn’t return a filename, Ruby searches as usual. A require that ends in LoadError still walks every directory first, so misses stay slow.

Bootsnap finds the file and Ruby still loads it. This wrapper only handles locating files. Caching the bytecode is the compile cache’s job.

The index behind find is built when Bootsnap starts:

  1. Bootsnap goes through the directories on $LOAD_PATH in order and lists every requirable file in each one, the .rb files and native extensions.
  2. Each file becomes one entry, from its name relative to the directory to the directory itself: "rack/request.rb" → /gems/rack/lib.
  3. find("rack/request") tries "rack/request.rb", then the native-extension name, then the bare name. The first match, joined to its directory, is the absolute path: /gems/rack/lib/rack/request.rb.

For a file in the index, that’s a few hash lookups at most, where Ruby alone would try one directory after another.

Bootsnap scans $LOAD_PATH into the Bootsnap index, then each require is one lookup in it Top part, when Bootsnap starts: three load path directories, /app/lib with billing.rb, /gems/rack/lib with rack.rb and rack/request.rb, and /gems/i18n/lib with i18n.rb. Each file becomes one row of the Bootsnap index: "billing.rb" to /app/lib, "rack.rb" and "rack/request.rb" to /gems/rack/lib, "i18n.rb" to /gems/i18n/lib. Below a divider, on each require: require "rack/request" gets .rb appended, is looked up once in the Bootsnap index, and comes out as /gems/rack/lib/rack/request.rb, a single file to open. $LOAD_PATH Bootsnap index /app/lib billing.rb /gems/rack/lib rack.rb rack/request.rb /gems/i18n/lib i18n.rb name dir "billing.rb" /app/lib "rack.rb" /gems/rack/lib "rack/request.rb" /gems/rack/lib "i18n.rb" /gems/i18n/lib on each require require "rack/request" "rack/request.rb" + .rb, one lookup /gems/rack/lib/rack/request.rb ✓ one file to open
Above the line, Bootsnap scans $LOAD_PATH into the Bootsnap index when it starts. Below it, each require is one lookup in that index instead of a walk down $LOAD_PATH.

Stable and volatile directories

Bootsnap persists the scan in a MessagePack file, tmp/cache/bootsnap/load-path-cache, keyed by Bootsnap version, RUBY_REVISION and RUBY_PLATFORM; a mismatch throws it away. To keep it from going stale, it splits directories into two kinds:

  • Stable directories are under Gem.path, the Bundler bundle path, or Ruby’s own rubylibdir. Installed gems don’t change under a running app, so Bootsnap trusts the stored scan forever.
  • Volatile directories are everything else, which in practice means your application. For these, Bootsnap stores the newest mtime (modification time) among the directory itself and every subdirectory that held a requirable file at the last scan. On boot it recomputes that, and rescans the directory if it changed.

Adding or removing a file changes its parent directory’s mtime, so this catches most new files cheaply.

Inside the compile cache

The load-path cache saves Ruby from searching for a file. The compile cache saves it from parsing and compiling that file once it’s found.

Before parsing and compiling a Ruby file, MRI calls RubyVM::InstructionSequence.load_iseq if that hook is defined. Bootsnap defines it to return a cached instruction sequence when a valid one exists, so MRI skips both steps. When the hook returns nil, MRI compiles the file as usual.

Bootsnap’s version of the hook:

module InstructionSequenceMixin
  def load_iseq(path)
    Bootsnap::CompileCache::ISeq.fetch(path.to_s)
  end
end

class << RubyVM::InstructionSequence
  prepend(InstructionSequenceMixin)
end

fetch hands off to Bootsnap’s C extension. Compiling stores compile_file(path).to_binary, and loading from the cache calls load_from_binary.

The difference is roughly:

Without a compile-cache hit:
  Read source → parse and compile → execute

With a compile-cache hit:
  Read cached instructions → execute

Where an entry lives

Each source file gets one cache file, named by an FNV-1a 64-bit hash of its absolute path. The first byte is the directory, the other seven the file name:

/workspace/app/models/user.rb
        │  FNV-1a 64
        ▼
0x682c3d1cedeba43e
        │
        ▼
tmp/cache/bootsnap/compile-cache-iseq/68/2c3d1cedeba43e

The same active_support.rb installed at two different paths has two unrelated cache entries.

What an entry contains

Every entry starts with a 32-byte header, followed by the cached bytecode or YAML data:

struct bs_cache_key {
  uint64_t ruby_version_digest;
  uint64_t mtime;
  uint64_t digest;
  uint32_t size;
  uint32_t data_size;
} __attribute__((packed));

ruby_version_digest hashes RUBY_DESCRIPTION, Bootsnap’s cache format version and Ruby’s compile options, so a Ruby upgrade invalidates everything. mtime and size describe the source when the entry was written, digest is an FNV-1a hash of its contents, and data_size is the payload length.

Layout of one compile-cache entry: a 32-byte header, then the payload The entry file tmp/cache/bootsnap/compile-cache-iseq/68/2c3d1cedeba43e starts with a 32-byte header: ruby_version_digest (8 bytes), mtime (8 bytes), digest (8 bytes), size (4 bytes) and data_size (4 bytes, the payload length). The payload that follows is the ISeq binary from to_binary, or MessagePack for YAML, and is data_size bytes long. tmp/cache/bootsnap/compile-cache-iseq/68/2c3d1cedeba43e 32-byte header payload, data_size bytes 0 8 16 24 28 32 ISeq binary or MessagePack … 8 B 8 B 8 B 4 B 4 B data_size size digest mtime ruby_version_digest
Every field the hit check compares sits in the first 32 bytes, so Bootsnap can decide without reading the source.

Hit, miss, stale, revalidated

When Ruby asks for a file, Bootsnap reads the source file’s metadata, its size and last-modified time (with the stat system call), and compares it with the stored header:

if (k1->ruby_version_digest == k2->ruby_version_digest &&
        k1->size == k2->size) {
    if (k1->mtime == k2->mtime) {
      return hit;
    }
    if (revalidation) {
      return stale;
    }
}
return miss;

In that code, stale only means “go check the digest”. What Bootsnap reports, and what BOOTSNAP_STATS counts, is one of four cases:

  • Hit. Same Ruby, same size, same mtime. Bootsnap loads the cached bytecode and never reads the source.

    If a file’s contents change but its size and mtime don’t, Bootsnap still serves the old bytecode. Editors and checkouts update mtimes, so in practice this almost never happens.

  • Miss. There’s no cache file for this path that Bootsnap can open. Bootsnap compiles the source and writes a new entry.

  • Stale. A cache file exists but doesn’t match: a different Ruby or size, a different mtime with revalidation off, or a different digest. Bootsnap compiles the source and overwrites the entry.

  • Revalidated. This one needs revalidation on. The mtime differs, so Bootsnap reads the source, hashes it, and compares that with the stored digest. They match, so it loads the cached bytecode instead of compiling. Hashing a file is much cheaper than compiling it.

YAML

The YAML cache uses the same machinery. Bootsnap prepends a patch to YAML.load_file and YAML.unsafe_load_file. On a miss it parses the YAML and stores the result as MessagePack.

How we fixed Bootsnap caching on NeetoDeploy

NeetoDeploy is the platform we build and run all our Neeto apps on. It builds images with Cloud Native Buildpacks (Paketo-based), and every pod starts as a fresh container from that image. Boot time matters everywhere, and most of all in production. But we also use review apps, and to save on costs they sleep when idle, so the next request wakes a new pod and waits for Rails to boot. That’s where you feel it.

Checking whether the cache helped

Bootsnap’s instrumentation hook can tell you. Events fire for the compile cache only:

require_relative "config/boot" # sets up Bootsnap

counts = Hash.new(0)
Bootsnap.instrumentation = ->(event, _path) { counts[event] += 1 }
require_relative "config/environment"
p counts # => {hit: …, miss: …, stale: …, revalidated: …}

Run it as a fresh process, with bundle exec ruby, not from a console that has already booted.

BOOTSNAP_STATS=1 prints the same counts at exit. Locally, the second boot after a Gemfile change showed 9,135 hits and 0 misses. Bootsnap was doing its job there.

But on a fresh production console pod, Bootsnap reported 11,610 misses and 443 hits, and the first boot took 22.6 seconds. With DISABLE_BOOTSNAP=1, which turns Bootsnap off, it took 23.0 and 25.3 seconds. Bootsnap was making almost no difference.

Why every pod started empty

The build logs and the buildpack source showed four reasons the cache never helped. Each one would have broken it on its own:

  1. Every pod is a fresh container, so the cache has to come from the image.
  2. The build deleted it. Our .slugignore listed /tmp/cache, and the last buildpack removed everything it listed.
  3. The build and runtime used different gem paths. The buildpack installs gems twice: a build-gems layer with every group, used during the build, and a launch-gems layer with production gems, used by pods. The only Rails boot in the build, our release step, loaded gems from build-gems.
  4. Image export reset every mtime to 1980-01-01, so the same build always produces a byte-identical image (reproducible builds). Even with matching paths, every entry would look stale.

The fix

The cache has to be built during the build, against the paths pods load from, and it has to reach the image intact. A shipped entry is only usable if the pod has the same absolute path and the same Ruby, and the same mtime unless revalidation is on. That takes one buildpack change and two small app edits.

Precompile in the release step, against launch-gems. Bootsnap keys every entry by absolute path, so a cache built against build-gems is useless to a pod that loads from launch-gems. Computing Bootsnap’s cache file names on a pod showed exactly that: an entry for build-gems/…/active_support.rb, none for launch-gems/…. The precompile has to name the runtime paths itself. It also can’t hang off assets:precompile, the obvious hook, because the assets buildpack skips that task when assets haven’t changed. The release step runs on every build, so bin/release does it:

# Release runs inside the NeetoDeploy build, where bundler points at build-gems, but pods boot
# from launch-gems. Precompile those paths so the image ships a bootsnap cache pods can use.
launch_gems = Dir["/layers/*/launch-gems"].first
system("bundle exec bootsnap precompile --cache-dir tmp/bootsnap-cache app/ lib/ #{launch_gems}") if launch_gems

Make launch-gems exist during the build. It was a launch-only layer. When the Gemfile.lock hadn’t changed, the buildpack reused it from the previous image and restored only its metadata, so the directory wasn’t there while the release step ran, and the precompile above was skipped. In the buildpack’s source, launch-gems had layer.Launch = true but no layer.Cache = true, unlike build-gems. Marking it cached is the platform change. Its contents are now restored during the build, where the release step can read them.

Move the cache, revalidate, and stay read-only. BOOTSNAP_CACHE_DIR moves the cache out of tmp/cache, its default, which .slugignore deletes.

Revalidation is off unless you set BOOTSNAP_REVALIDATE=true. We turn it on. It enables the digest check described earlier, so an entry whose mtime differs is still used when the contents match. We need it because image export resets every mtime to 1980, and without it every entry is stale.

Read-only is off unless you set BOOTSNAP_READONLY=true, and we turn it on for deployed processes. In read-only mode Bootsnap uses the cache and never writes to it. A miss compiles in memory, and the load-path cache isn’t saved. The write that read-only skips is easy to miss. After a revalidation, Bootsnap normally rewrites the entry’s header with the new mtime and calls fdatasync, so the next boot can take the fast path. Our mtimes never matched, so every new pod rewrote and synced every entry, and our web pods on staging took 51–61 seconds to become ready, up from 10–13. We caught it there, before it reached production. Read-only skips the write. It hashes every file on every boot instead, which is cheap next to writing and syncing it.

How Bootsnap decides between hit, miss, stale and revalidated Ruby loads user.rb. If there is no cache file at hash(path), it is a miss: compile and write an entry. If the Ruby version or the file size differ, it is stale: compile and overwrite. If the mtime is the same, it is a hit: load the bytecode. Otherwise, if revalidation is off, it is stale. If it is on, Bootsnap hashes the contents; a different digest is stale, the same digest is revalidated. After a revalidation, unless read-only is set, Bootsnap rewrites the header and calls fdatasync; in read-only mode it skips the write. Ruby loads user.rb cache file at hash(path)? same Ruby + same size? same mtime? revalidation on? hash contents: same digest? revalidated read-only? skip the write yes yes no yes yes yes no no yes no no no ✕ miss compile, write entry ✕ stale ✕ stale ✕ stale compile, overwrite compile, overwrite compile, overwrite ✓ hit load bytecode ⚠ rewrite header + fdatasync
Only a hit skips reading the source. A revalidation is cheap until it rewrites the header and calls fdatasync, and read-only skips that write.

config/boot.rb points every process at that cache, revalidates by content, and stays read-only outside development and test:

# The deploy image drops tmp/cache and resets file mtimes to 1980, so keep the
# cache built in bin/release elsewhere and revalidate by content. Deployed pods
# read it only: revalidating with writes fsyncs every entry (~60s boots on staging).
ENV["BOOTSNAP_CACHE_DIR"] ||= File.expand_path("../tmp/bootsnap-cache", __dir__)
ENV["BOOTSNAP_REVALIDATE"] ||= "true"
ENV["BOOTSNAP_READONLY"] ||= "true" unless ["", "development", "test"].include?(ENV["RAILS_ENV"].to_s)

require "bundler/setup"
require "bootsnap/setup"

Local boots with no RAILS_ENV set count as development and keep writing their cache, and ||= lets an environment variable override any of it. bootsnap precompile always runs with revalidation on and ignores BOOTSNAP_READONLY, so the build can still write the cache that read-only pods use.

Each build clones fresh and tmp/ is gitignored, so every image’s cache comes from that commit’s code. An unusable entry falls back to compiling the file, and a gem upgrade changes the gem’s path, so it gets new entries.

Results

I measured pod readiness from kube-state-metrics in our Prometheus: the time from container start to the pod passing its readiness probe.

max by (pod) (max_over_time(kube_pod_status_ready_time{pod=~"<app>-web-deployment.*"}[7d]))
- max by (pod) (max_over_time(kube_pod_container_state_started{pod=~"<app>-web-deployment.*"}[7d]))
Production web-pod readiness, start → readyRangeMedian
Before10–12 s11 s
After5–6 s6 s

Review-app web pods, the ones that wake from idle, went from a median of 12 seconds to 6.

What it costs

  • The precompile adds a few seconds to the release step.
  • The image grew by the cache, 138 MB in production.
  • The build cache grew by the size of launch-gems, 555 MB for NeetoChat.

A deploy adds a few seconds to the build and saves a few per pod. But every Rails process starts faster: autoscaled web pods, Sidekiq workers, console sessions, review apps waking up.

What I’d check in any app that uses Bootsnap

Measure a fresh production process, not a warm second boot:

BOOTSNAP_STATS=1 bin/rails runner 'nil'

If that shows mostly misses on a new container, the cache isn’t reaching it. Then check:

  • That the build runs bootsnap precompile --gemfile app/ lib/ config/. --gemfile covers every gem in the bundle, and the directories cover your app.
  • That it precompiles the paths your processes actually load from. If you’re unsure, compute a cache file name on the running container.
  • That nothing deletes the cache on the way: .dockerignore, .slugignore, cleanup steps.
  • If your build resets mtimes, that revalidation is on, and that deployed processes are read-only.
  • On a long-lived host, that something prunes the cache. Bootsnap never deletes old entries, and the README blames progressively slower deploys on exactly that.
  • That the cache isn’t on a network filesystem. The README says it’s “very likely to slow your application down quite a lot”.