Max Bernstein's Blog

Sorry for marking all the posts as unread

Wed, 31 Jan 2024 00:00:00 +0000

I noticed that the URLs were all a little off (had two slashes instead of one) and went in and fixed it. I did not think everyone's RSS software was going to freak out the way it did. PS: this is a special RSS-only post that is not visible on the site. Enjoy.

A survey of inlining heuristics

Wed, 03 Jun 2026 00:00:00 +0000

Compilers, especially method just-in-time compilers, operate on one function at a time. It is a natural code unit size, especially for a dynamic language JIT: at a given point in time, what more information can you gather about other parts of a running, changing system?

I don’t have any data to back this up—maybe I should go gather some—but on average, methods are small. Especially in languages such as Ruby that use method dispatch for everything, even instance variable (attribute, field, …) lookups, they are small. And everywhere.

This makes the compiler sad. If we are to continue to anthropomorphize them, compilers like having more context so they can optimize better. Consider the following silly-looking example that is actually representative of a surprising amount of real-world code:

class Point
  attr_reader :x, :y

  def initialize(x, y)
    @x = x
    @y = y
  end

  def distance(other)
    Math.sqrt((@x - other.x)**2 + (@y - other.y)**2)
  end
end

def distance_from_origin(x, y)
  Point.new(x, y).distance(Point.new(0, 0))
end

Right now, in the distance_from_origin method, I count 8 different method calls:

Point.new
Point#initialize
Point.new
Point#initialize
Point#distance
Float#**
Float#**
Math.sqrt

(Technically more, but the ivar lookups (including attr_reader!), addition, and subtraction are generally specialized and don’t push a frame, even in the interpreter.)

Furthermore, there are at least two heap allocations: one for each Point instance.

Last, there is a bunch of memory traffic to and from Point instances.

This all is a huge bummer! What should be a simple math operation is now overwhelmed with a bunch of other stuff. Point is certainly not a zero-cost abstraction.

Even if we had a bunch of other optimizations such as load-store elimination or escape analysis, they would not be able to do much: pretty much everything escapes and is effectful. That is, unless we inline. Inlining is the lever that enables a bunch of other optimization passes to kick in.

Inlining: the “easy” part

I wrote about the design and implementation of Cinder’s inliner (FB link, personal blog link) a couple of years ago. I wrote about arguably the simplest part, which is copying the callee body into the caller. It took me at least a week to get working. Probably closer to months if you consider all the plumbing through the rest of the JIT. In February during a small hackathon, I watched my colleague k0kubun prototype that bit of the inliner inside ZJIT in about 30 minutes.

There is more to do when pretty much every part of the VM is observable from the guest language: both Python and Ruby allow inspecting the state of the locals, the call stack, etc from user code. Sampling profilers also expect some amount of breadcrumbs to work with to inspect the stack. So there’s some more machinery still required to pretend like the callee function was not inlined. I talk about this a little bit in the Cinder blog post.

Even so, all of that can probably be designed and wired together in a couple of months. Then you will find yourself tuning the inliner for the next 10 years. This is much harder.

When: the harder part

The thing that makes inlining difficult, especially in a method JIT, is that you are trying to make an entire (dynamic!) system faster but you are only looking through a microscope and only capable of local reasoning¹. Whereas other optimizations such as strength reduction, inline caches, and value numbering are an un-alloyed good for the generated code, inlining can have negative effects. It is also perhaps the first optimization people add that has non-local impact.

If you inline wrong, your code size might blow up. This might thrash your CPU’s caches. Bummer, but happens to the best of us.

But also, if you inline wrong, you might get in the way of other helpful optimizations: if you hit some size limit after inlining method A, you might never get to inline B, which is the key to unlocking the performance of the method you are trying to optimize.

Last, inlining might hurt compile time. In situations where latency is paramount (think: interactive client JavaScript), adding tons more code into the fray might add noticeable hiccups, even if the long-term throughput improves. As always, in-band compilation is a trade-off because any time you spend compiling, you are not executing code.

You have to write your compiler to reason about all of this stuff. So you have heuristics. For example, here is Michael Pollan’s inliner heuristic:

Inline methods. Mostly small. Not too many.

I did a survey of a bunch of compilers, mostly JIT compilers, to see what their inlining heuristics look like. I also read (skimmed) some papers to see what those folks had to say. I wonder if they agree.

This post was a long time coming. I started working on it about five years ago but then when I quit working at Facebook I accidentally left behind all of the inliner research I did for Cinder’s inliner. So then I kind of just thought about it aimlessly for a while before redoing it this year. Anyway, here’s wonderwall.

The heuristics

Spoiler alert: all in all, people tend to look at:

Profiles of call target
Cumulative caller size (increasing as callees get inlined)
Callee size
Inline depth
Number of inlined calls at a certain depth
If recursion is present
Callee/caller call count ratio (if callee only called less than K% of calls to caller, don’t inline callee)
Callee stack usage
Polymorphism in callee
What mode the compiler is in (baseline vs more aggressive)
If the callee looks like it always raises/throws

And also have different interesting ways to pipe in profile information.

Last, some newer papers do some wild stuff:

Train neural networks to make inlining decisions
Let inlining drive the entire optimization pipeline, treating it as a search heuristic over a BFS walk of the call graph
Use AOT-gathered information to aid in JIT heuristics

Another thing to consider in inlining is how you gather and interpret profiles.

Call context and profiles: the other harder part

When you compile a function, you tend to specialize it based on the input it has historically been given. For a monomorphic input, maybe you guard that the type is still the same and otherwise jump into the interpreter. For a polymorphic input, maybe you check the top K (~4) common cases and otherwise jump into the interpreter. Fine.

But sometimes you can be compiling a polymorphic method bar that is actually monomorphic in its caller foo. That is, foo might only ever pass one kind of input to bar, but other callers pass all kinds of stuff. Here is a bit of a silly example to show what I mean:

class HashWithIndifferentAccess
  def initialize
    @hash = {}
  end

  # Allow reading from the Hash with either a String or a Symbol
  def [](key) = @hash[key.to_sym]

  # ...
end

# some method...
some_hash = HashWithIndifferentAccess.new
# ...
some_hash["abc"]

# some other method...
another_hash = HashWithIndifferentAccess.new
# ...
another_hash[:xyz]

Just kidding, not so silly at all. It’s a super common pattern in Rails. It makes key polymorphic in HashWithIndifferentAccess#[] even though for many of its callers, it may well be monomorphic (or even a constant).

In order to plumb this information through to the compiler, you have to figure out this call context relationship. There are a couple of common ways to do it.

Splitting

YJIT, for example, though it does not inline, splits methods based on the types of the arguments going in. This means that it clones the compiled code, generating a new version for each context. This does not give call context (“A calls B”) but gives type context (“B is called with integers, B’ is called with strings”).

A compiler could do type-based splitting in the interpreter or a baseline tier.

Profile splitting

If you don’t fancy duplicating the code, you can instead duplicate the profiles. You could either do this using type context (as above) or using call context. SpiderMonkey, for example, does “trial inlining” that allows callers to pass down a bit of memory for potential inline candidate callees to record their inline caches. Instead of each function holding its own ICScript, the caller allocates a unique ICScript for that potential-inline call-site. This gives each callee function (at least?) one level of call context.

Later, when inlining the callee into the caller, we don’t have other callers’ type information polluting the IR builder (or whatever reads the profiles).

Bytecode inlining

JavaScriptCore handles this by inlining bytecode into other bytecode. This is a gnarly transformation but gives the interpreter, even (!) access to call context. On tier-up to the compiler, all the inlining decisions have been made already.

Early tier with counters

HotSpot handles this with multiple tiers. The interpreter tiers up to the client compiler, C1. C1 profiles branch and call targets in compiled code. C1 may eventually recompile based on this new information. C1 may eventually tier up to C2, which copies C1 inlining decisions. This way, we get call context in profiles via inlining.

Inline and analyze and hope

One last thing you could do is just trust your type inference and branch folding in the optimizer. You could inline and do polymorphic specialization in the callee when building the IR, then hope that your branch pruning monomorphizes the inlined callee. It’s a little wasteful because the polymorphic code is built “for nothing”, but it might work fine?

Okay, onto the collected notes and half-baked commentary. Here’s a survey of a bunch of JIT compilers and how they reason about inlining heuristics.

Thanks

But before we get into that, thanks to Iain Ireland, CF Bolz-Tereick, and Ian Rogers for feedback on this blog post!

The survey: bits and bobbles

What follows is mostly a “bits and bobbles” section a la Phil Zucker.

We’ll start with Cinder, because when I wrote Cinder’s inliner I added only the simplest heuristics, mostly “don’t inline” signals. Over time, after I left, people tuned it a bit more.

Cinder

The inliner starts from the caller CFG, walking it to find suitable inlining candidates. Inlining candidates are only for call targets that are known—in Cinder’s case, only for monomorphic call targets—and pass some checks. The callee is only known by it’s function object, which includes its bytecode. There is no IR available for the callee until we decide to inline.

Most of the “can’t handle this” checks are related to argument handling. Python has a pretty complex calling convention, so if the caller/callee have not agreed on how the arguments should be passed through, the inliner doesn’t care to try and figure it out on its own. That is the responsibility of other parts of the compiler. Things in this canInline function could be considered “TODO”.

bool canInline(Function& caller, AbstractCall* call_instr) {
  // ...
  BorrowedRef<PyFunctionObject> func = call_instr->func;
  auto fail = [&](InlineFailureType failure_type) {
    dlogAndCollectFailureStats(caller, call_instr, failure_type);
    return false;
  };

  if (func->func_kwdefaults != nullptr) {
    return fail(InlineFailureType::kHasKwdefaults);
  }

  BorrowedRef<PyCodeObject> code{func->func_code};
  JIT_CHECK(PyCode_Check(code), "Expected PyCodeObject");

  if (code->co_kwonlyargcount > 0) {
    return fail(InlineFailureType::kHasKwOnlyArgs);
  }
  // ...
}

Failures are logged so they can be analyzed. If the Cinder team determines that there is some very frequent case they should handle, they will find out from the logs.

The inliner collects all candidate call instructions in one pass over the CFG. It loads the configurable “cost limit” from the options struct. Then it does one pass over the inlining candidates vector, inlining until it (maybe) hits the cost limit.

// ...
size_t cost_limit = getConfig().inliner_cost_limit;
size_t cost = codeCost(irfunc.code);

// Inline as many calls as possible, starting from the top of the function and
// working down.
for (auto& call : to_inline) {
  BorrowedRef<PyCodeObject> call_code{call.func->func_code};
  size_t new_cost = cost + codeCost(call_code);
  if (new_cost > cost_limit) {
    LOG_INLINER(
        "Inliner reached cost limit of {} when trying to inline {} into {}, "
        "inlining stopping early",
        new_cost,
        funcFullname(call.func),
        irfunc.fullname);
    break;
  }
  cost = new_cost;

  inlineFunctionCall(irfunc, &call);

  // We need to reflow types after every inline to propagate new type
  // information from the callee.
  reflowTypes(irfunc);
}
// ...

It does some graph maintenance work after inlining these calls, but that’s it.

This approach gets a surprising amount of utility for being so simple: it inlines constants (quite a few methods look like def foo(): return 5), small methods, and (at least, as far as I can remember) shrinks the compiled code size. All for very little compile time overhead.

There’s one other “standalone” Python JIT out there, PyPy. So we should look at that too.

PyPy

There are two inliners in PyPy. One is inside the RPython to C translation pipeline, which acts more like an ahead-of-time compiler². Then there is the tracing JIT bit, which has its own optimizer and heuristics. We’re going to look at the latter.

I talked to CF Bolz-Tereick about the inliner and their comment was that PyPy’s inlining heuristic is “yes”. There are a couple of exceptions, such as not inlining recursive functions or functions with loops. But the basic idea of tracing includes tracing through call instructions, which naturally means that you are “inlining”.

PyPy also does this neat thing where they treat frame pushes like normal allocation. Frame pushes, frame reads, and frame writes get written to the trace like normal object memory traffic and can get optimized away like other field reads and writes. This means that they can “just” use DCE to eliminate frame pushes and pops, whereas Cinder has some complicated mechanism to do it (which is my fault).

TODO get more details here

V8

V8 is a JS engine and it has over the years had many execution approaches. We’ll look at three of them since they all have or had their place in the history:

Hydrogen was the first real SSA IR and it looks very familiar to me, having worked on Cinder and now ZJIT. It is now defunct.
Turbofan was the replacement, going full Sea of Nodes. In the grand scheme of things it is a pretty fast compiler, but it does not hold back from doing some expensive rewrites. This was recently rewritten from Sea of Nodes to a mode traditional CFG and nicknamed Turboshaft.
Maglev is meant to coexist alongside Turbofan, preferring to speculate a little more eagerly and do fewer incremental rewrites in the name of compile time.³

They also each inline at different times in the pipeline, which made for a fun time trying to understand the different codebases.

V8 Hydrogen

Inlining happens during Hydrogen graph building

https://github.com/tekknolagi/v8/blob/a969ab67f8e1e7475d9b26468225c3a772890c64/src/crankshaft/hydrogen.cc#L9236

Don’t store function bytecode of all functions; need to re-parse callee text source to inline

Heuristics https://github.com/tekknolagi/v8/blob/a969ab67f8e1e7475d9b26468225c3a772890c64/src/crankshaft/hydrogen.cc#L7807

something about native context
check callee AST size against configurable limit
check inlining depth against configurable limit
don’t inline recursive functions
check current cumulative method size (as tracked by AST node count) against configurable limit

V8 TurboFan

https://docs.google.com/document/d/1VoYBhpDhJC4VlqMXCKvae-8IGuheBGxy32EOgC2LnT8/edit

https://github.com/v8/v8/blob/036842f4841326130a40adfcff38f85a9b4cd30a/src/compiler/js-inlining-heuristic.h#L14

Find candidates https://github.com/v8/v8/blob/036842f4841326130a40adfcff38f85a9b4cd30a/src/compiler/js-inlining-heuristic.cc#L134
Can inline https://github.com/v8/v8/blob/036842f4841326130a40adfcff38f85a9b4cd30a/src/compiler/js-inlining-heuristic.cc#L75
Force inline small functions https://github.com/v8/v8/blob/036842f4841326130a40adfcff38f85a9b4cd30a/src/compiler/js-inlining-heuristic.cc#L309
Loop over sorted (by comparator) list https://github.com/v8/v8/blob/036842f4841326130a40adfcff38f85a9b4cd30a/src/compiler/js-inlining-heuristic.cc#L847

V8 Maglev

When optimizing, add call instructions to the inline candidates list: https://github.com/v8/v8/blob/1a391f98cc7a9196369f2d6cab7df35ffbe92c08/src/maglev/maglev-graph-optimizer.cc#L1271

ProcessResult MaglevGraphOptimizer::VisitCall(Call* node,
                                              const ProcessingState& state) {
  // ...
  int bytecode_length = shared.GetBytecodeArray(broker()).length();
  float score =
      (call_frequency / bytecode_length) * (loop_depth_ > 0 ? 1.5 : 1.0);

  bool is_small_function =
      bytecode_length <
      reducer_.graph()->compilation_info()->flags().max_eager_inlined_bytecode;
  // ...
  MaglevCallSiteInfo* call_site = reducer_.zone()->New<MaglevCallSiteInfo>(
      MaglevCallerDetails{
          ...
          is_small_function, call_frequency,
          ...
      },
      score, bytecode_length);
  reducer_.PushInlineCandidate(call_site);
  // ...
}

https://github.com/v8/v8/blob/036842f4841326130a40adfcff38f85a9b4cd30a/src/maglev/maglev-inlining.h#L36

Unlike for example Cinder, Maglev looks like it does not have a lot of restrictions about what can get inlined into what, so its “can inline” signal is about budget. Actually two budgets: small budget and normal budget.

bool MaglevInliner::CanInlineCall() {
  // We stop inlining entirely if the small budget is exhausted.
  // Inlining decisions after that become bad if we stop inlining small
  // functions, but keep inlining large ones.
  return !graph_->inlineable_calls().empty() &&
         (graph_->total_inlined_bytecode_size() <
              max_inlined_bytecode_size_cumulative() ||
          graph_->total_inlined_bytecode_size_small() <
              max_inlined_bytecode_size_small_total());
}

Then its inlining loop is a greedy walk of the to-inline queue checking candidate sizes.

bool MaglevInliner::InlineCallSites() {
  DCHECK(CanInlineCall());
  while (!graph_->inlineable_calls().empty()) {
    // pop from inlineable_calls
    MaglevCallSiteInfo* call_site = ChooseNextCallSite();

    bool is_small_with_heapnum_input_outputs =
        IsSmallWithHeapNumberInputsOutputs(call_site);

    if (graph_->total_inlined_bytecode_size() >
        max_inlined_bytecode_size_cumulative()) {
      // We ran out of budget. Checking if this is a small-ish function that we
      // can still inline.
      if (graph_->total_inlined_bytecode_size_small() >
          max_inlined_bytecode_size_small_total()) {
        graph_->compilation_info()->set_could_not_inline_all_candidates();
        break;
      }

      if (!is_small_with_heapnum_input_outputs) {
        graph_->compilation_info()->set_could_not_inline_all_candidates();
        // Not that we don't break just rather just continue: next candidates
        // might be inlineable.
        continue;
      }
    }

    InliningResult result =
        BuildInlineFunction(call_site, is_small_with_heapnum_input_outputs);
    // ...
  }
  return true;
}

It runs this loop (which drains the queue) interleaved with the optimizer (which populates the queue).

bool MaglevInliner::Run() {
  if (graph_->inlineable_calls().empty()) return true;

  while (CanInlineCall()) {
    if (!InlineCallSites()) return false;
    RunOptimizer();
  }
  // ...
}

Confusingly, though, the optimizer also calls another function called CanInlineCall which checks if it legally can inline:

skip recursion
https://github.com/v8/v8/blob/1a391f98cc7a9196369f2d6cab7df35ffbe92c08/src/objects/shared-function-info-inl.h#L421
not called enough (min call frequency)
bytecode too big

bool MaglevGraphBuilder::ShouldEagerInlineCall( ~~appears unused? / dead declaration?~~ maybe src/maglev/maglev-graph-builder.cc is just not working on github search

MaybeReduceResult MaglevGraphBuilder::TryBuildCallKnownJSFunction( ~~also unused / dead declaration~~ same

JavaScriptCore

JavaScriptCore is funky! Unlike these other compilers that do inlining in their neat little SSA IRs, JSC inlines at the bytecode level⁴. This is their way of making sure that they get at least one level of call context into their interpreter inline caches, which will eventually give better information to the compiler.

JSC only inlines based on bytecode profile information, and only inlines bytecode??

TODO find better sources for bytecode inlining

SpiderMonkey

SpiderMonkey has another way of getting that call context without doing bytecode inlining: they add call context to their inline caches. Methods can pass down an ICScript to their callees where the callee writes its inline cache information. Then, when compiling, the callee is more likely to be monomorphized.

Wasm

https://github.com/mozilla-firefox/firefox/blob/438a3ce10eb77fb50d968463b7741117aec5bb4a/js/src/wasm/WasmHeuristics.h#L213

SpiderMonkey ICScript

Wasmtime and Cranelift

https://fitzgen.com/2025/11/19/inliner.html

HotSpot

Plan: run in interpreter; tier up to C1; profile call targets; inline in C1; profile branch counts; tier up to C2, which copies C1 inlining decisions in bytecode parser

HotSpot C2

https://github.com/openjdk/jdk/blob/a05d5d2514c835f2bfeaf7a8c7df0ac241f0177f/src/hotspot/share/opto/bytecodeInfo.cpp#L116

https://github.com/openjdk/jdk/blob/497dca2549a9829530670576115bf4b8fab386b3/src/hotspot/share/opto/bytecodeInfo.cpp#L197

https://github.com/openjdk/jdk/blob/497dca2549a9829530670576115bf4b8fab386b3/src/hotspot/share/opto/parse.hpp#L42

https://github.com/openjdk/jdk/blob/497dca2549a9829530670576115bf4b8fab386b3/src/hotspot/share/opto/doCall.cpp#L185

Not too small

Walk up the call stack to figure out what to compile

Handling the right thing to inline: def foo(a) = a.each {|x| x } want to compile foo, inline each, inline block, not compile block separately (probably)

HotSpot C1

https://bernsteinbear.com/assets/img/design-hotspot-client-compiler.pdf

https://github.com/openjdk/jdk/blob/d854a04231a437a6af36ae65780961f40f336343/src/hotspot/share/c1/c1_GraphBuilder.cpp#L755

https://github.com/openjdk/jdk/blob/d854a04231a437a6af36ae65780961f40f336343/src/hotspot/share/c1/c1_GraphBuilder.cpp#L3854

skip callees with exception handlers (unless explicitly allowed with a CLI flag)
skip synchronized callees (unless explicitly allowed with a CLI flag)
skip classes with unlinked callees
skip uninitialized classes
…

heuristics:

max inline level (default 9)
max recursive inline level (default 1)
callee bytecode size (max for top level is 35 bytecodes, but falls off by 10% per inline level)

callee stack usage (max of 10 slots)

      // Additional condition to limit stack usage for non-recursive calls.
      if ((callee_recursive_level == 0) &&
          (callee->max_stack() + callee->max_locals() - callee->size_of_parameters() > C1InlineStackLimit)) {
        INLINE_BAILOUT("callee uses too much stack");
      }

max total method size (default 8000 bytecodes)

TruffleRuby

TruffleRuby uses weighted compile queue

Graal https://ieeexplore.ieee.org/document/8661171

.NET

https://github.com/dotnet/runtime/blob/2d638dc1179164a08d9387cbe6354fe2b7e4d823/docs/design/coreclr/jit/inlining-plans.md

https://github.com/dotnet/runtime/blob/0b3f3ab1ecf4de06459e5f0e2b7cb3baf70ef981/src/coreclr/jit/inline.def#L94

https://github.com/dotnet/runtime/blob/0b3f3ab1ecf4de06459e5f0e2b7cb3baf70ef981/src/coreclr/jit/inlinepolicy.cpp

https://github.com/dotnet/runtime/blob/0b3f3ab1ecf4de06459e5f0e2b7cb3baf70ef981/docs/design/coreclr/jit/inline-size-estimates.md?plain=1#L5 https://github.com/dotnet/runtime/blob/0b3f3ab1ecf4de06459e5f0e2b7cb3baf70ef981/src/coreclr/jit/fginline.cpp

https://github.com/dotnet/runtime/issues/10303

https://github.com/AndyAyersMS/PerformanceExplorer/blob/master/notes/notes-aug-2016.md

Dart

https://github.com/dart-lang/sdk/blob/391212f3da8cc0790fc532d367549042216bd5ca/runtime/vm/compiler/backend/inliner.cc#L49

https://github.com/dart-lang/sdk/blob/391212f3da8cc0790fc532d367549042216bd5ca/runtime/vm/compiler/backend/inliner.cc#L1023

https://web.archive.org/web/20170830093403id_/https://link.springer.com/content/pdf/10.1007/978-3-540-78791-4_5.pdf

DEFINE_FLAG(int,
            deoptimization_counter_inlining_threshold,
            12,
            "How many times we allow deoptimization before we stop inlining.");
DEFINE_FLAG(bool, trace_inlining, false, "Trace inlining");
DEFINE_FLAG(charp, inlining_filter, nullptr, "Inline only in named function");

// Flags for inlining heuristics.
DEFINE_FLAG(int,
            inline_getters_setters_smaller_than,
            10,
            "Always inline getters and setters that have fewer instructions");
DEFINE_FLAG(int,
            inlining_depth_threshold,
            6,
            "Inline function calls up to threshold nesting depth");
DEFINE_FLAG(
    int,
    inlining_size_threshold,
    25,
    "Always inline functions that have threshold or fewer instructions");
DEFINE_FLAG(int,
            inlining_callee_call_sites_threshold,
            1,
            "Always inline functions containing threshold or fewer calls.");
DEFINE_FLAG(int,
            inlining_callee_size_threshold,
            160,
            "Do not inline callees larger than threshold");
DEFINE_FLAG(int,
            inlining_small_leaf_size_threshold,
            50,
            "Do not inline leaf callees larger than threshold");
DEFINE_FLAG(int,
            inlining_caller_size_threshold,
            50000,
            "Stop inlining once caller reaches the threshold.");
DEFINE_FLAG(int,
            inlining_hotness,
            10,
            "Inline only hotter calls, in percents (0 .. 100); "
            "default 10%: calls above-equal 10% of max-count are inlined.");
DEFINE_FLAG(int,
            inlining_recursion_depth_threshold,
            1,
            "Inline recursive function calls up to threshold recursion depth.");
DEFINE_FLAG(int,
            max_inlined_per_depth,
            500,
            "Max. number of inlined calls per depth");

An adaptive strategy for inline substitution (PDF)

  // Inlining heuristics based on Cooper et al. 2008.
  InliningDecision ShouldWeInline(const Function& callee,
                                  intptr_t instr_count,
                                  intptr_t call_site_count) {
    // Pragma or size heuristics.
    if (inliner_->AlwaysInline(callee)) {
      return InliningDecision::Yes("AlwaysInline");
    } else if (inlined_size_ > FLAG_inlining_caller_size_threshold) {
      // Prevent caller methods becoming humongous and thus slow to compile.
      return InliningDecision::No("--inlining-caller-size-threshold");
    } else if (instr_count > FLAG_inlining_callee_size_threshold) {
      // Prevent inlining of callee methods that exceed certain size.
      return InliningDecision::No("--inlining-callee-size-threshold");
    }
    // Inlining depth.
    const int callee_inlining_depth = callee.inlining_depth();
    if (callee_inlining_depth > 0 &&
        ((callee_inlining_depth + inlining_depth_) >
         FLAG_inlining_depth_threshold)) {
      return InliningDecision::No("--inlining-depth-threshold");
    }
    // Situation instr_count == 0 denotes no counts have been computed yet.
    // In that case, we say ok to the early heuristic and come back with the
    // late heuristic.
    if (instr_count == 0) {
      return InliningDecision::Yes("need to count first");
    } else if (instr_count <= FLAG_inlining_size_threshold) {
      return InliningDecision::Yes("--inlining-size-threshold");
    } else if (call_site_count <= FLAG_inlining_callee_call_sites_threshold) {
      return InliningDecision::Yes("--inlining-callee-call-sites-threshold");
    }
    return InliningDecision::No("default");
  }

HHVM

tracelet based

https://github.com/facebook/hhvm/blob/eeba7ad1ffa372a9b8cc9d1ec7f5295d45627009/hphp/runtime/vm/jit/inlining-decider.h#L89

  // Refuse if the cost exceeds our thresholds.
  // We measure the cost of inlining each callstack and stop when it exceeds a
  // certain threshold.  (Note that we do not measure the total cost of all the
  // inlined calls for a given caller---just the cost of each nested stack.)
  cost = costOfInlining(callerSk, callee, regionAndUnit, annotationsPtr);
  if (cost <= Cfg::HHIR::AlwaysInlineVasmCostLimit) {
    return accept(folly::sformat("cost={} within always-inline limit", cost));
  }

  if (region.instrSize() > irgs.budgetBCInstrs) {
    return refuse(folly::sformat(
      "exhausted bytecode budget: budgetBCInstrs={}, regionSize={}",
      irgs.budgetBCInstrs, region.instrSize()));
  }
  auto maxTotalCost = adjustedMaxVasmCost(irgs, region, inlineDepth(irgs));
  int maxCost = maxTotalCost;
  if (Cfg::HHIR::InliningUseStackedCost) {
    maxCost -= irgs.inlineState.cost;
  }
  const auto baseProfCount = s_baseProfCount.load();
  const auto callerProfCount = irgen::curProfCount(irgs);
  const auto calleeProfCount = irgen::calleeProfCount(irgs, region);
  if (cost > maxCost) {
    auto const depth = inlineDepth(irgs);
    return refuse(folly::sformat(
      "too expensive: cost={} : maxCost={} : "
      "baseProfCount={} : callerProfCount={} : calleeProfCount={} : depth={}",
      cost, maxCost, baseProfCount, callerProfCount, calleeProfCount, depth));
  }

  return accept(folly::sformat("small region with return: cost={} : "
                               "maxTotalCost={} : maxCost={} : baseProfCount={}"
                               " : callerProfCount={} : calleeProfCount={}",
                               cost, maxTotalCost, maxCost, baseProfCount,
                               callerProfCount, calleeProfCount));

ART

https://github.com/LineageOS/android_art/blob/8ce603e0c68899bdfbc9cd4c50dcc65bbf777982/compiler/optimizing/inliner.h

// Instruction limit to control memory.
static constexpr size_t kMaximumNumberOfTotalInstructions = 1024;

// Maximum number of instructions for considering a method small,
// which we will always try to inline if the other non-instruction limits
// are not reached.
static constexpr size_t kMaximumNumberOfInstructionsForSmallMethod = 3;

// Limit the number of dex registers that we accumulate while inlining
// to avoid creating large amount of nested environments.
static constexpr size_t kMaximumNumberOfCumulatedDexRegisters = 32;

// Limit recursive call inlining, which do not benefit from too
// much inlining compared to code locality.
static constexpr size_t kMaximumNumberOfRecursiveCalls = 4;

// Limit recursive polymorphic call inlining to prevent code bloat, since it can quickly get out of
// hand in the presence of multiple Wrapper classes. We set this to 0 to disallow polymorphic
// recursive calls at all.
static constexpr size_t kMaximumNumberOfPolymorphicRecursiveCalls = 0;

// Controls the use of inline caches in AOT mode.
static constexpr bool kUseAOTInlineCaches = true;

// Controls the use of inlining try catches.
static constexpr bool kInlineTryCatches = true;

void HInliner::UpdateInliningBudget() {
  if (total_number_of_instructions_ >= kMaximumNumberOfTotalInstructions) {
    // Always try to inline small methods.
    inlining_budget_ = kMaximumNumberOfInstructionsForSmallMethod;
  } else {
    inlining_budget_ = std::max(
        kMaximumNumberOfInstructionsForSmallMethod,
        kMaximumNumberOfTotalInstructions - total_number_of_instructions_);
  }
}

bool HInliner::IsInliningEncouraged(const HInvoke* invoke_instruction,
                                    ArtMethod* method,
                                    const CodeItemDataAccessor& accessor) const {
  if (CountRecursiveCallsOf(method) > kMaximumNumberOfRecursiveCalls) {
    LOG_FAIL(stats_, MethodCompilationStat::kNotInlinedRecursiveBudget)
        << "Method "
        << method->PrettyMethod()
        << " is not inlined because it has reached its recursive call budget.";
    return false;
  }

  size_t inline_max_code_units = codegen_->GetCompilerOptions().GetInlineMaxCodeUnits();
  if (accessor.InsnsSizeInCodeUnits() > inline_max_code_units) {
    LOG_FAIL(stats_, MethodCompilationStat::kNotInlinedCodeItem)
        << "Method " << method->PrettyMethod()
        << " is not inlined because its code item is too big: "
        << accessor.InsnsSizeInCodeUnits()
        << " > "
        << inline_max_code_units;
    return false;
  }

  if (graph_->IsCompilingBaseline() &&
      accessor.InsnsSizeInCodeUnits() > CompilerOptions::kBaselineInlineMaxCodeUnits) {
    LOG_FAIL_NO_STAT() << "Reached baseline maximum code unit for inlining  "
                       << method->PrettyMethod();
    outermost_graph_->SetUsefulOptimizing();
    return false;
  }

  if (invoke_instruction->GetBlock()->GetLastInstruction()->IsThrow()) {
    LOG_FAIL(stats_, MethodCompilationStat::kNotInlinedEndsWithThrow)
        << "Method " << method->PrettyMethod()
        << " is not inlined because its block ends with a throw";
    return false;
  }

  return true;
}

if (outermost_graph_->IsCompilingBaseline() &&
    (current->IsInvokeVirtual() || current->IsInvokeInterface()) &&
    ProfilingInfoBuilder::IsInlineCacheUseful(current->AsInvoke(), codegen_)) {
  uint32_t maximum_inlining_depth_for_baseline =
      InlineCache::MaxDexPcEncodingDepth(
          outermost_graph_->GetArtMethod(),
          codegen_->GetCompilerOptions().GetInlineMaxCodeUnits());
  if (depth_ + 1 > maximum_inlining_depth_for_baseline) {
    LOG_FAIL_NO_STAT() << "Reached maximum depth for inlining in baseline compilation: "
                       << depth_ << " for " << callee_graph->GetArtMethod()->PrettyMethod();
    outermost_graph_->SetUsefulOptimizing();
    return false;
  }
}

JikesRVM

https://github.com/JikesRVM/JikesRVM/blob/5072f19761115d987b6ee162f49a03522d36c697/rvm/src/org/jikesrvm/compilers/opt/inlining/DefaultInlineOracle.java#L55

Other/research

Partial inlining

Understanding and Exploiting Optimal Function Inlining (PDF)

machine learning

Automatic construction of inlining heuristics using machine learning

Machine-Learning-Based Optimization Heuristics in Dynamic Compilers (PDF)

Guiding Inlining Decisions Using Post-Inlining Transformations (PDF)

U Can’t Inline This! (PDF)

Towards better inlining decisions using inlining trials

RhizomeRuby inlining

An Optimization-Driven Incremental Inline Substitution Algorithm for Just-in-Time Compilers (PDF)

Automatic Tuning of Inlining Heuristics (PDF)

Inlining-Benefit Prediction with Interprocedural Partial Escape Analysis (PDF)

Inlining of Virtual Methods (PDF)

A Study of Type Analysis for Speculative Method Inlining in a JIT Environment (PDF)

A Comparative Study of Static and Profile-Based Heuristics for Inlining (PDF)

clusters from Custom benefit-driven inliner in Falcon JIT (PDF)

Graal

https://github.com/oracle/graal/blob/5dde777cba22a99ebe3f19745d03ddfbc35c563c/compiler/src/jdk.graal.compiler/src/jdk/graal/compiler/phases/common/inlining/policy/GreedyInliningPolicy.java

https://github.com/oracle/graal/blob/5dde777cba22a99ebe3f19745d03ddfbc35c563c/compiler/src/jdk.graal.compiler/src/jdk/graal/compiler/phases/common/inlining/InliningPhase.java

https://github.com/oracle/graal/blob/5dde777cba22a99ebe3f19745d03ddfbc35c563c/compiler/src/jdk.graal.compiler/src/jdk/graal/compiler/phases/common/inlining/info/elem/InlineableGraph.java#L148

There are some newer papers, especially in Java land, that try to do a lot of analysis ahead-of-time and bundle the resulting information in .class files. Then the JIT can read it and see more than local context.

Or, if you are an AOT compiler, you can probably do a lot more whole system reasoning—both for time budget reasons and also because you can see more functions at once. ↩
Check it out if you like. I stumbled across it by accident. ↩
See also “Turbolev”, which seems to merge Maglev (CFG) with Turbofan (Sea of Nodes)… somehow. ↩
Potentially a misunderstanding based on a private conversation. I’m working on tracking down the implementation… ↩

Checking assembly with Z3

Mon, 01 Jun 2026 00:00:00 +0000

Short post today. New ZJIT contributor dak2 submitted a PR to fix an overflow bug in fixnum division in ZJIT. We did the division fine, but lied about the type of the result in the case of dividing FIXNUM_MIN by -1. You can see how this is special-cased in CRuby:

static inline void
rb_fix_divmod_fix(VALUE a, VALUE b, VALUE *divp, VALUE *modp)
{
    // ...
    if (x == FIXNUM_MIN && y == -1) {
        if (divp) *divp = LONG2NUM(-FIXNUM_MIN);
        if (modp) *modp = LONG2FIX(0);
        return;
    }
    // ...
}

Since -FIXNUM_MIN (note the negative) does not fit in a fixnum, it gets promoted to a bignum. It’s one of two special cases in fixnum division that does not produce a fixnum, the other being dividing by zero (which produces an error).

$ irb
irb(main):001> LONG_MAX = 2**63 - 1
=> 9223372036854775807
irb(main):002> FIXNUM_MAX = LONG_MAX / 2
=> 4611686018427387903
irb(main):003> LONG_MIN = -LONG_MAX - 1
=> -9223372036854775808
irb(main):004> FIXNUM_MIN = LONG_MIN / 2
=> -4611686018427387904
irb(main):005> (-FIXNUM_MIN) < FIXNUM_MAX
=> false
irb(main):006>
$

This is due to the numbers being two’s complement and therefore having more negative numbers than positive numbers (because of zero).

dak2’s proposed patch included a branchless test for this left == FIXNUM_MIN and right == -1 case, making us leave JIT code and enter the interpreter rather than handle it inline. The patch encodes this branchless test as xor, xor, or, test, je in our platform-independent low-level IR (LIR):

// Side exit on FIXNUM_MIN / -1, which overflows to a Bignum, not a Fixnum.
// Branchless (left == FIXNUM_MIN && right == -1): (left ^ MIN) | (right ^ -1) == 0.
let left_diff = asm.xor(left, Opnd::from(VALUE::fixnum_from_isize(RUBY_FIXNUM_MIN)));
let right_diff = asm.xor(right, Opnd::from(VALUE::fixnum_from_isize(-1)));
let combined = asm.or(left_diff, right_diff);
asm.test(combined, combined);
asm.je(jit, side_exit(jit, state, FixnumDivOverflow));

I didn’t understand why those were equivalent. Rather than try and bang my head against it, I thought I’d let Z3 try. After all, I’ve been watching CF have fun with it for a couple years now.

Z3 is an SMT solver. The core trick to use Z3 as a “proof engine” that I learned from CF¹ is to make Z3 search for counter-examples by negating the condition:

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "z3-solver",
# ]
# ///

import z3

WORD_SIZE = 64
LONG_MAX   = z3.BitVecVal(2**63 - 1, WORD_SIZE)
LONG_MIN   = -LONG_MAX - 1
FIXNUM_MIN = LONG_MIN / 2

def prove(cond):
    solver = z3.Solver()
    assert solver.check(z3.Not(cond)) == z3.unsat, solver.model()

left = z3.BitVec("left", WORD_SIZE)
right = z3.BitVec("right", WORD_SIZE)
# The original C condition
lhs = z3.And(left == FIXNUM_MIN, right == -1)
# The new branchless LIR
rhs = ((left ^ FIXNUM_MIN) | (right ^ -1)) == 0
prove(lhs == rhs)

This negation is required because the core model of Z3 is a machine that finds example values. This means that there is an implicit “exists” in front of your condition. To disprove this, Z3 needs to search all inputs. However, if you negate the condition, it becomes a “for all”. This means that in order to disprove the “for all”, Z3 only needs to find a single counterexample.

$ uv run prove_fixnum_min.py
$

Z3 did not complain, so the new code must be fine. Just as a quick check, in case assert is turned off or something, I like to see tests fail. So after modifying one of the constants from -1 to 0:

$ uv run prove_fixnum_min.py
Traceback (most recent call last):
  File "/path/prove_fixnum_min.py", line 26, in <module>
    prove(lhs == rhs)
    ~~~~~^^^^^^^^^^^^
  File "/path/prove_fixnum_min.py", line 18, in prove
    assert solver.check(z3.Not(cond)) == z3.unsat, solver.model()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: [right = 18446744073709551615, left = 13835058055282163712]
$

Neat.

Thanks

Thanks to CF Bolz-Tereick for reading and giving feedback on this post.

They note that this is a “standard technique” and they did not come up with it. ↩

Travel notes: RubyKaigi Hakodate

Mon, 18 May 2026 00:00:00 +0000

I just got back from a three and a half week trip to Japan. It was the longest trip I have ever been on (aside from studying abroad in Germany, which felt different). I made the following wild circuit with only a backpack and a duffel:

Tokyo
Toyama
Kanazawa
Nara ish
Ito
Hakodate
Nikko
Mashiko
Karuizawa
Tokyo

This trip was split into three parts: time with my immediate family, going to a conference, and then time with my partner. They were all great and also I am glad to be home.

I’ll post my abbreviated travel notes here, including activity and food recommendations.

Part one

We started in Tokyo but we were only there for about 40 hours. We focused our time mostly on arts and crafts: we did a kintsugi workshop, spent time at an artists cooperative, and then did a lot of walking around. This was a good intro to the trip, because everyone kept waking up at 4am and crashing at 7pm due to the jet lag. 4am wakeup makes for nice morning walks to 7-Eleven.

I brought my family to T’s Tantan in Tokyo Station because I’m vegetarian and it’s otherwise hard to find ramen that approaches kosher in Japan. It continues to be great and I really appreciate having a steady vegetarian option available. Many years ago when I visited Tokyo there was a place that served a delicious tomato-based vegetarian ramen, but I hear it has since permanently closed. Bummer.

We took the shinkansen to Kanazawa. I love the train. It’s fast. It’s quiet. You can eat your snacks on board and gaze out the window as the world whizzes by. It’s nice.

We toured a soy sauce factory (meh; they don’t let you in the room where the magic happens) and the old town (pretty!) before finally eventually ending up at our small hotel in Toyama: Satoyama Auberge Maki No Oto. I highly recommend this hotel. It is beautiful, the staff is lovely, the food was excellent, and they were very accomodating of me being vegetarian.

We continued on to Toyama, which is a port town. We got to talking with an older local guy who told us all about his favorite local spots. We learned after leaving that this guy has extraordinarily fancy taste and they were all either Michelin starred or at least Michelin rated and with a lead time of months. We opted to instead go to a local brewery, which had a ghost pepper beer (!) and pizza.

We then moved on via train to Osaka, where we transferred to a car to head (eventually) to our hotel in the hills near Nara. We toured the Daimon sake brewery. They explained every little thing about the process, which was especially interesting to me, as I’ve done some small amount of homebrewing and I bake. They sounded similar. We had a tasting and even got to talk to Daimon-san. I recommend going.

I also recommend the Akame 48 waterfalls walk/hike, which has some exquisite falls, and Murou Art Forest. They had some really wonderful installations.

My brother and I parted ways from the rest of my family in Osaka: they headed further west and we headed north to Itō on the Izu peninsula. We got a surprise perfectly clear view of Fuji along the way.

It’s beautiful there. They don’t seem to welcome foreigners in a lot of their restaurants (we were turned away several times) but one place had a guy who enthusiastically welcomed us in. We ended that evening enjoying a some food and a beer while also being stared at by a 300lb completely tattooed guy. It was a little unsettling but we left without incident.

My brother and I made our way to Tokyo for the day before his flight and before my train north to Hakodate for RubyKaigi. I once again did that thing where I walked around in humid 80F heat with a large backpack and pants and was extraordinarily warm toward the end of the day. After about a liter of Aquarius on the train north I felt better.

Part two

I stayed at Yunokawa Prince Hotel Nagisatei which I would like to especially call out for having an enormous, diverse, and very vegetarian friendly breakfast. Every morning I got to try new and tasty things and even feel full after. It was great.

Hakodate is beautiful in the spring. I arrived at peak cherry blossom season and Goryokaku, their star shaped fort, is absolutely decked out in cherry blossoms. It is also moderately swarmed by tourists (in this case, three cruise ships). It didn’t feel over-crowded though. I enjoyed eating at The Bear King which had a vegetarian friendly option.

The next day was the committer meeting. I don’t remember a ton from it other than people talking at length about the semantics of deep freezing an object (do you freeze its class? its class’s superclass? …?). I picked up my badge and also got to check out my colleague Chris Salzberg’s bar SOLENOID! It’s a neat spot. I headed out to go find some dinner.

This is about when I got a message on my phone that there was going to be an earthquake, so I walked back into the bar and said “hey, did you get this?” just before everything started shaking. It was the biggest earthquake I’ve experienced, but I was metaphorically not too shaken up. Then we got the tsunami warning.

Chris’s bar is already something like 8 meters above sea level and at the foot of Mt Hakodate. With the city sirens going off and the police directing traffic with batons, though, I decided my best bet was just to march directly up the mountain to get more elevation. Since the tsunami wasn’t scheduled to arrive for about 20 or 30 minutes and my hotel was across the sea-level part of town, I parked myself on a little concrete post. Chris found me eventually. Someone told us that there was a middle school offering refuge, so we went and hung out on the side of the gymnasium. They were really nice about it.

On Wednesday, the conference started. It was really well signed and organized. My usual complaint with conferences is that there’s nothing to eat for vegetarians (or that we get mashed with the gluten-free people and each group only gets a salad and bad bread) but that did not happen! They had really stellar vegetarian bento. They had a lot of leftovers toward the end of lunch so I even went and got a second. This was about when I started freaking out because my speaking slot was approaching and I wasn’t yet feeling my talk.

Normally when I give a talk, I get up in front of people and I pace and gesticulate and productively complain and throw in some fun anecdotes and the audience, one way or another, ends up learning about JITs at scale, or Scheme semantics, or something. It’s what I’d done for my little lunch talk at Brown two weeks prior. I even titled that talk One must imagine compiler engineers happy so there was plenty of room for educational complaining. But this RubyKaigi talk was in front of an enormous crowd and toward a more general audience than I was used to addressing. The slides did not feel like they were flowing until about twenty minutes before my talk.

In the end it went alright. I realized about 40 seconds in that I had way too much content so I ended up speaking rapidly for 30 minutes straight, completely unaware of the audience (which you can’t see anyway because of the lights). I only really noticed people when I made a dumb six-seven joke and Aaron laughed.

The rest of the conference I was able to relax and enjoy other people’s talks. I got some good hallway track in, too. I think there’s a good group of people who are interested in Ruby tracing (for example, Perfetto in ZJIT) so maybe we will make something happen.

We had a nice small dinner at Yasai Bar Miruya, which was vegan (!) and had some nice sake. The host was very friendly, too.

I nerd-sniped John and J into implementing a VM for the Universal Machine. This was a daunting homework assignment back in undergrad but it was a fun project later in life.

S joined toward the end of the conference. She’s also vegetarian so we got some really excellent vegetarian ramen at MAIDO Ramen.

Finally, S and I headed south on the shinkansen for Nikko.

Part three

Nikko is small, beautiful, and a tourist day-trip town. Dinner closes early. Shops close earlier. Since we were staying there we had to make sure to track down and visit the one or two vegetarian places before they shuttered.

S and I, along with J and J, took the bus up from Nikko, up the windiest switchbacks, to the Kegon Falls. We were going to take a boat across the lake, but the water level was too low for the dock on the other side, so we ended up half hiking and half taking a bus. Then we continued our hike through the Senjōgahara Marshland (beautiful), to the Yudaki Cascades (lovely), which also had a surprise restaurant and ice cream shop at the base! It’s called Yutaki Rest House. After some great (vegetarian friendly!! wow!!) udon, we marched up the waterfall and around Yuno Lake at the top to Yumoto Onsen. In order to make the last reasonable bus back to town, we just enjoyed putting our feet in the foot bath.

One day was rainy. In the evening, J and I thought it would be fun to continue our Universal Machine implementations. As Norman Ramsey would say, “my implementation is 90 lines long and runs sandmark in under six seconds.”

We also enjoyed doing a tour of the shrines right above Nikko. The shrines are resplendent against the backdrop of forest.

Pro bus tip: you can either pay by IC card or credit card. No need to grab a ticket if you do that.

S and I shipped our bags (thanks, Yamato) before continuing on to the small town of Moka, the staging area for our big pottery festival day. Unfortunately, there was no good way to get there: there was no reasonable series of trains and no taxi would take us. Ultimately we ended up taking the train to Utsonomiya and catching the long local bus to Moka. About twenty minutes into this ride, in the middle of nowhere, bus nearly empty, the bus driver pulled over and ran over to us looking kind of panicked. He asked where we were going and was visibly relieved when we said Moka. I suppose we are not the usual riders. Very nice of him.

Upon arrival, S introduced me to CoCo ICHIBANYA, which is also super vegetarian friendly. I loved it. We ate really well before walking to our tiny hotel.

We did not really know what to expect from the Mashiko pottery festival. The internet said it would be crowded and to arrive early, so we got up at 6:30am for estimated 7am departure on the tiny train from Moka to Mashiko. On most trains you can pay with an IC card but we were out in the sticks so we asked the only other guy on the platform how to pay for the train. He said he had no idea and that this was his first time here. When the train showed up completely packed to the gills and we had to (politely) push onto it, we started to realize that this was The Event and it was going to be mayhem.

Also, fun fact: the way the Moka train payment works is that you grab a little ticket from the train, and, upon arrival, wait in line to present your ticket to two very overwhelmed looking people at a table, who charge you, and you pay in cash.

Onto Mashiko: the festival was packed. There’s pottery everywhere the eye can see. There are tents and there are full buildings. It varies in quality and artistry from fine to jaw-droppingly spectacular. You could completely stock your kitchen from this fair alone and it would even be cost-effective. The main bummer for us is that we had to get pottery safely back home. We limited ourselves to a reasonable assortment but we really wanted to buy a beautiful painted 20 inch plate with a bird on a branch.

After a ton of walking around, we took another long long bus back to Utsonomiya and continued onto Karuizawa. We didn’t know what to expect from Karuizawa but, having been, I could probably concisely describe it as “Aspen for people from Tokyo”. It was… fine. We loved our hotel, Tsuruya Ryokan. The manager was very excited when we borrowed a Studio Ghibli DVD from their collection.

We continued on to Tokyo, our final stop. We our usual tour of stationery stores and bakeries—the bread was something to write home about (har har). We enjoyed a (vegetarian!! friendly!!) kaiseki meal at Hyoki Shabu-shabu Ginza before enjoying some live music at Rocky Top.

We also recommend Jikasei MENSHO for vegetarian ramen.

Bakery checklist:

BOUL’ANGE NIHONBASHI (check! good croissants)
Bricolage bread & co (check! good everything)
Brasserie Viron Marunouchi
Beaver Bread
Bricolage bread & co.
Bartizan Bread Factory
Gontran Cherrier Tokyo Aoyama Shop
Comme’N Tokyo
Shiomi Bakery
BRØD
The Little BAKERY <!–
https://www.jocjapantravel.com/kanto-tokyo-bakeries/ –>

We had an uneventful and reasonably easy trip home. Whew. Long post for a long trip. See you next year in Miyazaki!

Partial static single information form

Tue, 12 May 2026 00:00:00 +0000

In compilers, static single information form (SSI) is a common extension to static single assignment form (SSA). It was introduced by C. Scott Ananian in 1999 in his MS thesis (PDF) ¹.

SSI extends your existing SSA intermediate representation by discovering facts from your existing program and reifying them as path-dependent/flow-sensitive IR nodes. That might sound complicated, but at least the basic idea is pretty natural. I talk a little bit about it in What I talk about when I talk about IRs and I’ll rehash here in more depth, starting with some motivating examples. Consider this admittedly contrived example:

v0: Integer = ...
if v0 > 0:
    # ...
    v1: PositiveInteger = AbsoluteValue v0
    # ...

We should be able to learn from the comparison that in some branches in the IR, v0 is positive. In that region, we can add a new IR instruction v2 that attaches that knowledge right in the instruction’s type field (yay, sparseness!) and then rewrite uses of v0 to now use v2.

v0: Integer = ...
if v0 > 0:
    v2: PositiveInteger = RefineType v0, Positive
    # ...
    v1: PositiveInteger = AbsoluteValue v2
    # ...

Because we’ve done that, our (imaginary) optimization rule that gets rid of AbsoluteValue on known-positive integers can kick in, and we can delete the invocation of AbsoluteValue. Yay, optimization!

But a couple of questions remain, at least for me:

Where/when in the compiler pipeline do we insert and remove these type refinements?
Do we need to refine after every conditional?
Do we need to implement the whole into-SSI and out-of-SSI algorithms from all the complicated-looking papers?

We’ll go through them, starting with the compiler pipeline.

The original SSI paper starts with (I think?) SSA form and places some number of new refinement nodes based on conditionals. I have admittedly not tried very hard, but the into-SSI algorithms look complicated and kind of heavyweight. As a reward, you get “linear” into-SSI time complexity.

But I am a humble compiler engineer, and I don’t have the time to go through and load all of this into my head. Instead what I have seen done and have been doing is to take a shortcut: build partial SSI during SSA construction².

Most of the time this is from bytecode, but it could also be from some other non-SSA IR. In any case, this is an excellent shortcut for two reasons:

It lets me cleanly separate adding the type refinements (pretty straightforward) from the hard part of doing all of the operand rewriting and phi placement and marking and all manner of other nonsense.
In addition to separating the concerns, the hard part is already done by SSA construction. We can actually just skip it! SSA construction handles phi placement, operand rewriting, all of it. It probably fits neatly into a naive or a Braun-style (PDF) construction.

This is pretty compelling. We can learn from the bytecode with a very small amount of marginal new complexity. See my implementation in ZJIT, for example. All it really does is modify the abstract interpreter state when building SSA out of branchnil, branchif, and branchunless bytecode instructions to take into account the new refined values.

This is fine for branches that are already in the user’s source program but sometimes optimization, especially of dynamic languages, adds new branches that were not there before. And sometimes these branches get added much later, long after SSA construction. What then? Can we do something similar and rely on existing infrastructure?

During SSA optimization

Implicit in this “can we do it” is the assumption that your IR tracks data dependencies from use to corresponding def, but not from def to uses. Sea of Nodes (at least the Simple implementation), is an IR that tracks both directions all the time for easier rewriting. Many IRs do not do this, so we will continue assuming that there’s no “easy way out”.

JIT optimization of dynamic language compilers often adds synthetic Guard instructions to the IR that enforce pre-conditions. These guards allow optimizing happy/fast path cases in JIT code while leaving the interpreter as a fallback. For example, we might be able to optimize two back-to-back setinstancevariable instructions (a very dynamic operation in the world of ideas, but fast when concretely implemented using object shapes) from:

x = ...
setinstancevariable x, :@a, 1
setinstancevariable x, :@b, 2
# ... use x somewhere ...

which is very generic and involves calling into C code that might raise an exception, to something more like:

x = ...
v0 = GuardHeapObject x
v1 = GuardShape v0, 0xcafe
v2 = Const 1
StoreField v1, 0x8, v2
v3 = GuardHeapObject x
v4 = GuardShape v3, 0xcafe
v5 = Const 2
StoreField v4, 0x10, v5
# ... use x somewhere ...

which is much faster (assuming shape stability at run-time). There’s an irritating problem, though, which is that we have a bunch of duplicate instructions littered around the IR now because our optimizer worked on each instruction individually. Kind of a “template optimizer” situation. Now we need some pass to clean up the detritus.

Global value numbering (GVN) will do a good job of de-duplicating instructions. It should notice that we already have an instruction that looks like GuardHeapObject x called v0 and rewrite v3 into v3 = v0. That’s great because we have de-duplicated the guard. GVN may not get everything, though; if some instructions later use x, they will not get rewritten to instead use the output of these new guard instructions. To do that, we need to add some kind of canonicalize pass or augment GVN with some canonicalization feature. That canonicalization would handle rewriting operands to use the “latest version” of some value, so to speak. See the canonicalization section of Chris Fallin’s excellent aegraphs blog post for more (and of course the (currently block-local) implementation in ZJIT).

def canonicalize(bb)
  rewrite_map = {}
  bb.map do |i|
    i.map_operands! do |o|
      rewrite_map[o] || o
    end
    if i.opcode == :guardtype
      rewrite_map[i.operands[0]] = i
    end
    i
  end
end

Where I’m going with all of this, though, is that you may already have some dominance-based instruction rewriting mechanism in your compiler, either as part of GVN or separately! And you can use this to do a very low code into-partial-SSI in the middle of your optimizer.

This means you could very well get away with inserting RefineType instructions in successor blocks of conditionals and get the into-SSI “for free”.

After which conditionals do we refine?

That’s up to you. There’s a trade-off between compile-time and run-time, especially in JITs. Inserting more instructions and rewriting more times may slow down your compiler. It’s a cheap lunch, not a free one.

How does this compare to the complicated looking papers?

I don’t know. I don’t have a good grasp of how this “partial SSI” compares to the “full SSI”. I don’t plan on implementing full SSI in the near future.

I will note that this partial SSI approach doesn’t do two things:

It doesn’t split variables with a new sigma node, and it generally inserts the refine node within the target block rather than above the branch
(For canonicalize only) It doesn’t insert new phi nodes; it just leaves both IR nodes available and, instead of re-merging, drops them

I can’t tell what impact this has.

In other compilers

Like Simple, TruffleRuby is built on a Sea of Nodes IR (Graal). Chris Seaton has an excellent blog post about TruffleRuby’s use of “stamp nodes” (“Pi nodes”³). The replaceAtUsagesAndDelete function does a lot of heavy lifting, I think because Graal tracks uses.

Cinder mostly inserts RefineType instructions in the HIR builder, before into-SSA, and then lets the SSA construction take care of things. That’s where I learned this trick, actually. Here is one example of refining the type of the matched operand when building IR for pattern matching.

Luau is working on something like this, but for their type checker. Chatting with someone on their team is actually part of the reason I got motivated to write this post.

Android ART looks like it has HBoundType and inserts them in reference type propagation. This handles class checks, null checks, and instanceof checks.

Aside: logic for e.g. HeapObject upgrade

Last, I want to talk a little bit about some interesting reasoning you can do when you have two implementations of something that you can switch between. For example, JIT (+ interpreter), or aliasing and non-aliasing cases in C code, or the weirdo NULL-UB reasoning LLVM can do to C code, things like that.

In ZJIT, we currently insert RefineTypes opportunistically in “easy” cases when building our HIR from the interpreter bytecode.

For example, if in the bytecode there is a branch that compares some value x with nil, it will have two outgoing control-flow edges: one block where x is definitely nil, and one block where x is definitely not nil. In each of these control-flow edges, we can insert corresponding type refinement hints. That’s pretty standard. But we can also do weirder stuff.

CRuby has a notion of heap objects vs immediate objects. Many (most?) objects are heap objects. However, integer 5, for example is not allocated on the heap but instead represented by a tagged bit pattern that pretends to be an address: the whole value is encoded in the pointer itself.

We encode this knowledge in the HIR’s type system: “heapness” and “immediateness” each get a bit in the type lattice. We use this in the optimizer to reason about effects, among other things.

We can’t know a lot of the time what type a thing is, so we pessimistically type most objects flowing through bytecode as BasicObject. This type encapsulates the entire world of possible values that could go on the stack or in a local variable.

On most heap objects, with only a few exceptions, you can write instance variables (fields, attributes, whatever you want to call them). You can never write an instance variable to an immediate. This means that if we observe the following pattern in the bytecode:

x: BasicObject = ...
setinstancevariable x, :@abc, 1

Then after building and emitting HIR for the setinstancevariable opcode, we can upgrade the type of x from a BasicObject to a HeapBasicObject. We can do this because if it weren’t a heap-allocated object, we would have left the compiled code and entered the interpreter.

This is another SSI-type thing you can do in your compiler.

Conclusion

Uhh I guess the conclusion is that you don’t have to do full SSI and partial SSI is available and not too scary? Does your compiler do this? Reader, please write in.

…and optimized in 2002 (PDF), revisited in 2009 (PDF), implemented in LLVM in 2010 (PDF), investigated in 2017 for abstract compilation (PDF), and probably more. The 2009 paper by Boissinot, Brisk, Darte, and Rastello even shows that both Ananian and Singer’s papers have bugs, while perhaps unintentionally also making an excellent pun about the literature being “sparse”. ↩
This blog post is different than the what the LLVM paper (PDF) calls partial SSI. Partial for different reasons. Maybe it’s not even single information anymore. ↩
Today I learned that this terminology comes from the ABCD paper (PDF). ↩

Value numbering

Sat, 04 Apr 2026 00:00:00 +0000

Welcome back to compiler land. Today we’re going to talk about value numbering, which is like SSA, but more.

Static single assignment (SSA) gives names to values: every expression has a name, and each name corresponds to exactly one expression. It transforms programs like this:

x = 0
x = x + 1
x = x + 1

where the variable x is assigned more than once in the program text, into programs like this:

v0 = 0
v1 = v0 + 1
v2 = v1 + 1

where each assignment to x has been replaced with an assignment to a new fresh name.

It’s great because it makes clear the differences between the two x + 1 expressions. Though they textually look similar, they compute different values. The first computes 1 and the second computes 2. In this example, it is not possible to substitute in a variable and re-use the value of x + 1, because the xs are different.

But what if we see two “textually” identical instructions in SSA? That sounds much more promising than non-SSA because the transformation into SSA form has removed (much of) the statefulness of it all. When can we re-use the result?

Identifying instructions that are known at compile-time to always produce the same value at run-time is called value numbering.

Eliminating common subexpressions

To understand value numbering, let’s extend the above IR snippet with two more instructions, v3 and v4.

v0 = 0
v1 = v0 + 1
v2 = v1 + 1
v3 = v0 + 1  # new
v4 = do_something(v2, v3)  # new

In this new snippet, v3 looks the same as v1: adding v0 and 1. Assuming our addition operation is some ideal mathematical addition, we can absolutely re-use v1; no need to compute the addition again. We can rewrite the IR to something like:

v0 = 0
v1 = v0 + 1
v2 = v1 + 1
v3 = v1
v4 = do_something(v2, v3)

This is kind of similar to the destructive union-find representation that JavaScriptCore and a couple other compilers use, where the optimizer doesn’t eagerly re-write all uses but instead leaves a little breadcrumb Identity/Assign instruction¹.

We could then run our copy propagation pass (“union-find cleanup”?) and get:

v0 = 0
v1 = v0 + 1
v2 = v1 + 1
v4 = do_something(v2, v1)

Great. But how does this happen? How does an optimizer identify reusable instruction candidates that are “textually identical”? Generally, there is no actual text in the IR.

One popular solution is to compute a hash of each instruction. Then any instructions with the same hash (that also compare equal, in case of collisions) are considered equivalent. This is called hash-consing.

When trying to figure all this out, I read through a couple of different implementations. I particularly like the Maxine VM implementation. For example, here is the valueNumber (hashing) and valueEqual functions for most binary operations, slightly modified for clarity:

public abstract class Instruction extends Value { ... }

// The base class for binary operations
public abstract class Op2 extends Instruction {
    // Each binary operation has an opcode and two opearands
    public final int opcode;  // (IMUL, IADD, ...)
    Value x;
    Value y;

    @Override
    public int valueNumber() {
        // There are other fields but only opcode, and operands get hashed.
        // Always set at least one bit in case the hash wraps to zero.
        return 0x20000000
        | (opcode
           + 7  * System.identityHashCode(x)
           + 11 * System.identityHashCode(y));
    }

    @Override
    public boolean valueEqual(Instruction i) {
        if (i instanceof Op2) {
            Op2 o = (Op2) i;
            return opcode == o.opcode && x == o.x && y == o.y;
        }
        return false;
    }
}

The rest of the value numbering implementation assumes that if a valueNumber function returns 0, it does not wish to be considered for value numbering. Why might an instruction opt-out of value numbering?

Pure vs impure

An instruction might opt out of value numbering if it is not “pure”.

Some instructions are not pure. Purity is in the eye of the beholder, but in general it means that an instruction does not interact with the state of the outside world, except for trivial computation on its operands. (What does it mean to de-duplicate/cache/reuse printf?)

A load from an array object is also not a pure operation². The load operation implicitly relies on the state of the memory. Also, even if the array was known-constant, in some runtime systems, the load might raise an exception. Changing the source location where an exception is raised is generally frowned upon. Languages such as Java often have requirements about where exceptions are raised codified in their specifications.

We’ll work only on pure operations for now, but we’ll come back to this later. We do often want to optimize impure operations as well!

We’ll start off with the simplest form of value numbering, which operates only on linear sequences of instructions, like basic blocks or traces.

Local value numbering

Let’s build a small implementation of local value numbering (LVN). We’ll start with straight-line code—no branches or anything tricky.

Most compiler optimizations on control-flow graphs (CFGs) iterate over the instructions “top to bottom”³ and it seems like we can do the same thing here too.

From what we’ve seen so far optimizing our made-up IR snippet, we can do something like this:

initialize a map from instruction numbers to instruction pointers
for each instruction i
- if i wants to participate in value numbering
  - if i’s value number is already in the map, replace all pointers to i in the rest of the program with the corresponding value from the map
  - otherwise, add i to the map

The find-and-replace, remember, is not a literal find-and-replace, but instead something like:

instr.opcode = "Assign"
instr.operands[0] = replacement

instr.make_equal_to(replacement)

(if you have been following along with the toy optimizer series)

This several-line function (as long as you already have a hash map and a union-find available to you) is enough to build local value numbering! And real compilers are built this way, too.

If you don’t believe me, take a look at this slightly edited snippet from Maxine’s value numbering implementation. It has all of the components we just talked about: iterating over instructions, map lookup, and some substitution.

// Local value numbering
BlockBegin block = ...;
ValueMap currentMap = new ValueMap();
InstructionSubstituter subst = new InstructionSubstituter();

// visit all instructions of this block
for (Instruction instr = block.next(); instr != null; instr = instr.next()) {
    // attempt value numbering (uses valueNumber() and valueEqual())
    //
    // return a previous instruction if it exists in the map, or insert the
    // current instruction into the map and return it
    Instruction f = currentMap.findInsert(instr);
    if (f != instr) {
        // remember the replacement in the union-find
        subst.setSubst(instr, f);
    }
}

This alone will get you pretty far. Code generators of all shapes tend to leave messy repeated computations all over their generated code and this will make short work of them.

Sometimes, though, your computations are spread across control flow—over multiple basic blocks. What do you do then?

Global value numbering

Computing value numbers for an entire function is called global value numbering (GVN) and it requires dealing with control flow (if, loops, etc). I don’t just mean that for an entire function, we run local value numbering block-by-block. Global value numbering implies that expressions can be de-duplicated and shared across blocks.

Let’s tackle control flow case by case.

First is the simple case from above: one block. In this case, we can go top to bottom with our value numbering and do alright.

The second case is also reasonable to handle: one block flowing into another. In this case, we can still go top to bottom. We just have to find a way to iterate over the blocks.

If we’re not going to share value maps between blocks, the order doesn’t matter. But since the point of global value numbering is to share values, we have to iterate them in topological order (reverse post order (RPO)). This ensures that predecessors get visited before successors. If you have bb0 -> bb1, we have to visit first bb0 and then bb1.

Because of how SSA works and how CFGs work, the second block can “look up” into the first block and use the values from it. To get global value numbering working, we have to copy bb0’s value map before we start processing bb1 so we can re-use the instructions.

Maybe something like:

value_map = ValueMap()
for block in function.reverse_post_order():
    local_value_numbering(block, value_map)

Then the expressions can accrue across blocks. bb1 can re-use the already-computed Add v0, 1 from bb0 because it is still in the map.

…but this breaks as soon as you have control-flow splits. Consider the following shape graph:

We’re going to iterate over that graph in one of two orders: A B C or A C B. In either case, we’re going to be adding all this stuff into the value map from one block (say, B) that is not actually available to its sibling block (say, C).

When I say “not available”, I mean “would not have been computed before”. This is because we execute either A then B or A then C. There’s no world in which we execute B then C.

But alright, look at a third case where there is such a world: a control-flow join. In this diagram, we have two predecessor blocks B and C each flowing into D. In this diagram, B always flows into D and also C always flows into D. So the iterator order is fine, right?

Well, still no. We have the same sibling problem as before. B and C still can’t share value maps.

We also have a weird question when we enter D: where did we come from? If we came from B, we can re-use expressions from B. If we came from C, we can re-use expressions from C. But we cannot in general know which predecessor block we came from.

The only block we know for sure that we executed before D is A. This means we can re-use A’s value map in D because we can guarantee that all execution paths that enter D have previously gone through A.

This relationship is called a dominator relationship and this is the key to one style of global value numbering that we’re going to talk about in this post. A block can always use the value map from any other block that dominates it. For completeness’ sake, in the diamond diagram, A dominates each of B and C, too.

We can compute dominators a couple of ways⁴, but that’s a little bit out of scope for this blog post. If we assume that we have dominator information available in our CFG, we can use that for global value numbering. And that’s just what—you guessed it—Maxine VM does.

It iterates over all blocks in reverse post-order, doing local value numbering, threading through value maps from dominator blocks. In this case, their method dominator gets the immediate dominator: the “closest” dominator block of all the blocks that dominate the current one.

public class GlobalValueNumberer {
    final HashMap<BlockBegin, ValueMap> valueMaps;
    final InstructionSubstituter subst;
    ValueMap currentMap;

    public GlobalValueNumberer(IR ir) {
        this.subst = new InstructionSubstituter(ir);
        // reverse post-order
        List<BlockBegin> blocks = ir.linearScanOrder();
        valueMaps = new HashMap<BlockBegin, ValueMap>(blocks.size());
        optimize(blocks);
        subst.finish();
    }

    void optimize(List<BlockBegin> blocks) {
        int numBlocks = blocks.size();
        BlockBegin startBlock = blocks.get(0);

        // initial value map, with nesting 0
        valueMaps.put(startBlock, new ValueMap());

        for (int i = 1; i < numBlocks; i++) {
            // iterate through all the blocks
            BlockBegin block = blocks.get(i);
            BlockBegin dominator = block.dominator();

            // create new value map with increased nesting
            currentMap = new ValueMap(valueMaps.get(dominator));

            // << INSERT LOCAL VALUE NUMBERING HERE >>

            // remember value map for successors
            valueMaps.put(block, currentMap);
        }
    }
}

And that’s it! That’s the core of Maxine’s GVN implementation. I love how short it is. For not very much code, you can remove a lot of duplicate pure SSA instructions.

This does still work with loops, but with some caveats. From p7 of Briggs GVN:

The φ-functions require special treatment. Before the compiler can analyze the φ-functions in a block, it must previously have assigned value numbers to all of the inputs. This is not possible in all cases; specifically, any φ-function input whose value flows along a back edge (with respect to the dominator tree) cannot have a value number. If any of the parameters of a φ-function have not been assigned a value number, then the compiler cannot analyze the φ-function, and it must assign a unique, new value number to the result.

It also talks about eliminating useless phis, which is optional, but would the strengthen global value numbering pass: it makes more information transparent.

But what if we want to handle impure instructions?

State management and invalidation

Languages such as Java allow for reading fields from the this/self object within methods as if the field were a variable name. This makes code like the following common:

class CPU {
    private void exec_adc() {
        int result_int = regA + fetched_data + flagCARRY;
        byte result = (byte) result_int;
        // ...
        int a = result_int ^ regA;
        int b = result_int ^ fetched_data;
        // ...
        regA = result;
    }
}

Each of these reference to regA and fetched_data is an implicit reference to this.regA or this.fetched_data, which is semantically a field load off an object. You can see it in the bytecode (thanks, Matt Godbolt):

class CPU {
  int regA;

  int fetched_data;

  int flagCARRY;

  CPU();
         0: aload_0
         1: invokespecial #1                  // Method java/lang/Object."<init>":()V
         4: return


  private void exec_adc();
         0: aload_0
         1: getfield      #7                  // Field regA:I
         4: aload_0
         // ...
        20: getfield      #7                  // Field regA:I
        23: ixor
        24: istore_3
        25: iload_1
        26: aload_0
        27: getfield      #13                 // Field fetched_data:I
        30: ixor
        31: istore        4
        33: aload_0
        34: iload_2
        35: putfield      #7                  // Field regA:I
        38: return
}

When straightforwardly building an SSA IR from the JVM bytecode for this method, you will end up with a bunch of IR that looks like this:

v0 = LoadField self, :regA
v1 = LoadField self, :fetched_data
v2 = LoadField self, :flagCARRY
v3 = IntAdd v0, v1
v4 = IntAdd v3, v2
// ...
v7 = LoadField self, :regA
v8 = IntXor v4, v7
v9 = LoadField self, :fetched_data
v10 = IntXor v4, v9
// ...
StoreField self, :regA, ...

Pretty much the same as the bytecode. Even though no code in the middle could modify the field regA (which would require a re-load), we still have a duplicate load. Bummer.

I don’t want to re-hash this too much but it’s possible to fold Load and store forwarding into your GVN implementation by either:

doing load-store forwarding as part of local value numbering and clearing memory information from the value map at the end of each block, or
keeping track of effects across blocks

See, there’s nothing fundamentally stopping you from tracking the state of your heap at compile-time across blocks. You just have to do a little more bookkeeping. In our dominator-based GVN implementation, for example, you can:

track heap write effects for each block
at the start of each block B, union all of the “kill” sets for every block back to its immediate dominator
finally, remove the stuff that got killed from the dominator’s value map

Not so bad.

Maxine doesn’t do global memory tracking, but they do a limited form of load-store forwarding while building their HIR from bytecode: see GraphBuilder which uses the MemoryMap to help track this stuff. At least they would not have the same duplicate LoadField instructions in the example above!

We’ve now looked at one kind of value numbering and one implementation of it. What else is out there?

Out in the world

Apparently, you can get better results by having a unified hash table (p9 of Briggs GVN) of expressions, not limiting the value map to dominator-available expressions. Not 100% on how this works yet. They note:

Using a unified hash-table has one important algorithmic consequence. Replacements cannot be performed on-line because the table no longer reflects availability.

Which is the first time that it occurred to me that hash-based value numbering with dominators was an approximation of available expression analysis.

There’s also a totally different kind of value numbering called value partitioning (p12 of Briggs GVN). See also a nice blog post about this by Allen Wang from the Cornell compiler course. I think this mostly replaces the hashing bit, and you still need some other thing for the available expressions bit.

Ben Titzer and Seth Goldstein have some good slides from CMU. Where they talk about the worklist dataflow approach. Apparently this is slower but gets you more available expressions than just looking to dominator blocks. I wonder how much it differs from dominator+unified hash table.

While Maxine uses hash table cloning to copy value maps from dominator blocks, there are also compilers such as Cranelift that use scoped hash maps to track this information more efficiently. (Though Amanieu notes that you may not need a scoped hash map and instead can tag values in your value map with the block they came from, ignoring non-dominating values with a quick check. The dominance check makes sense but I haven’t internalized how this affects the set of available expressions yet.)

You may be wondering if this kind of algorithm even helps at all in a dynamic language JIT context. Surely everything is too dynamic, right? Actually, no! The JIT hopes to eliminate a lot of method calls and dynamic behaviors, replacing them with guards, assumptions, and simpler operations. These strength reductions often leave behind a lot of repeated instructions. Just the other day, Kokubun filed a value-numbering-like PR to clean up some of the waste.

ART has a recent blog post about speeding up GVN.

Implementations

Wrapping up; bits and bobbles

Go forth and give your values more numbers.

There’s been an ongoing discussion with Phil Zucker on SSI, GVN, acyclic egraphs, and scoped union-find. TODO summarize

Acyclic e-graphs

Commutativity; canonicalization

Seeding alternative representations into the GVN

Aegraphs and union-find during GVN https://cfallin.org/blog/2026/04/09/aegraph/ canonicalize

https://github.com/bytecodealliance/rfcs/blob/main/accepted/cranelift-egraph.md

https://github.com/bytecodealliance/wasmtime/issues/9049

https://github.com/bytecodealliance/wasmtime/issues/4371

Partial redundancy elimination

Writing this post is roughly the time when I realized that the whole time I was wondering why Cinder did not use union-find for rewriting, it actually did! Optimizing instruction X = A + 0 by replacing with X = Assign A followed by copy propagation is equivalent to union-find. ↩
In some forms of SSA, like heap-array SSA or sea of nodes, it’s possible to more easily de-duplicate loads because the memory representation has been folded into (modeled in) the IR. ↩
The order is a little more complicated than that: reverse post-order (RPO). And there’s a paper called “A Simple Algorithm for Global Data Flow Analysis Problems” that I don’t yet have a PDF for that claims that RPO is optimal for solving dataflow problems. ↩
There’s the iterative dataflow way (described in the Cooper paper (PDF)), Lengauer-Tarjan (PDF), the Engineered Algorithm (PDF), hybrid/Semi-NCA approach (PDF), … ↩

Using Perfetto in ZJIT

Fri, 27 Mar 2026 00:00:00 +0000

Originally published on Rails At Scale.

Look! A trace of slow events in a benchmark! Hover over the image to see it get bigger.

A sneak preview of what the trace looks like.

Now read on to see what the slow events are and how we got this pretty picture.

The rules

The first rule of just-in-time compilers is: you stay in JIT code. The second rule of JIT is: you STAY in JIT code!

When control leaves the compiled code to run in the interpreter—what the ZJIT team calls either a “side-exit” or a “deopt”, depending on who you talk to—things slow down. In a well-tuned system, this should happen pretty rarely. Right now, because we’re still bringing up the compiler and runtime system, it happens more than we would like.

We’re reducing the number of exits over time.

Lies, damned lies, and statistics

We can track our side-exit reduction progress with --zjit-stats, which, on process exit, prints out a tidy summary of the counters for all of the bad stuff we track. It’s got side-exits. It’s got calls to C code. It’s got calls to slow-path runtime helpers. It’s got everything.

Here is a chopped-up sample of stats output for the Lobsters benchmark, which is a large Rails app:

$ WARMUP_ITRS=0 MIN_BENCH_ITRS=20 MIN_BENCH_TIME=0 ruby --zjit-stats benchmarks/lobsters/benchmark.rb
...
***ZJIT: Printing ZJIT statistics on exit***
...
Top-20 side exit reasons (100.0% of total 12,549,876):
                   guard_type_failure: 6,020,734 (48.0%)
                  guard_shape_failure: 5,556,147 (44.3%)
  block_param_proxy_not_iseq_or_ifunc:   445,358 ( 3.5%)
                   unhandled_hir_insn:   215,168 ( 1.7%)
                        compile_error:   181,474 ( 1.4%)
...
compiled_iseq_count:                               5,581
failed_iseq_count:                                     2
compile_time:                                    1,443ms
...
guard_type_count:                            133,425,094
guard_type_exit_ratio:                              4.5%
guard_shape_count:                            49,386,694
guard_shape_exit_ratio:                            11.3%
...
code_region_bytes:                            31,571,968
side_exit_size_ratio:                              33.1%
zjit_alloc_bytes:                             19,329,659
total_mem_bytes:                              50,901,627
...
ratio_in_zjit:                                     82.8%
$

(I’ve cut out significant chunks of the stats output and replaced them with ... because it’s overwhelming the first time you see it.)

The first thing you might note is that the thing I just described as terrible for performance is happening over twelve million times. The second thing you might notice is that despite this, we’re staying in JIT code seemingly a high percentage of the time. Or are we? Is 80% high? Is a 4.5% class guard miss ratio high? What about 11% for shapes? It’s hard to say.

The counters are great because they’re quick and they’re reasonably stable proxies for performance. There’s no substitute for painstaking measurements on a quiet machine but if the counter for Bad Slow Thing goes down (and others do not go up), we’re probably doing a good job.

But they’re not great for building intuition. For intuition, we want more tangible feeling numbers. We want to see things.

Building intuition

The third thing is that you might ask yourself “self, where are these exits coming from?” Unfortunately, counters cannot tell you that. For that, we want stack traces. This lets us know where in the guest (Ruby) code triggers an exit.

Ideally also we would want some notion of time: we would want to know not just where these events happen but also when. Are the exits happening early, at application boot? At warmup? Even during what should be steady state application time? Hard to say.

So we need more tools. Thankfully, Perfetto exists. Perfetto is a system for visualizing and analyzing traces and profiles that your application generates. It has both a web UI and a command-line UI.

We can emit traces for Perfetto and visualize them there.

A look at Perfetto

Take a look at this sample ZJIT Perfetto trace generated by running Ruby with --zjit-trace-exits¹. What do you see?

I see a couple arrows on the left. Arrows indicate “instant” point-in-time events. Then I see a mess of purple to the right of that until the end of the trace.

Hover over an arrow. Find out that each arrow is a side-exit. Scream silently.

But it’s a friendly arrow. It tells you what the side-exit reason is. If you click it, it even tells you the stack trace in the pop-up panel on the bottom. If we click a couple of them, maybe we can learn more.

We can also zoom by mousing over the track, holding Ctrl, and scrolling. That will get us look closer. But there are so many…

Fortunately, Perfetto also provides a SQL interface to the traces. We can write a query to aggregate all of the side exit events from the slice table and line them up with the topmost method from the backtrace arguments in the args table:

SELECT
  s.name AS reason,
  a.display_value AS method,
  COUNT(*) AS count
FROM slice s
JOIN args a ON a.arg_set_id = s.arg_set_id AND a.key = '0'
GROUP BY s.name, a.display_value
ORDER BY count DESC

This pulls up a query box at the bottom showing us that there are a couple big hotspots:

Query results showing in columns left to right: reason for side-exit, method that exited, and count. The top three are above 1k but it quickly falls off after that.

It even has a helpful option to export the results Markdown table so I can paste (an edited version) into this blog post:

reason	method	count
GuardShape(ShapeId(2475))	ActiveModel::AttributeRegistration::ClassMethods#attribute_types	5119
GuardShape(ShapeId(2099268))	ActiveRecord::ConnectionAdapters::AbstractAdapter#extended_type_map_key	2295
GuardType(FalseClass)	ActiveModel::Type::Value#cast	1025
GuardShape(ShapeId(2099698))	ActiveRecord::Associations#association_instance_get	904
BlockParamProxyNotIseqOrIfunc	ActiveRecord::AttributeMethods::Read#_read_attribute	902
GuardShape(ShapeId(526450))	Rack::Request::Env#get_header	636
GuardType(Class[class_exact*:Class@VALUE(0x128c60100)])	ActiveRecord::Base._reflections	622
GuardType(ObjectSubclass[class_exact:Story])	ActiveRecord::Associations#association	565
GuardShape(ShapeId(2098982))	ActiveRecord::Reflection::AssociationReflection#polymorphic?	510
GuardType(StringSubclass[class_exact:ActiveSupport::SafeBuffer])	ActionView::OutputBuffer#<<	500
GuardShape(ShapeId(2475))	ActiveRecord::AttributeMethods::PrimaryKey::ClassMethods#primary_key	492
GuardType(ObjectSubclass[class_exact:ActiveModel::Type::String])	ActiveModel::Type::Value#deserialize	442
GuardShape(ShapeId(2098982))	ActiveRecord::Reflection::AssociationReflection#deprecated?	376
GuardType(ObjectSubclass[class_exact:Bundler::Dependency])	Gem::Dependency#matches_spec?	355
UnhandledHIRInvokeBuiltin	Time#initialize	346

Looks like we should figure out why we’re having shape misses so much and that will clear up a lot of exits. (Hint: it’s because once we make our first guess about what we think the object shape will be, we don’t re-assess… yet.)

This has been a taste of Perfetto. There’s probably a lot more to explore. Please join the ZJIT Zulip and let us know if you have any cool tracing or exploring tricks.

Now I’ll explain how you too can use Perfetto from your system. Adding support to ZJIT was pretty straightforward.

Implementation

The first thing is that you’ll need some way to get trace data out of your system. We write to a file with a well-known location (/tmp/perfetto-PID.fxt), but you could do any number of things. Perhaps you can stream events over a socket to another process, or to a server that aggregates them, or store them internally and expose a webserver that serves them over the internet, or… anything, really.

Once you have that, you need a couple lines of code to emit the data. Perfetto accepts a number of formats. For example, in his excellent blog post, Tristan Hume opens with such a simple snippet of code for logging Chromium Trace JSON-formatted events (lightly modified by me):

event_name = ...
timestamp = ...
duration = ...
f = open('trace.json','a')
f.write("[\n")

# ... emit some events here ...

# Log a single event
f.write('{"name": "%s", "ts": %d, "dur": %d, "cat": "hi", "ph": "X", "pid": 1, "tid": 1, "args": {}},\n' %
  (event_name, timestamp, duration))

# ... emit some events here ...

# ... at process exit, close the file ...
f.write("]") # this closing ] isn't actually required
f.close()

This snippet is great. It shows, end-to-end, writing a stream of one event. It is a complete (X) event, as opposed to either:

two discrete timestamped begin (B) and end (E) events that book-end something, or
an instant (i) event that has no duration, or
a couple other event types in the Chromium Trace Event Format doc

It was enough to get me started. Since it’s JSON, and we have a lot of side exits, the trace quickly ballooned to 8GB large for a several second benchmark. Not great. Now, part of this is our fault—we should side exit less—and part of it is just the verbosity of JSON.

Thankfully, Perfetto ingests more compact binary formats, such as the Fuchsia trace format. In addition to being more compact, FXT even supports string interning. After modifying the tracer to emit FXT, we ended with closer to 100MB for the same benchmark.

We can reduce further by sampling—not writing every exit to the trace, but instead every K exits (for some (probably prime) K). This is why we provide the --zjit-trace-exits-sample-rate=K option.

Check out the trace writer implementation from the point this article was written.

Tracing more things

We could trace:

When methods get compiled
How big the generated code is
How long each compile phase takes
When (and where) invalidation events happen
When (and where) allocations happen from JITed code
Garbage collection events
and more!

Conclusion

Visualizations are awesome. Get your data in the right format so you can ask the right questions easily. Thanks for Perfetto!

Also, looks like visualizations are now available in Perfetto canary. Time to go make some fun histograms…

This is also sampled/strobed, so not every exit is in there. This is just 1/K of them for some K that I don’t remember. ↩

A fuzzer for the Toy Optimizer

Wed, 25 Feb 2026 00:00:00 +0000

Another entry in the Toy Optimizer series.

It’s hard to get compiler optimizers right. Even if you build up a painstaking test suite by hand, you will likely miss corner cases, especially corner cases at the interactions of multiple components or multiple optimization passes.

I wanted to see if I could write a fuzzer to catch some of these bugs automatically. But a fuzzer alone isn’t much use without some correctness oracle—in this case, we want a more interesting bug than accidentally crashing the optimizer. We want to see if the optimizer introduces a correctness bug in the program.

So I set off in the most straightforward way possible, inspired by my hazy memories of a former CF blog post.

Generating programs

Generating random programs isn’t so bad. We have program generation APIs and we can dynamically pick which ones we want to call. I wrote a small loop that generates loads from and stores to the arguments at random offsets and with random values, and escapes to random instructions with outputs. The idea with the escape is to keep track of the values as if there was some other function relying on them.

def generate_program():
    bb = Block()
    args = [bb.getarg(i) for i in range(3)]
    num_ops = random.randint(0, 30)
    ops_with_values = args[:]
    for _ in range(num_ops):
        op = random.choice(["load", "store", "escape"])
        arg = random.choice(args)
        a_value = random.choice(ops_with_values)
        offset = random.randint(0, 4)
        if op == "load":
            v = bb.load(arg, offset)
            ops_with_values.append(v)
        elif op == "store":
            value = random.randint(0, 10)
            bb.store(arg, offset, value)
        elif op == "escape":
            bb.escape(a_value)
        else:
            raise NotImplementedError(f"Unknown operation {op}")
    return bb

This generates random programs. Here is an example stringified random program:

var0 = getarg(0)
var1 = getarg(1)
var2 = getarg(2)
var3 = load(var2, 0)
var4 = load(var0, 1)
var5 = load(var1, 1)
var6 = escape(var0)
var7 = store(var0, 2, 3)
var8 = store(var2, 0, 7)

No idea what would generate something like this, but oh well.

Verifying programs

Then we want to come up with our invariants. I picked the invariant that, under the same preconditions, the heap will look the same after running an optimized program as it would under an un-optimized program¹. So we can delete instructions, but if we don’t have a load-bearing store, store the wrong information, or cache stale loads, we will probably catch that.

def verify_program(bb):
    before_no_alias = interpret_program(bb, ["a", "b", "c"])
    a = "a"
    before_alias = interpret_program(bb, [a, a, a])
    optimized = optimize_load_store(bb)
    after_no_alias = interpret_program(optimized, ["a", "b", "c"])
    after_alias = interpret_program(optimized, [a, a, a])
    assert before_no_alias == after_no_alias
    assert before_alias == after_alias

I have a very silly verifier that tests two cases: one where the arguments do not alias and one where they are all the same object. Generating partial aliases would be a good extension here.

Last, we have the interpreter.

Running programs

The interpreter is responsible for keeping track of the heap (as indexed by (object, offset) pairs) as well as the results of the various instructions.

We keep track of the escaped values so we can see results of some instructions even if they do not get written back to the heap. Maybe we should be escapeing all instructions with output instead of only random ones. Who knows.

def interpret_program(bb, args):
    heap = {}
    ssa = {}
    escaped = []
    for op in bb:
        if op.name == "getarg":
            ssa[op] = args[get_num(op, 0)]
        elif op.name == "store":
            obj = ssa[op.arg(0)]
            offset = get_num(op, 1)
            value = get_num(op, 2)
            heap[(obj, offset)] = value
        elif op.name == "load":
            obj = ssa[op.arg(0)]
            offset = get_num(op, 1)
            value = heap.get((obj, offset), "unknown")
            ssa[op] = value
        elif op.name == "escape":
            value = op.arg(0)
            if isinstance(value, Constant):
                escaped.append(value.value)
            else:
                escaped.append(ssa[value])
        else:
            raise NotImplementedError(f"Unknown operation {op.name}")
    heap["escaped"] = escaped
    return heap

Then we return the heap so that the verifier can check.

The harness

Then we run a bunch of random tests through the verifier!

def test_random_programs():
    # Remove random.seed if using in CI... instead print the seed out so you
    # can reproduce crashes if you find them
    random.seed(0)
    num_programs = 100000
    for i in range(num_programs):
        program = generate_program()
        verify_program(program)

The number of programs is configurable. Or you could make this while True. But due to how simple the optimizer is, we will find all the possible bugs pretty quickly.

I initially started writing this post because I thought I had found a bug, but it turns out that I had, with CF’s help, in 2022, walked through every possible case in the “buggy” situation, and the optimizer handles those cases correctly. That explains why the verifier didn’t find that bug!

Testing the verifier

So does it work? If you run it, it’ll hang for a bit and then report no issues. That’s helpful, in a sense… it’s revealing that it is unable to find a certain class of bug in the optimizer.

Let’s comment out the main load-bearing pillar of correctness in the optimizer—removing aliasing writes—and see what happens.

We get a crash nearly instantly:

$ uv run --with pytest pytest loadstore.py -k random
...
=========================================== FAILURES ============================================
_____________________________________ test_random_programs ______________________________________

    def test_random_programs():
        random.seed(0)
        num_programs = 100000
        for i in range(num_programs):
            program = generate_program()
>           verify_program(program)

loadstore.py:617:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

bb = [Operation(getarg, [Constant(0)], None, None), Operation(getarg, [Constant(1)], None, None), Operation(getarg, [Consta...], None, None)], None, None), Operation(load, [Operation(getarg, [Constant(0)], None, None), Constant(0)], None, None)]

    def verify_program(bb):
        before_no_alias = interpret_program(bb, ["a", "b", "c"])
        a = "a"
        before_alias = interpret_program(bb, [a, a, a])
        optimized = optimize_load_store(bb)
        after_no_alias = interpret_program(optimized, ["a", "b", "c"])
        after_alias = interpret_program(optimized, [a, a, a])
        assert before_no_alias == after_no_alias
>       assert before_alias == after_alias
E       AssertionError: assert {('a', 0): 4,...', 3): 1, ...} == {('a', 0): 9,...', 3): 1, ...}
E
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {('a', 0): 4} != {('a', 0): 9}
E         Use -v to get more diff

loadstore.py:610: AssertionError
==================================== short test summary info ====================================
FAILED loadstore.py::test_random_programs - AssertionError: assert {('a', 0): 4,...', 3): 1, ...} == {('a', 0): 9,...', 3): 1, ...}
=============================== 1 failed, 15 deselected in 0.04s ================================
$

We should probably use bb_to_str(bb) and bb_to_str(optimized) to print out the un-optimized and optimized traces in the assert failure messages. But we get a nice diff of the heap automatically, which is neat. And it points to an aliasing problem!

Full code

See the full code.

Extensions

Synthesize (different) types for non-aliasing objects and add them in info
Shrink/reduce failing examples down for easier debugging
Use Hypothesis for property-based testing, which CF notes also gives you shrinking
Use Z3 to encode the generated programs instead of randomly interpreting them

Thanks

Thank you to CF Bolz-Tereick for feedback on this post!

CF notes that this notion of equivalence works for this optimizer but not for one that does allocation removal (escape analysis). If we removed allocations and writes to them, we would be changing the heap results and our verifier would appear to fail. This means we have to, if we are to delete allocations, pick a more subtle definition of equivalence.

Perhaps something that looks like escape analysis in the verifier’s interpreter? ↩

Type-based alias analysis in the Toy Optimizer

Mon, 16 Feb 2026 00:00:00 +0000

Another entry in the Toy Optimizer series.

Last time, we did load-store forwarding in the context of our Toy Optimizer. We managed to cache the results of both reads from and writes to the heap—at compile-time!

We were careful to mind object aliasing: we separated our heap information into alias classes based on what offset the reads/writes referenced. This way, if we didn’t know if object a and b aliased, we could at least know that different offsets would never alias (assuming our objects don’t overlap and memory accesses are on word-sized slots). This is a coarse-grained heuristic.

Fortunately, we often have much more information available at compile-time than just the offset, so we should use it. I mentioned in a footnote that we could use type information, for example, to improve our alias analysis. We’ll add a lightweight form of type-based alias analysis (TBAA) (PDF) in this post.

Representing types

We return once again to Fil Pizlo land, specifically How I implement SSA form. We’re going to be using the hierarchical heap effect representation from the post in our implementation, but you can use your own type representation if you have one already.

This representation divides the heap into disjoint regions by type. Consider, for example, that Array objects and String objects do not overlap. A LinkedList pointer is never going to alias an Integer pointer. They can therefore be reasoned about separately.

But sometimes you don’t have perfect type information available. If you have in your language an Object base class of all objects, then the Object heap overlaps with, say, the Array heap. So you need some way to represent that too—just having an enum doesn’t work cleanly.

Here is an example simplified type hierarchy:

Any
  Object
    Array
    String
  Other

Where Other might represent different parts of the runtime’s data structures, and could be further segmented into GC, Thread, etc.

Fil’s idea is that we can represent each node in that hierarchy with a tuple of integers [start, end) (inclusive, exclusive) that represent the pre- and post-order traversals of the tree. Or, if tree traversals are not engraved into your bones, they represent the range of all the nested objects within them.

Any [0, 3)
  Object [0, 2)
    Array [0, 1)
    String [1, 2)
  Other [2, 3)

Then the “does this write interfere with this read” check—the aliasing check—is a range overlap query.

Here’s a perhaps over-engineered Python implementation of the range and heap hierarchy based on the Ruby generator and C++ runtime code from JavaScriptCore:

class HeapRange:
    def __init__(self, start: int, end: int) -> None:
        self.start = start
        self.end = end

    def __repr__(self) -> str:
        return f"[{self.start}, {self.end})"

    def is_empty(self) -> bool:
        return self.start == self.end

    def overlaps(self, other: "HeapRange") -> bool:
        # Empty ranges interfere with nothing
        if self.is_empty() or other.is_empty():
            return False
        return self.end > other.start and other.end > self.start


class AbstractHeap:
    def __init__(self, name: str) -> None:
        self.name = name
        self.parent = None
        self.children = []
        self.range = None

    def add_child(self, name: str) -> None:
        result = AbstractHeap(name)
        result.parent = self
        self.children.append(result)
        return result

    def compute(self, start: int) -> None:
        current = start
        if not self.children:
            self.range = HeapRange(start, current + 1)
            return
        for child in self.children:
            child.compute(current)
            current = child.range.end
        self.range = HeapRange(start, current)


Any = AbstractHeap("Any")
Object = Any.add_child("Object")
Array = Object.add_child("Array")
String = Object.add_child("String")
Other = Any.add_child("Other")
Any.compute(0)

Where Any.compute(0) kicks off the tree-numbering scheme.

Fil’s implementation also covers a bunch of abstract heaps such as SSAState and Control because his is used for code motion and whatnot. That can be added on later but we will not do so in this post.

So there you have it: a type representation. Now we need to use it in our load-store forwarding.

Load-store forwarding

Recall that our load-store optimization pass looks like this:

def optimize_load_store(bb: Block):
    opt_bb = Block()
    # Stores things we know about the heap at... compile-time.
    # Key: an object and an offset pair acting as a heap address
    # Value: a previous SSA value we know exists at that address
    compile_time_heap: Dict[Tuple[Value, int], Value] = {}
    for op in bb:
        if op.name == "store":
            obj = op.arg(0)
            offset = get_num(op, 1)
            store_info = (obj, offset)
            current_value = compile_time_heap.get(store_info)
            new_value = op.arg(2)
            if eq_value(current_value, new_value):
                continue
            compile_time_heap = {
                load_info: value
                for load_info, value in compile_time_heap.items()
                if load_info[1] != offset
            }
            compile_time_heap[store_info] = new_value
        elif op.name == "load":
            load_info = (op.arg(0), get_num(op, 1))
            if load_info in compile_time_heap:
                op.make_equal_to(compile_time_heap[load_info])
                continue
            compile_time_heap[load_info] = op
        opt_bb.append(op)
    return opt_bb

At its core, it iterates over the instructions, keeping a representation of the heap at compile-time. Reads get cached, writes get cached, and writes also invalidate the state of compile-time information about fields that may alias.

In this case, our may alias asks only if the offsets overlap. This means that the following unit test will fail:

def test_store_to_same_offset_different_heaps_does_not_invalidate_load():
    bb = Block()
    var0 = bb.getarg(0)
    var0.info = Array
    var1 = bb.getarg(1)
    var1.info = String
    var2 = bb.store(var0, 0, 3)
    var3 = bb.store(var1, 0, 4)
    var4 = bb.load(var0, 0)
    bb.escape(var4)
    opt_bb = optimize_load_store(bb)
    assert (
        bb_to_str(opt_bb)
        == """\
var0 = getarg(0)
var1 = getarg(1)
var2 = store(var0, 0, 3)
var3 = store(var1, 0, 4)
var4 = escape(3)"""
    )

This test is expecting the write to var0 to still remain cached even though we wrote to the same offset in var1—because we have annotated var0 as being an Array and var1 as being a String. If we account for type information in our alias analysis, we can get this test to pass.

After doing a bunch of fussing around with the load-store forwarding (many rewrites), I eventually got it down to a very short diff:

+def may_alias(left: Value, right: Value) -> bool:
+    return (left.info or Any).range.overlaps((right.info or Any).range)
+
+
 def optimize_load_store(bb: Block):
     opt_bb = Block()
     # Stores things we know about the heap at... compile-time.
@@ -138,6 +210,10 @@ def optimize_load_store(bb: Block):
                 load_info: value
                 for load_info, value in compile_time_heap.items()
                 if load_info[1] != offset
+                or not may_alias(load_info[0], obj)
             }
             compile_time_heap[store_info] = new_value

If we don’t have any type/alias information, we default to “I know nothing” (Any) for each object. Then we check range overlap.

The boolean logic in optimize_load_store looks a little weird, maybe. But we can also rewrite (via DeMorgan’s law) as:

{
    ... for ...
    if not (load_info[1] == offset
            and may_alias(load_info[0], obj))
}

So, keeping all the cached field state about fields that are known by offset and by type not to alias. Maybe that is clearer (but not as nice a diff).

Note that the type representation is not so important here! You could use a bitset version of the type information if you want. The important things are that you can cheaply construct types and check overlap between them.

Nice, now our test passes! We can differentiate between memory accesses on objects of different types.

But what if we knew more?

Object provenance / allocation site

Sometimes we know where an object came from. For example, we may have seen it get allocated in the trace. If we saw an object’s allocation, we know that it does not alias (for example) any object that was passed in via a parameter. We can use this kind of information to our advantage.

For example, in the following made up IR snippet:

trace(arg0):
  v0 = malloc(8)
  v1 = malloc(16)
  ...

We know that (among other facts) v0 doesn’t alias arg0 or v1 because we have seen its allocation site.

I saw this in the old V8 IR Hydrogen’s lightweight alias analysis¹:

enum HAliasing {
  kMustAlias,
  kMayAlias,
  kNoAlias
};

HAliasing Query(HValue* a, HValue* b) {
  // The same SSA value always references the same object.
  if (a == b) return kMustAlias;

  if (a->IsAllocate() || a->IsInnerAllocatedObject()) {
    // Two non-identical allocations can never be aliases.
    if (b->IsAllocate()) return kNoAlias;
    if (b->IsInnerAllocatedObject()) return kNoAlias;
    // An allocation can never alias a parameter or a constant.
    if (b->IsParameter()) return kNoAlias;
    if (b->IsConstant()) return kNoAlias;
  }
  if (b->IsAllocate() || b->IsInnerAllocatedObject()) {
    // An allocation can never alias a parameter or a constant.
    if (a->IsParameter()) return kNoAlias;
    if (a->IsConstant()) return kNoAlias;
  }

  // Constant objects can be distinguished statically.
  if (a->IsConstant() && b->IsConstant()) {
    return a->Equals(b) ? kMustAlias : kNoAlias;
  }
  return kMayAlias;
}

There is plenty of other useful information such as:

If we know at compile-time that object A has 5 at offset 0 and object B has 7 at offset 0, then A and B don’t alias (thanks, CF)
- In the RPython JIT in PyPy, this is used to determine if two user (Python) objects don’t alias because we know the contents of the user (Python) class field
Object size (though perhaps that is a special case of the above bullet)
Field size/type
Deferring alias checks to run-time
- Have a branch if (a == b) { ... } else { ... }
…

If you have other fun ones, please write in.

Interacting with other instructions

We only handle loads and stores in our optimizer. Unfortunately, this means we may accidentally cache stale information. Consider: what happens if a function call (or any other opaque instruction) writes into an object we are tracking?

The conservative approach is to invalidate all cached information on a function call. This is definitely correct, but it’s a bummer for the optimizer. Can we do anything?

Well, perhaps we are calling a well-known function or a specific IR instruction. In that case, we can annotate it with effects in the same abstract heap model: if the instruction does not write, or only writes to some heaps, we can at least only partially invalidate our heap.

known_builtin_functions = {
  "Array_length": Effects(reads=Array, writes=Empty()),
  "Object_setShape": Effects(reads=Empty(), writes=Object),
  "String_setEncoding": Effects(reads=Empty(), writes=String),
}

However, if the function is unknown or otherwise opaque, we need at least more advanced alias information and perhaps even (partial) escape analysis.

Consider: even if an instruction takes no operands, we have no idea what state it has access to. If it writes to any object A, we cannot safely cache information about any other object B unless we know for sure that A and B do not alias. And we don’t know what the instruction writes to. So we may only know we can cache information about B because it was allocated locally and has not escaped.

Storing vs computing on the fly

Some runtimes such as ART pre-compute all of their alias information in a bit matrix. This makes more sense if you are using alias information in a full control-flow graph, where you might need to iterate over the graph a few times. In a trace context, you can do a lot in one single pass—no need to make a matrix.

When is this useful? How much?

As usual, this is a toy IR and a toy optimizer, so it’s hard to say how much faster it makes its toy programs.

In general, though, there is a dial for analysis and optimization that goes between precision and speed. This is a happy point on that dial, only a tiny incremental analysis cost bump above offset-only invalidation, but for higher precision. I like that tradeoff.

Also, it is very useful in JIT compilers where generally the managed language is a little better-behaved than a C-like language. Somewhere in your IR there will be a lot of duplicate loads and stores from a strength reduction pass, and this can clean up the mess.

Wrapping up

See the full code.

Thanks for joining as I work through a small use of type-based alias analysis for myself. I hope you enjoyed.

See also two mechanisms for dynamic type checks by Andy Wingo. CRuby uses the latter technique described in the article.

Thanks

Thank you to Chris Gregory for helpful feedback.

I made a fork of V8 to go spelunk around the Hydrogen IR. I reset the V8 repo to the last commit before they deleted it in favor of their new Sea of Nodes based IR called TurboFan. ↩

A multi-entry CFG design conundrum

Thu, 22 Jan 2026 00:00:00 +0000

Background and bytecode design

The ZJIT compiler compiles Ruby bytecode (YARV) to machine code. It starts by transforming the stack machine bytecode into a high-level graph-based intermediate representation called HIR.

We use a more or less typical¹ control-flow graph (CFG) in HIR. We have a compilation unit, Function, which has multiple basic blocks, Block. Each block contains multiple instructions, Insn. HIR is always in SSA form, and we use the variant of SSA with block parameters instead of phi nodes.

Where it gets weird, though, is our handling of multiple entrypoints. See, YARV handles default positional parameters (but not default keyword parameters) by embedding the code to compute the defaults inside the callee bytecode. Then callers are responsible for figuring out what offset in the bytecode they should start running the callee, depending on the amount of arguments the caller provides.²

In the following example, we have a function that takes two optional positional parameters a and b. If neither is provided, we start at offset 0000. If just a is provided, we start at offset 0005. If both are provided, we can start at offset 0010.

$ ruby --dump=insns -e 'def foo(a=compute_a, b=compute_b) = a + b'
...
== disasm: #<ISeq:foo@-e:1 (1,0)-(1,41)>
local table (size: 2, argc: 0 [opts: 2, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 2] a@0<Opt=0> [ 1] b@1<Opt=5>
0000 putself                                                          (   1)
0001 opt_send_without_block   <calldata!mid:compute_a, argc:0, FCALL|VCALL|ARGS_SIMPLE>
0003 setlocal_WC_0            a@0
0005 putself
0006 opt_send_without_block   <calldata!mid:compute_b, argc:0, FCALL|VCALL|ARGS_SIMPLE>
0008 setlocal_WC_0            b@1
0010 getlocal_WC_0            a@0[Ca]
0012 getlocal_WC_0            b@1
0014 opt_plus                 <calldata!mid:+, argc:1, ARGS_SIMPLE>[CcCr]
0016 leave                    [Re]
$

(See the jump table debug output: [ 2] a@0<Opt=0> [ 1] b@1<Opt=5>)

Unlike in Python, where default arguments are evaluated at function creation time, Ruby computes the default values at function call time. This includes arbitrary function calls, raising exceptions, doing long I/O, or whatever your heart desires. For this reason, embedding the default code inside the callee makes a lot of sense; we have a full call frame already set up, so any optimizations (!), side-exits, exception handling machinery, profiling, etc doesn’t need special treatment.

Since the caller knows what arguments it is passing, and often to what function, we can efficiently support this in the JIT. We just need to know what offset in the compiled callee to call into. The interpreter can also call into the compiled function, which just has a stub to do dispatch to the appropriate entry block.

This has led us to design the HIR to support multiple function entrypoints. Instead of having just a single entry block, as most control-flow graphs do, each of our functions now has an array of function entries: one for the interpreter, at least one for the JIT, and more for default parameter handling. Each of these entry blocks is separately callable from the outside world.

Here is what the (slightly cleaned up) HIR looks like for the above example:

Optimized HIR:
fn foo@tmp/branchnil.rb:4:
bb0():
  EntryPoint interpreter
  v1:BasicObject = LoadSelf
  v2:BasicObject = GetLocal :a, l0, SP@5
  v3:BasicObject = GetLocal :b, l0, SP@4
  v4:CPtr = LoadPC
  v5:CPtr[CPtr(0x16d27e908)] = Const CPtr(0x16d282120)
  v6:CBool = IsBitEqual v4, v5
  IfTrue v6, bb2(v1, v2, v3)
  v8:CPtr[CPtr(0x16d27e908)] = Const CPtr(0x16d282120)
  v9:CBool = IsBitEqual v4, v8
  IfTrue v9, bb4(v1, v2, v3)
  Jump bb6(v1, v2, v3)
bb1(v13:BasicObject):
  EntryPoint JIT(0)
  v14:NilClass = Const Value(nil)
  v15:NilClass = Const Value(nil)
  Jump bb2(v13, v14, v15)
bb2(v27:BasicObject, v28:BasicObject, v29:BasicObject):
  v65:HeapObject[...] = GuardType v27, HeapObject[class_exact*:Object@VALUE(0x1043aed00)]
  v66:BasicObject = SendWithoutBlockDirect v65, :compute_a (0x16d282148)
  Jump bb4(v27, v66, v29)
bb3(v18:BasicObject, v19:BasicObject):
  EntryPoint JIT(1)
  v20:NilClass = Const Value(nil)
  Jump bb4(v18, v19, v20)
bb4(v38:BasicObject, v39:BasicObject, v40:BasicObject):
  v69:HeapObject[...] = GuardType v38, HeapObject[class_exact*:Object@VALUE(0x1043aed00)]
  v70:BasicObject = SendWithoutBlockDirect v69, :compute_b (0x16d282148)
  Jump bb6(v38, v39, v70)
bb5(v23:BasicObject, v24:BasicObject, v25:BasicObject):
  EntryPoint JIT(2)
  Jump bb6(v23, v24, v25)
bb6(v49:BasicObject, v50:BasicObject, v51:BasicObject):
  v73:Fixnum = GuardType v50, Fixnum
  v74:Fixnum = GuardType v51, Fixnum
  v75:Fixnum = FixnumAdd v73, v74
  CheckInterrupts
  Return v75

If you’re not a fan of text HIR, here is an embedded clickable visualization of HIR thanks to our former intern Aiden porting Firefox’s Iongraph:

(You might have to scroll sideways and down and zoom around. Or you can open it in its own window.)

Each entry block also comes with block parameters which mirror the function’s parameters. These get passed in (roughly) the System V ABI registers.

This is kind of gross. We have to handle these blocks specially in reverse post-order (RPO) graph traversal. And, recently, I ran into an even worse case when trying to implement the Cooper-style “engineered” dominator algorithm: if we walk backwards in block dominators, the walk is not guaranteed to converge. All non-entry blocks are dominated by all entry blocks, which are only dominated by themselves. There is no one “start block”. So what is there to do?

The design conundrum

Approach 1 is to keep everything as-is, but handle entry blocks specially in the dominator algorithm too. I’m not exactly sure what would be needed, but it seems possible. Most of the existing block infra could be left alone, but it’s not clear how much this would “spread” within the compiler. What else in the future might need to be handled specially?

Approach 2 is to synthesize a super-entry block and make it a predecessor of every interpreter and JIT entry block. Inside this approach there are two ways to do it: one (2.a) is to fake it and report some non-existent block. Another (2.b) is to actually make a block and a new instruction that is a quasi-jump instruction. In this approach, we would either need to synthesize fake block arguments for the JIT entry block parameters or add some kind of new LoadArg<i> instruction that reads the argument i passed in.

(suggested by Iain Ireland, as seen in the IBM COBOL compiler)

Approach 3 is to duplicate the entire CFG per entrypoint. This would return us to having one entry block per CFG at the expense of code duplication. It handles the problem pretty cleanly but then forces code duplication. I think I want the duplication to be opt-in instead of having it be the only way we support multiple entrypoints. What if it increases memory too much? The specialization probably would make the generated code faster, though.

(suggested by Ben Titzer)

None of these approaches feel great to me. The probable candidate is 2.b where we have LoadArg instructions. That gives us flexibility to also later add full specialization without forcing it.

Cameron Zwarich also notes that this this is an analogue to the common problem people have when implementing the reverse: postdominators. This is because often functions have multiple return IR instructions. He notes the usual solution is to transform them into branches to a single return instruction.

Do you have this problem? What does your compiler do?

Update: a conclusion

We have decided to go with the superblock approach.

We use extended basic blocks (EBBs), but this doesn’t matter for this post. It makes dominators and predecessors slightly more complicated (now you have dominating instructions), but that’s about it as far as I can tell. We’ll see how they fare in the face of more complicated analysis later. ↩
Keyword parameters have some mix of caller/callee presence checks in the callee because they are passed in un-ordered. The caller handles simple constant defaults whereas the callee handles anything that may raise. Check out Kevin Newton’s awesome overview. ↩

The GDB JIT interface

Tue, 30 Dec 2025 00:00:00 +0000

GDB is great for stepping through machine code to figure out what is going on. It uses debug information under the hood to present you with a tidy backtrace and also determine how much machine code to print when you type disassemble.

This debug information comes from your compiler. Clang, GCC, rustc, etc all produce debug data in a format called DWARF and then embed that debug information inside the binary (ELF, Mach-O, …) when you do -ggdb or equivalent.

Unfortunately, this means that by default, GDB has no idea what is going on if you break in a JIT-compiled function. You can step instruction-by-instruction and whatnot, but that’s about it. This is because the current instruction pointer is nowhere to be found in any of the existing debug info tables from the host runtime code, so your terminal is filled with ???. See this example from the V8 docs:

#8  0x08281674 in v8::internal::Runtime_SetProperty (args=...) at src/runtime.cc:3758
#9  0xf5cae28e in ?? ()
#10 0xf5cc3a0a in ?? ()
#11 0xf5cc38f4 in ?? ()
#12 0xf5cbef19 in ?? ()
#13 0xf5cb09a2 in ?? ()
#14 0x0809e0a5 in v8::internal::Invoke (...) at src/execution.cc:97

Fortunately, there is a JIT interface to GDB. If you implement a couple of functions in your JIT and run them every time you finish compiling a function, you can get the debugging niceties for your JIT code too. See again a V8 example:

#6  0x082857fc in v8::internal::Runtime_SetProperty (args=...) at src/runtime.cc:3758
#7  0xf5cae28e in ?? ()
#8  0xf5cc3a0a in loop () at test.js:6
#9  0xf5cc38f4 in test.js () at test.js:13
#10 0xf5cbef19 in ?? ()
#11 0xf5cb09a2 in ?? ()
#12 0x0809e1f9 in v8::internal::Invoke (...) at src/execution.cc:97

Unfortunately, the GDB docs are somewhat sparse. So I went spelunking through a bunch of different projects to try and understand what is going on.

The big picture (and the old interface)

GDB expects your runtime to expose a function called __jit_debug_register_code and a global variable called __jit_debug_descriptor. GDB automatically adds its own internal breakpoints at this function, if it exists. Then, when you compile code, you call this function from your runtime.

In slightly more detail:

Compile a function in your JIT compiler. This gives you a function name, maybe other metadata, an executable code address, and a code size
Generate an entire ELF/Mach-O/… object in-memory (!) for that one function, describing its name, code region, maybe other DWARF metadata such as line number maps
Write a jit_code_entry linked list node that points at your object (“symfile”)
Link it into the __jit_debug_descriptor linked list
Call __jit_debug_register_code, which gives GDB control of the process so it can pick up the new function’s metadata
Optionally, break into (or crash inside) one of your JITed functions
At some point, later, when your function gets GCed, unregister your code by editing the linked list and calling __jit_debug_register_code again

This is why you see compiler projects such as V8 including large swaths of code just to make object files:

V8
Cinder
Zend PHP
CoreCLR/.NET
QEMU
JavaScriptCore
LuaJIT
ART
- which looks like it does something smart about grouping the JIT code entries together (RepackEntries), but I’m not sure exactly what it does
HHVM
TomatoDotNet
Jato JVM
a minimal example
monoruby
Mono
It looks like Dart used to have support for this but has since removed it
wasmtime

Because this is a huge hassle, GDB also has a newer interface that does not require making an ELF/Mach-O/…+DWARF object.

Custom debug info (the new interface)

This new interface requires writing a binary format of your choice. You make the writer and you make the reader. Then, when you are in GDB, you load your reader as a shared object.

The reader must implement the interface specified by GDB:

GDB_DECLARE_GPL_COMPATIBLE_READER;
extern struct gdb_reader_funcs *gdb_init_reader (void);
struct gdb_reader_funcs
{
  /* Must be set to GDB_READER_INTERFACE_VERSION.  */
  int reader_version;

  /* For use by the reader.  */
  void *priv_data;

  gdb_read_debug_info *read;
  gdb_unwind_frame *unwind;
  gdb_get_frame_id *get_frame_id;
  gdb_destroy_reader *destroy;
};

The read function pointer does the bulk of the work and is responsible for matching code ranges to function names, line numbers, and more.

Here are some details from Sanjoy Das.

Only a few runtimes implement this interface. Most of them stub out the unwind and get_frame_id function pointers:

I think it also requires at least the reader to proclaim it is GPL via the macro GDB_DECLARE_GPL_COMPATIBLE_READER.

Since I wrote about the perf map interface recently, I have it on my mind. Why can’t we reuse it in GDB?

Adapting to the Linux perf interface

I suppose it would be possible to try and upstream a patch to GDB to support the Linux perf map interface for JITs. After all, why shouldn’t it be able to automatically pick up symbols from /tmp/perf-...? That would be great baseline debug info for “free”.

In the meantime, maybe it is reasonable to create a re-usable custom debug reader:

When registering code, write the address and name to /tmp/perf-... as you normally would
Write the filename as the symfile (does this make /tmp the magic number?)
Have the debug info reader just parse the perf map file

It would be less flexible than both the DWARF and custom readers support: it would only be able to handle filename and code region. No embedding source code for GDB to display in your debugger. But maybe that is okay for a partial solution?

Update: Here is my small attempt at such a plugin.

The n-squared problem

V8 notes in their GDB JIT docs that because the JIT interface is a linked list and we only keep a pointer to the head, we get O(n²) behavior. Bummer. This becomes especially noticeable since they register additional code objects not just for functions, but also trampolines, cache stubs, etc.

Garbage collection

Since GDB expects the code pointer in your symbol object file not to move, you have to make sure to have a stable symbol file pointer and stable executable code pointer. To make this happen, V8 disables its moving GC.

Additionally, if your compiled function gets collected, you have to make sure to unregister the function. Instead of doing this eagerly, ART treats the GDB JIT linked list as a weakref and periodically removes dead code entries from it.

Max Bernstein's Blog

Sorry for marking all the posts as unread

A survey of inlining heuristics

Inlining: the “easy” part

When: the harder part

The heuristics

Call context and profiles: the other harder part

Splitting

Profile splitting

Bytecode inlining

Early tier with counters

Inline and analyze and hope

Thanks

The survey: bits and bobbles

Cinder

PyPy

V8

V8 Hydrogen

V8 TurboFan

V8 Maglev

JavaScriptCore

SpiderMonkey

Wasmtime and Cranelift

HotSpot

TruffleRuby

.NET

Dart

HHVM

ART

JikesRVM

Other/research

Graal

Checking assembly with Z3

Thanks

Travel notes: RubyKaigi Hakodate

Part one

Part two

Part three

Partial static single information form

When do we insert type refinements?

During SSA optimization

After which conditionals do we refine?

How does this compare to the complicated looking papers?

In other compilers

Aside: logic for e.g. HeapObject upgrade

Conclusion

Value numbering

Eliminating common subexpressions

Pure vs impure

Local value numbering

Global value numbering

State management and invalidation

Out in the world

Implementations

Wrapping up; bits and bobbles

Acyclic e-graphs

Partial redundancy elimination

Using Perfetto in ZJIT

The rules

Lies, damned lies, and statistics

Building intuition

A look at Perfetto

Implementation

Tracing more things

Conclusion

A fuzzer for the Toy Optimizer

Generating programs

Verifying programs

Running programs

The harness

Testing the verifier

Full code

Extensions

Thanks

Type-based alias analysis in the Toy Optimizer

Representing types

Load-store forwarding

Object provenance / allocation site

Interacting with other instructions

Storing vs computing on the fly