In the previous post I wrote about releasing ComposableThreads for Rust (also published on crates.io). I’ve kept improving it since, so let me lay out where that’s gone.

As I mentioned last time, while benchmarking I found that rayon’s join() was faster than I expected. At first I figured that if I just tuned the stackless (async) implementation properly, I could catch up to rayon’s performance, but it became clear there’s a limit to how far tuning async alone can get you. Thinking it through: as long as you’re building a mechanism to suspend a task synchronously, you can’t avoid the bookkeeping needed to track task state, or the branching cost of returning to the scheduler once. Async is marketed as “zero overhead,” but the cost of pulling out a continuation is unavoidable — I realized there was no catching up to rayon’s performance without changing the API itself. So I reworked the implementation to bring in rayon’s API as another model.

  • stackful: ordinary (stackful) user-level threads. Can block on a mutex or barrier
  • stackless: async/await based. Driven by polling as a Future, so no stack allocation
  • scoped: a binary-only model where you just hand over the computation you want to run in parallel as a closure, without ever pulling out a continuation. No task descriptor, no heap allocation — the lightest of the three

Looking into this scoped model, it seems like it doesn’t really have a name in the HPC / Cilk-style fork/join research literature. Writing fork/join without pulling out a continuation is, if you think about it, a pretty ordinary thing that already exists in the world — TBB’s parallel_invoke, OpenMP’s parallel for — but there didn’t seem to be a name for it as a distinct programming model in its own right. So within ComposableThreads I gave it the provisional name “scoped,” since the continuation-equivalent work is kept confined to a closure’s scope.

I also experimented with identifying “which worker am I right now” without using thread-local storage (TLS) — taking advantage of the fact that the stack pointer lives in a special region, and working backward from it to the worker. I confirmed the implementation itself works, but in practice TLS wasn’t actually that costly, and the cost of computing things backward turned out to be larger — so at least on my local macOS setup, it didn’t translate into a performance win.

I re-ran the benchmarks too. Running a microbenchmark locally, I compared standalone stackful, stackless, dual (supports both stackful and stackless), and scoped systems against rayon’s join.

First, there’s a clear performance gap between stackless and scoped. I tuned the stackless implementation together with AI, poring over the assembly, and it’s gotten fairly close to its limit. And then, for some reason, scoped ended up faster than rayon — but that implementation was basically written by AI as-is, and I haven’t traced through the details, so honestly I’m not sure why. It’s possible it’s skipping some abstraction machinery that rayon has. Either way, I think this makes it pretty clear how much performance changes just by imposing the constraint of “never pull out a continuation” on the programming model. Stackless comes out somewhat faster than stackful since it skips the context switch stackful needs and doesn’t have to manage a call stack. Dual is slower since it has to branch between stackful and stackless. Both of those are roughly what I’d expect.

Much of this is still close to a prototype and the interface hasn’t settled yet, but I’d like to keep iterating on tuning it into something more usable.