yelgabo

November 2026 · CUDA, H200, Claude Code agents

Rank 308 to 158: the last day of the Open Kernel Challenge

On the last day of the Open Kernel Challenge, my decode layer went from rank 308 to rank 158 overall. The final file runs one Muse Spark decoder layer 2.280x faster than SGLang on an NVIDIA H200. On the judge's weighted mix that is 9,922.9 us, down from 10,146.4 us that morning.

2.280x

faster than SGLang on one decoder layer

308 → 158

overall rank, morning of Nov 4 to the close

2.3%

speedup gained on the last day (2.229x to 2.280x)

$181

total GPU rental for the campaign

The challenge gives you one layer of Muse Spark Open. It is the slice a single GPU runs when the model is split 8 ways: the QKV projection, gated attention, the output projection, a router, the routed experts in NVFP4, and the norms in between.

You write it as one Python file with CUDA in strings. The judge times it against SGLang's kernels at twelve cells: decode and 8-token verify, each at batch 1, 16 and 32, on a sliding-window layer and a full-attention layer. Your score is the geometric mean of the per-load speedups. The overall leaderboard is ordered by this one problem.

One layer, six components, timed at twelve cells

Top: what the file runs, left to right. Bottom: the twelve cells the judge times it at, which combine into one score.

What one Muse Spark layer runs, and the twelve cells it is timed on Top: one decoder layer, the slice one GPU runs when the model is split 8 ways, as a pipeline of six steps in order: QKV projection, which makes queries, keys and values; gated attention, sliding-window or full; output projection; norms and residual in the glue-boundary kernel; the router, which picks 8 of 256 experts per token; and the routed experts, with NVFP4 weights. Bottom: a grid of twelve timed cells. Rows are the sliding-window layer and the full-attention layer. Columns are decode and 8-token verify, each at batch 1, 16 and 32. Each cell yields a speedup over SGLang. The twelve speedups are multiplied and the twelfth root taken, giving the score: the geometric mean of the 12 speedups over SGLang. What one layer runs, one GPU's slice of an 8-way split QKV projection Gated attention Output projection Norms and residual Router Routed experts queries, keys and values sliding-window or full glue-boundary kernel picks 8 of 256 per token NVFP4 weights The twelve cells it is timed on, each against SGLang's kernels decode 8-token verify batch 1 batch 16 batch 32 batch 1 batch 16 batch 32 sliding-window layer full-attention layer speedup speedup speedup speedup speedup speedup speedup speedup speedup speedup speedup speedup multiply all 12, take the 12th root Score = geometric mean of the 12 speedups over SGLang

Every cell counts equally in the score, whichever kernel it stresses. The overall leaderboard is ordered by this one problem.

I did not write most of the CUDA by hand. I ran the campaign as the lead of a team of Claude Code agents, each working one lane: attention, the dense projections, the MoE, launch overlap, synchronization. This post covers the first week briefly, then what moved the number on the final day, and what I would do differently. Terms are defined in the glossary at the end.

How the campaign ran

The campaign ran for eight days, from October 28 to the close at midnight Pacific on November 4. One Claude Code session acted as lead. It read the profiles, decided what to try, and spawned an agent per lane, each with its own worktree and a checkpoint file to resume from.

Measurement ran on a rented H200 on RunPod, one job at a time behind a GPU lock. The judge allowed 20 public runs and 40 private checks per problem per day, so every candidate had to earn a slot.

One lead, one agent per lane, one GPU, a rationed judge

How work moved from the lead session through the GPU to the judge.

How the agent campaign was organised A single lead Claude Code session reads the profiles and decides what to try. It spawns one lane agent per lane: attention, dense projections, MoE, launch overlap and synchronization, each in its own worktree. Each lane agent writes a checkpoint file to resume from. Lane agents queue candidate files for a rented H200 on RunPod; the queue sits behind a GPU lock, so the H200 runs one job at a time and sends timings back to the lane agents. The lead submits the chosen file to the judge, the organizers' service, which allows 20 public runs and 40 private checks per problem per day. Lead session one Claude Code session reads the profiles and decides what to try Judge the organizers' service 20 public runs + 40 private checks per problem per day submits chosen file Lane agents attention, dense projections, MoE, launch overlap, synchronization each in its own worktree spawns one per lane Checkpoint file per agent, to resume from writes checkpoint behind a GPU lock candidate file candidate file candidate file queue candidate Rented H200 on RunPod one job at a time runs next timings Stack = parallel lane agents. Cylinder = a stored file. Queue items = candidate files waiting for the GPU.
Most of the speedup came early; the last day added 2.3% and about 150 places

Public row on the decode-layer board, speedup over SGLang (x)

1.4 1.6 1.8 2.0 2.2 2.4 Oct 28 Oct 29 Oct 30 Oct 31 Nov 1 Nov 2 Nov 3 Nov 4 Nov 5 Oct 28: 1.403x Nov 1: 2.126x, rank 388 Nov 3 05:40: 2.171x, rank 369 Nov 3 22:05: 2.203x, rank 353 Nov 4 00:55: 2.215x, rank 322 Nov 4 02:05: 2.219x, rank 311 Nov 4 09:20: 2.229x, rank 308 Nov 4 19:07: 2.280x, rank 146 Oct 28: 1.40x Nov 1: 2.13x, rank 388 Nov 4 09:20: 2.23x, rank 308 Nov 4 19:07: 2.28x, rank 146 +2.3% in ten hours

Rank 158 at the close (others kept improving after 19:07). A disqualified routing-slack row on Nov 3 is left out. Source: public rows from the campaign ledger and leaderboard reads, Oct 28 to Nov 4, 2026.

The board is dense near the top. On the morning of November 4, rank 100 needed 2.297x and I was at 2.229x, so 3% of layer time separated rank 308 from rank 100. That is why a 2.3% gain on the last day moved me about 150 places.

The first seven days

The first submission on October 27 was the PyTorch layer with each component's kernel ported in: 1.39x. Most of the early gain came from the next day.

October 28: the rewrite. The agents rewrote the three big pieces from scratch. Attention became one launch with a cooperative finalize. The projections got skinny kernels that stream compressed weights. The MoE became one persistent kernel: routing, both expert matrix multiplies, a grid barrier, then a final pass. By that evening the layer was at 1.78x.

That grid barrier is the same one I removed a week later. It sat in my own first design, and in the public code I later built on.

October 29 to 31: tuning, then a plateau. Per-batch kernel choices, a leaner attention epilogue, and a fused route kernel took the layer to 2.01x by October 29. Then it stalled. Stream-K variants of the MoE lost by 7 to 31 us, and a side-by-side test against vjeux's public entry showed its attention ahead of mine on six of the rows.

November 1: switching base. The best public layer at the time was keell0renz's entry, at 2.26x on its own board row. Rather than keep tuning my own file, I forked it with attribution, repaired its precision at the attention boundary, and kept my expert kernels where they were faster. That gave 2.13x on the judge, rank 388.

November 2 and 3: small wins and one mistake. Programmatic dependent launch at every batch size, a few fused boundaries, and many 0.3% tweaks took the row to 2.22x by the small hours of November 4. On November 3 I also shipped routing slack: at decode 1, near-tie expert picks were swapped for experts this GPU did not own, which the task text seemed to allow. The organizers disqualified those runs, I withdrew the rest, and from then on routing stayed exact.

The grid barrier that did not need to exist

The biggest single gain of the day was two lines. The MoE kernel ran the experts' matrix work across every CTA on the GPU, then hit a grid-wide barrier, then ran a final pass that sums each token's eight expert outputs. The barrier made every token wait for the slowest CTA, even though a token only needs its own eight experts.

The replacement is a completion counter per expert. When a CTA finishes its share of an expert, bar.sync orders all of that CTA's writes. Then one thread bumps the expert's counter with a release (red.release.gpu).

The final pass for a token polls the counters of its eight experts with an acquire (ld.acquire.gpu). It reads their outputs only once each counter reaches its expected count.

With a counter per expert, a token's final pass waits only for its own experts

Illustrative timeline, not measured timings. Rows are CTAs doing expert matrix work; time runs left to right.

Grid barrier versus a completion counter per expert Two illustrative timelines share a time axis running left to right. Five CTAs do expert matrix work. CTA 0 holds some of token B's experts and finishes last; CTA 1 and CTA 2 hold token A's experts and finish early; CTA 3 holds more of token B's experts; CTA 4 holds other experts. Before: a grid barrier sits at the slowest CTA's finish, and the final passes for token A and token B both start after it. After: when a CTA finishes its share of an expert, bar.sync orders its writes and one thread bumps the expert's counter with red.release.gpu. Token A's final pass polls its experts' counters with ld.acquire.gpu and starts as soon as CTA 2, the last of its CTAs, releases, well before the slowest CTA finishes. Token B depends on the slowest CTA, so its final pass starts at the same time as before. Before: grid barrier CTA 0, slowest CTA 1 CTA 2 CTA 3 CTA 4 token A token B token B's experts token A's experts token A's experts token B's experts other experts grid barrier both final passes start after the barrier final pass final pass After: a counter per expert CTA 0, slowest CTA 1 CTA 2 CTA 3 CTA 4 token A token B token B's experts token A's experts token A's experts token B's experts other experts bar.sync, then red.release.gpu polls with ld.acquire.gpu depends on the slowest CTA, as before final pass final pass time (illustrative)

Token A gains because all of its experts finish early; token B still waits for the slowest CTA, because one of its experts runs there. Each dot is a CTA's release on an expert's counter after bar.sync.

With the batch-32 decode build converted, the judge moved from 10,110 us to 10,046 us. The same change on the larger verify builds followed later in the day.

Why is it still correct with the barrier gone?

  • Happens-before is preserved per expert. Each writer's stores are ordered before its release by the CTA barrier, and the reader's acquire synchronizes with every release it counted. That is the same guarantee the grid barrier gave, scoped to the data the reader actually touches.
  • The spin cannot deadlock. The kernel is a cooperative launch, so every CTA is resident at once and every counter it waits on will eventually be bumped.
  • The poll is unbounded. A wrong expected count now hangs the kernel, which you notice, instead of falling through to half-written data, which you might not.

The embarrassing part is that this code was already in the file. Counters like these shipped in keell0renz's public entry, which my file was built on. Three things kept them off:

  • The team switched them off on November 1, because the poll was capped at 4 million iterations and could fall through to unready data. The repair note said the feature was "not inherently forbidden", and nobody went back to it.
  • A later rule-out of the verify builds rested on a misread of the finisher path.
  • My rule for rationing judge checks on the final day pointed at the batch-1 cells. This barrier only exists at batch 16 and up.

What brought it back was Aleksey Smolenchuk's write-up of his agent swarm, Operating a long-horizon agentic optimization swarm. It lists "removing a grid-wide barrier in the expert step" among his kept changes. Checking that list against my own file found the disabled counters within minutes.

A publication bug that no test caught

After the barrier win, I asked an agent that had not seen our reasoning to review every synchronization point in the layer. It found a real gap in the MoE helper warp's completion credit. That is the counter later phases wait on before they read the first phase's output.

The old sequence: every lane runs __threadfence() after its stores, then __syncwarp(), then lane 0 does a relaxed atomicAdd on the counter.

An earlier rule-out had called this a valid PTX release pattern. But a release pattern is per thread: lane 0's own fence, then lane 0's own atomic. The other 31 lanes' stores reach lane 0 only through the warp relay, after lane 0's fence. So the PTX model does not order them before the release.

Lane 0's release covers every lane's stores only when its fence comes after __syncwarp()

The MoE helper warp's completion credit, old and fixed. Time runs downward; lane k stands for any of the other 31 lanes.

Old and fixed fence placement in the helper warp's completion credit Two sequence diagrams side by side, time flowing downward, each with lane k (one of the other 31 lanes), lane 0, and the counter. Old: both lanes do their stores, then each runs __threadfence(), then __syncwarp() spans both lanes, then lane 0 does a relaxed atomicAdd on the counter. Lane k's stores reach lane 0 only through the warp relay at __syncwarp(), after lane 0's fence, so the PTX model does not order them before the release made of lane 0's own fence and lane 0's own atomic. Fixed: both lanes do their stores with no per-lane fence, then __syncwarp(), then lane 0 alone runs fence.sc.gpu, then lane 0 does the atomicAdd on the counter. Lane k's stores now reach lane 0 before its fence, so lane 0's fence and atomic form a release that covers every lane's stores. Old lane k (one of the other 31) lane 0 stores stores __threadfence() __threadfence() __syncwarp() lane k's stores reach lane 0 after its fence relaxed atomicAdd counter Lane 0's fence ran before lane k's stores reached it, so the PTX model does not order them before the release. Fixed lane k (one of the other 31) lane 0 stores stores no per-lane fences __syncwarp() lane k's stores reach lane 0 before its fence fence.sc.gpu atomicAdd counter The fence now sits after the relay, so lane 0's fence and atomic form a release that covers every lane's stores.

A release pattern is per thread: lane 0's own fence, then lane 0's own atomic. No test ever failed on the old order; the fix passed the reference check at all twelve cells and cost nothing in speed.

The fix moves the fence to where it does its job. The 31 per-lane fences are gone, and lane 0 alone runs fence.sc.gpu after __syncwarp() and before its atomic. Now lane 0's fence and atomic form a release that covers every lane's stores. It passed the reference check at all twelve cells and cost nothing in speed.

No test had ever failed on this, and I never saw it misbehave. I took the fix anyway: otherwise the file's correctness would rest on what the hardware happens to do, not on what the memory model promises.

Handing the evening to an agent team

The file behind my final row came out of EAT, a harness from the paper Evolving Agent Teams (code). A supervisor agent designs a team of Claude Code agents, the harness runs your official eval at the end of each iteration, and the supervisor reads the transcripts and redesigns the team. It never writes the solution itself.

EAT's supervisor redesigns the team after every scored iteration

Start at the supervisor and follow the arrows. A stack means several agents at once; a cylinder is a file.

How the EAT loop works A supervisor agent, which never writes the solution, designs a team of Claude Code agents, including an integrator. The team reads task.md, which holds the rules every agent gets: no routing slack, keep every fence, never touch the judge. The team edits the candidate file, one Python file. After every iteration the harness runs eval.sh, which copies the file to the H200 and times all 12 cells against SGLang. The score and the team's transcripts go back to the supervisor, which redesigns the team. The team also logs results to a strict results file with a row for every loss and null, and later agents read it so they do not repeat them. The November 4 run had a 400 dollar budget and a hard stop at 22:30, and spent 131.94 dollars over five iterations. task.md no routing slack keep every fence never touch the judge Supervisor agent never writes the solution eval.sh run after every iteration times all 12 cells against SGLang Team of Claude Code agents including an integrator candidate file one Python file results file a row for every loss and null designs the team transcripts given to every agent edits copied to the H200 score logs results what not to repeat The Nov 4 run $400 budget, hard stop 22:30 5 iterations, $131.94 spent

The supervisor never edits the file; the team makes every change. Source: the EAT run on November 4, 2026, and the EAT paper's description of the harness.

Setting it up took three files:

  • a task.md with the rules every agent gets: no routing slack, keep every fence, never touch the judge;
  • an eval.sh that copies the file to the H200 and times all twelve cells against SGLang;
  • an env file.

I gave it a $400 budget and a hard stop at 22:30. Its first iteration scored 0: a stacked file had grown to 1,057,604 bytes, and the judge rejects anything over 1,000,000. Stripping comments brought it to 915,316 bytes, and the eval now enforces the cap.

Time (PT)VersionWhat changed
20:01EAT v7 (EAT's last)A packed attention path at verify 16 and 32, full layers only
19:25EAT v6Batch-32 decode moved onto the faster glue-boundary kernel
18:46EAT v5Batch-1 MoE setup computed warp-parallel; MoE launched with PDL at verify 16 and 32
18:23EAT v4 (the row file)The glue boundary's cross-CTA sums sent with st.async and mbarrier completion, its PDL trigger moved to kernel entry
17:16EAT v3My comment-stripped stack, under the size cap
15:59EAT v2A held QKV kernel swap at verify 16
15:50EAT v1The grid-barrier removal on the verify builds

EAT v4 is the row file: one of its public runs drew 9,922.9 us. EAT v7 was faster on EAT's own box measurements. On the judge, though, it averaged 10,029.6 us against v4's 10,054.0, a gap well inside the judge's spread, and none of its runs beat the row. The whole run cost $131.94 over five iterations.

EAT did two things I would not have. Its integrator noticed that every A/B in the repo ran with the MoE's PDL edge off, while the eval ran with it on, and re-measured in the eval's configuration. And it kept a strict results file, with a row for every loss and null, so later agents did not repeat them.

It also did one thing badly. Twice, an integrator finished its work and then slept for half an hour, waiting for handoffs that had already arrived. On a deadline, that cost most of an iteration.

The judge is noisy, so the same file went in many times

The judge's run-to-run spread is larger than almost every change I made. Most single changes were worth 0.1 to 0.5% of layer time; the same file's results spread over about 2.4%.

A typical change is far smaller than the judge's spread

Each bar is a size in percent of layer time on the judge. The dashed line marks where the judge's spread ends.

A typical change compared with the judge's spread and the gaps that mattered Horizontal bars on a shared scale from 0 to 3.5 percent of layer time. A typical single change is worth 0.1 to 0.5 percent. The judge's run-to-run spread for one fixed file is about 2.4 percent, so a typical change sits well inside it. The whole last-day gain, from 2.229x to 2.280x, is 2.3 percent. The gap from rank 308 to rank 100 on the morning of November 4, 2.229x against 2.297x, is about 3 percent. A dashed line at 2.4 percent runs through every bar. 0% 0.5% 1% 1.5% 2% 2.5% 3% 3.5% share of layer time Typical single change most changes I made Judge spread, one file same bytes, run to run Whole last-day gain 2.229x to 2.280x Gap, rank 308 to rank 100 2.229x vs 2.297x, Nov 4 morning judge spread ends 0.1 to 0.5% about 2.4% 2.3% about 3%

A typical change fits several times over inside one file's run-to-run spread. The whole last-day gain is about the size of that spread, and the gap to rank 100 is a little larger. Source: judge results for the same bytes, and the board on November 4, 2026.

Each file spreads over about 240 us on the judge; the two files differ by about 24 us

Judge results for the same bytes, weighted layer time in us (lower is faster). Filled: private check. Hollow: public run.

9,900 9,950 10,000 10,050 10,100 10,150 10,200 weighted layer time (us), lower is faster board row 9,922.9 us EAT v49 results EAT v78 results EAT v4 public run: 9,922.9 us EAT v4 check: 9,978.2 us EAT v4 check: 9,980.7 us EAT v4 check: 10,059.1 us EAT v4 check: 10,062.6 us EAT v4 public run: 10,081.2 us EAT v4 check: 10,111.4 us EAT v4 check: 10,129.2 us EAT v4 public run: 10,160.8 us EAT v7 check: 9,949.3 us EAT v7 public run: 9,963.7 us EAT v7 public run: 10,000.1 us EAT v7 check: 10,013.8 us EAT v7 check: 10,030.1 us EAT v7 check: 10,083.4 us EAT v7 public run: 10,094.7 us EAT v7 check: 10,101.3 us mean 10,054.0 mean 10,029.6

Gap between the means: 24.5 us. Spread within EAT v4: 237.9 us. Source: okc judge results (checks and public runs), Nov 4, 2026.

The board keeps your best run. So a fixed file has a distribution, and each public run is a draw from it. Identical bytes are deduplicated, so each repeat carried a different trailing comment.

I spent the last runs on the file with the lower mean, not the one with the luckiest single result. The mean is what decides how likely the next draw is to beat the row.

The same logic paid off on a standalone board. Ten runs of the ported MoE kernel ranged from 2.349x to 2.389x, and the tenth set a new MoE row. On attention and the dense projections, the old rows came from lucky runs, and ten fresh runs of each never got close.

To be honest about what the leaderboard number means: it is a best-of-N. My final row is 9,922.9 us, but the same file averages about 10,054 us, which is 2.25x rather than 2.28x.

The judge also rejects any submission over 1,000,000 bytes, and my stacked file hit 1,057,604. The rules allow stripping comments as long as the source stays readable. A script that strips them and asserts the code is unchanged brought it to 915,316 bytes.

What I would do differently

  1. Inventory the synchronization points early, and ask of each what the consumer needs. A grid barrier, a kernel boundary and an early dependency wait all make a consumer wait for every producer. Where it reads a known subset, a counter per subset with release and acquire keeps correctness and drops the wait on stragglers.
  2. Treat a feature switched off for safety as a lead. The biggest win of the last day was code already in the file, disabled two days earlier for a fixable reason. A grep for force-offs would have found it on day one.
  3. Rank work by where the time is, over every cell. My rule for rationing judge checks pointed effort at the batch-1 cells, and the largest pool of time sat at batch 16 and up. A rule for spending quota should not decide what gets engineered.
  4. Measure in the eval's configuration. Every A/B in the repo had a launch-overlap edge switched off that the real eval had on. EAT's integrator caught it; I should have.
  5. Read the clock, never estimate it. On November 4 the schedule in my notes drifted 3.5 hours from the real time. After that, every timestamp in the ledger came from date in the same command.
  6. Plan public runs around the mean, not the best draw. The board shows a best-of-N; the mean decides your odds on the next run.

Final numbers at the close: decode layer 9,922.9 us (2.280x over SGLang), rank 158 overall; MoE board 2.389x. Total GPU rental: $181.

Glossary

TermMeaning
A/BTiming the old and the new file back to back several times on the same GPU and comparing each cell, so a small change can be told apart from noise.
Acquire / releaseA pair of memory-ordering operations. Everything a thread wrote before a release is visible to another thread once its acquire reads the released value.
CheckA private judge run: the same measurement as a public run, but kept off the board. 40 per problem per day.
Cooperative launchA kernel launch that guarantees every CTA is on the GPU at the same time, so CTAs can wait on each other without deadlocking.
CTACooperative thread array, also called a thread block: a group of threads that run together on one streaming multiprocessor and share fast on-chip memory.
Decode / verifyThe two kinds of serving step. Decode produces one new token per request. Verify checks an 8-token draft from speculative decoding in a single step.
EATEvolving Agent Teams: a harness where a supervisor agent designs, runs and redesigns a team of Claude Code agents against a fixed eval.
EAT v1 to v7The successive best files EAT produced on November 4. v4 set my final row; v7 was its last.
FenceAn instruction that orders a thread's memory operations, for example fence.sc.gpu, so other threads see them in a guaranteed order.
Geometric meanThe average used for the score: multiply the per-load speedups and take the nth root, so each load counts equally.
Glue-boundary kernelIn this file, the small kernel between the attention block and the experts that applies the residual and the norms. Its norm needs sums of squares from several CTAs.
Grid barrierA point every CTA in a kernel must reach before any can continue.
H200The NVIDIA data-center GPU the challenge is timed on.
JudgeThe organizers' service that checks a submission for correctness and times it against SGLang.
LaneTwo meanings here. In CUDA, one of the 32 threads in a warp. In the campaign, one agent's area of work, such as attention or the MoE.
mbarrier / st.asyncHopper hardware features. An mbarrier counts arrivals and bytes in shared memory; st.async writes into another CTA's shared memory and signals that CTA's mbarrier when the data lands.
MoEMixture of experts: a router picks 8 of 256 expert networks for each token, and only those run.
Muse Spark OpenThe open model whose decoder layer the challenge uses.
NVFP4NVIDIA's 4-bit floating-point weight format with per-block scales, used for the expert weights.
PDLProgrammatic dependent launch: lets the next kernel start its setup before the previous one finishes, then wait only at the point where it needs the previous kernel's output.
Public runA judge run that counts on the leaderboard. 20 per problem per day; the board keeps your best.
PTXNVIDIA's virtual instruction set, and the memory model that says which orderings are guaranteed.
QKV projectionThe matrix multiply that turns the layer input into attention queries, keys and values.
Routing slackSwapping a near-tie expert pick for a different expert to save time. The organizers disqualified runs that did this.
RowYour entry on the leaderboard, set by your best public run.
SGLangThe open-source serving engine whose kernels are the baseline every submission is timed against.
Sliding-window / full attentionThe two layer kinds in the model. A sliding-window layer attends to recent tokens only; a full layer attends to the whole context.
SpeedupSGLang's time divided by yours. 2.28x means the layer runs in 1/2.28 of SGLang's time.
Tensor / expert parallelismSplitting a model across GPUs: tensor parallelism splits each weight matrix, expert parallelism splits the set of experts. The challenge times one GPU's slice of an 8-way split.
usMicroseconds.
Warp32 threads that execute together on a GPU.