November 2026 · CUDA, H200, Claude Code agents
Rank 308 to 158: the last day of the Open Kernel Challenge
On the last day of the Open Kernel Challenge, my decode layer went from rank 308 to rank 158 overall. The final file runs one Muse Spark decoder layer 2.280x faster than SGLang on an NVIDIA H200. On the judge's weighted mix that is 9,922.9 us, down from 10,146.4 us that morning.
2.280x
faster than SGLang on one decoder layer
308 → 158
overall rank, morning of Nov 4 to the close
2.3%
speedup gained on the last day (2.229x to 2.280x)
$181
total GPU rental for the campaign
The challenge gives you one layer of Muse Spark Open. It is the slice a single GPU runs when the model is split 8 ways: the QKV projection, gated attention, the output projection, a router, the routed experts in NVFP4, and the norms in between.
You write it as one Python file with CUDA in strings. The judge times it against SGLang's kernels at twelve cells: decode and 8-token verify, each at batch 1, 16 and 32, on a sliding-window layer and a full-attention layer. Your score is the geometric mean of the per-load speedups. The overall leaderboard is ordered by this one problem.
Top: what the file runs, left to right. Bottom: the twelve cells the judge times it at, which combine into one score.
Every cell counts equally in the score, whichever kernel it stresses. The overall leaderboard is ordered by this one problem.
I did not write most of the CUDA by hand. I ran the campaign as the lead of a team of Claude Code agents, each working one lane: attention, the dense projections, the MoE, launch overlap, synchronization. This post covers the first week briefly, then what moved the number on the final day, and what I would do differently. Terms are defined in the glossary at the end.
How the campaign ran
The campaign ran for eight days, from October 28 to the close at midnight Pacific on November 4. One Claude Code session acted as lead. It read the profiles, decided what to try, and spawned an agent per lane, each with its own worktree and a checkpoint file to resume from.
Measurement ran on a rented H200 on RunPod, one job at a time behind a GPU lock. The judge allowed 20 public runs and 40 private checks per problem per day, so every candidate had to earn a slot.
How work moved from the lead session through the GPU to the judge.
Public row on the decode-layer board, speedup over SGLang (x)
Rank 158 at the close (others kept improving after 19:07). A disqualified routing-slack row on Nov 3 is left out. Source: public rows from the campaign ledger and leaderboard reads, Oct 28 to Nov 4, 2026.
The board is dense near the top. On the morning of November 4, rank 100 needed 2.297x and I was at 2.229x, so 3% of layer time separated rank 308 from rank 100. That is why a 2.3% gain on the last day moved me about 150 places.
The first seven days
The first submission on October 27 was the PyTorch layer with each component's kernel ported in: 1.39x. Most of the early gain came from the next day.
October 28: the rewrite. The agents rewrote the three big pieces from scratch. Attention became one launch with a cooperative finalize. The projections got skinny kernels that stream compressed weights. The MoE became one persistent kernel: routing, both expert matrix multiplies, a grid barrier, then a final pass. By that evening the layer was at 1.78x.
That grid barrier is the same one I removed a week later. It sat in my own first design, and in the public code I later built on.
October 29 to 31: tuning, then a plateau. Per-batch kernel choices, a leaner attention epilogue, and a fused route kernel took the layer to 2.01x by October 29. Then it stalled. Stream-K variants of the MoE lost by 7 to 31 us, and a side-by-side test against vjeux's public entry showed its attention ahead of mine on six of the rows.
November 1: switching base. The best public layer at the time was keell0renz's entry, at 2.26x on its own board row. Rather than keep tuning my own file, I forked it with attribution, repaired its precision at the attention boundary, and kept my expert kernels where they were faster. That gave 2.13x on the judge, rank 388.
November 2 and 3: small wins and one mistake. Programmatic dependent launch at every batch size, a few fused boundaries, and many 0.3% tweaks took the row to 2.22x by the small hours of November 4. On November 3 I also shipped routing slack: at decode 1, near-tie expert picks were swapped for experts this GPU did not own, which the task text seemed to allow. The organizers disqualified those runs, I withdrew the rest, and from then on routing stayed exact.
The grid barrier that did not need to exist
The biggest single gain of the day was two lines. The MoE kernel ran the experts' matrix work across every CTA on the GPU, then hit a grid-wide barrier, then ran a final pass that sums each token's eight expert outputs. The barrier made every token wait for the slowest CTA, even though a token only needs its own eight experts.
The replacement is a completion counter per expert. When a CTA finishes its share of an expert, bar.sync orders all of that CTA's writes. Then one thread bumps the expert's counter with a release (red.release.gpu).
The final pass for a token polls the counters of its eight experts with an acquire (ld.acquire.gpu). It reads their outputs only once each counter reaches its expected count.
Illustrative timeline, not measured timings. Rows are CTAs doing expert matrix work; time runs left to right.
Token A gains because all of its experts finish early; token B still waits for the slowest CTA, because one of its experts runs there. Each dot is a CTA's release on an expert's counter after bar.sync.
With the batch-32 decode build converted, the judge moved from 10,110 us to 10,046 us. The same change on the larger verify builds followed later in the day.
Why is it still correct with the barrier gone?
- Happens-before is preserved per expert. Each writer's stores are ordered before its release by the CTA barrier, and the reader's acquire synchronizes with every release it counted. That is the same guarantee the grid barrier gave, scoped to the data the reader actually touches.
- The spin cannot deadlock. The kernel is a cooperative launch, so every CTA is resident at once and every counter it waits on will eventually be bumped.
- The poll is unbounded. A wrong expected count now hangs the kernel, which you notice, instead of falling through to half-written data, which you might not.
The embarrassing part is that this code was already in the file. Counters like these shipped in keell0renz's public entry, which my file was built on. Three things kept them off:
- The team switched them off on November 1, because the poll was capped at 4 million iterations and could fall through to unready data. The repair note said the feature was "not inherently forbidden", and nobody went back to it.
- A later rule-out of the verify builds rested on a misread of the finisher path.
- My rule for rationing judge checks on the final day pointed at the batch-1 cells. This barrier only exists at batch 16 and up.
What brought it back was Aleksey Smolenchuk's write-up of his agent swarm, Operating a long-horizon agentic optimization swarm. It lists "removing a grid-wide barrier in the expert step" among his kept changes. Checking that list against my own file found the disabled counters within minutes.
A publication bug that no test caught
After the barrier win, I asked an agent that had not seen our reasoning to review every synchronization point in the layer. It found a real gap in the MoE helper warp's completion credit. That is the counter later phases wait on before they read the first phase's output.
The old sequence: every lane runs __threadfence() after its stores, then __syncwarp(), then lane 0 does a relaxed atomicAdd on the counter.
An earlier rule-out had called this a valid PTX release pattern. But a release pattern is per thread: lane 0's own fence, then lane 0's own atomic. The other 31 lanes' stores reach lane 0 only through the warp relay, after lane 0's fence. So the PTX model does not order them before the release.
__syncwarp()The MoE helper warp's completion credit, old and fixed. Time runs downward; lane k stands for any of the other 31 lanes.
A release pattern is per thread: lane 0's own fence, then lane 0's own atomic. No test ever failed on the old order; the fix passed the reference check at all twelve cells and cost nothing in speed.
The fix moves the fence to where it does its job. The 31 per-lane fences are gone, and lane 0 alone runs fence.sc.gpu after __syncwarp() and before its atomic. Now lane 0's fence and atomic form a release that covers every lane's stores. It passed the reference check at all twelve cells and cost nothing in speed.
No test had ever failed on this, and I never saw it misbehave. I took the fix anyway: otherwise the file's correctness would rest on what the hardware happens to do, not on what the memory model promises.
Handing the evening to an agent team
The file behind my final row came out of EAT, a harness from the paper Evolving Agent Teams (code). A supervisor agent designs a team of Claude Code agents, the harness runs your official eval at the end of each iteration, and the supervisor reads the transcripts and redesigns the team. It never writes the solution itself.
Start at the supervisor and follow the arrows. A stack means several agents at once; a cylinder is a file.
The supervisor never edits the file; the team makes every change. Source: the EAT run on November 4, 2026, and the EAT paper's description of the harness.
Setting it up took three files:
- a
task.mdwith the rules every agent gets: no routing slack, keep every fence, never touch the judge; - an
eval.shthat copies the file to the H200 and times all twelve cells against SGLang; - an env file.
I gave it a $400 budget and a hard stop at 22:30. Its first iteration scored 0: a stacked file had grown to 1,057,604 bytes, and the judge rejects anything over 1,000,000. Stripping comments brought it to 915,316 bytes, and the eval now enforces the cap.
| Time (PT) | Version | What changed |
|---|---|---|
| 20:01 | EAT v7 (EAT's last) | A packed attention path at verify 16 and 32, full layers only |
| 19:25 | EAT v6 | Batch-32 decode moved onto the faster glue-boundary kernel |
| 18:46 | EAT v5 | Batch-1 MoE setup computed warp-parallel; MoE launched with PDL at verify 16 and 32 |
| 18:23 | EAT v4 (the row file) | The glue boundary's cross-CTA sums sent with st.async and mbarrier completion, its PDL trigger moved to kernel entry |
| 17:16 | EAT v3 | My comment-stripped stack, under the size cap |
| 15:59 | EAT v2 | A held QKV kernel swap at verify 16 |
| 15:50 | EAT v1 | The grid-barrier removal on the verify builds |
EAT v4 is the row file: one of its public runs drew 9,922.9 us. EAT v7 was faster on EAT's own box measurements. On the judge, though, it averaged 10,029.6 us against v4's 10,054.0, a gap well inside the judge's spread, and none of its runs beat the row. The whole run cost $131.94 over five iterations.
EAT did two things I would not have. Its integrator noticed that every A/B in the repo ran with the MoE's PDL edge off, while the eval ran with it on, and re-measured in the eval's configuration. And it kept a strict results file, with a row for every loss and null, so later agents did not repeat them.
It also did one thing badly. Twice, an integrator finished its work and then slept for half an hour, waiting for handoffs that had already arrived. On a deadline, that cost most of an iteration.
The judge is noisy, so the same file went in many times
The judge's run-to-run spread is larger than almost every change I made. Most single changes were worth 0.1 to 0.5% of layer time; the same file's results spread over about 2.4%.
Each bar is a size in percent of layer time on the judge. The dashed line marks where the judge's spread ends.
A typical change fits several times over inside one file's run-to-run spread. The whole last-day gain is about the size of that spread, and the gap to rank 100 is a little larger. Source: judge results for the same bytes, and the board on November 4, 2026.
Judge results for the same bytes, weighted layer time in us (lower is faster). Filled: private check. Hollow: public run.
Gap between the means: 24.5 us. Spread within EAT v4: 237.9 us. Source: okc judge results (checks and public runs), Nov 4, 2026.
The board keeps your best run. So a fixed file has a distribution, and each public run is a draw from it. Identical bytes are deduplicated, so each repeat carried a different trailing comment.
I spent the last runs on the file with the lower mean, not the one with the luckiest single result. The mean is what decides how likely the next draw is to beat the row.
The same logic paid off on a standalone board. Ten runs of the ported MoE kernel ranged from 2.349x to 2.389x, and the tenth set a new MoE row. On attention and the dense projections, the old rows came from lucky runs, and ten fresh runs of each never got close.
To be honest about what the leaderboard number means: it is a best-of-N. My final row is 9,922.9 us, but the same file averages about 10,054 us, which is 2.25x rather than 2.28x.
The judge also rejects any submission over 1,000,000 bytes, and my stacked file hit 1,057,604. The rules allow stripping comments as long as the source stays readable. A script that strips them and asserts the code is unchanged brought it to 915,316 bytes.
What I would do differently
- Inventory the synchronization points early, and ask of each what the consumer needs. A grid barrier, a kernel boundary and an early dependency wait all make a consumer wait for every producer. Where it reads a known subset, a counter per subset with release and acquire keeps correctness and drops the wait on stragglers.
- Treat a feature switched off for safety as a lead. The biggest win of the last day was code already in the file, disabled two days earlier for a fixable reason. A grep for force-offs would have found it on day one.
- Rank work by where the time is, over every cell. My rule for rationing judge checks pointed effort at the batch-1 cells, and the largest pool of time sat at batch 16 and up. A rule for spending quota should not decide what gets engineered.
- Measure in the eval's configuration. Every A/B in the repo had a launch-overlap edge switched off that the real eval had on. EAT's integrator caught it; I should have.
- Read the clock, never estimate it. On November 4 the schedule in my notes drifted 3.5 hours from the real time. After that, every timestamp in the ledger came from
datein the same command. - Plan public runs around the mean, not the best draw. The board shows a best-of-N; the mean decides your odds on the next run.
Final numbers at the close: decode layer 9,922.9 us (2.280x over SGLang), rank 158 overall; MoE board 2.389x. Total GPU rental: $181.
Glossary
| Term | Meaning |
|---|---|
| A/B | Timing the old and the new file back to back several times on the same GPU and comparing each cell, so a small change can be told apart from noise. |
| Acquire / release | A pair of memory-ordering operations. Everything a thread wrote before a release is visible to another thread once its acquire reads the released value. |
| Check | A private judge run: the same measurement as a public run, but kept off the board. 40 per problem per day. |
| Cooperative launch | A kernel launch that guarantees every CTA is on the GPU at the same time, so CTAs can wait on each other without deadlocking. |
| CTA | Cooperative thread array, also called a thread block: a group of threads that run together on one streaming multiprocessor and share fast on-chip memory. |
| Decode / verify | The two kinds of serving step. Decode produces one new token per request. Verify checks an 8-token draft from speculative decoding in a single step. |
| EAT | Evolving Agent Teams: a harness where a supervisor agent designs, runs and redesigns a team of Claude Code agents against a fixed eval. |
| EAT v1 to v7 | The successive best files EAT produced on November 4. v4 set my final row; v7 was its last. |
| Fence | An instruction that orders a thread's memory operations, for example fence.sc.gpu, so other threads see them in a guaranteed order. |
| Geometric mean | The average used for the score: multiply the per-load speedups and take the nth root, so each load counts equally. |
| Glue-boundary kernel | In this file, the small kernel between the attention block and the experts that applies the residual and the norms. Its norm needs sums of squares from several CTAs. |
| Grid barrier | A point every CTA in a kernel must reach before any can continue. |
| H200 | The NVIDIA data-center GPU the challenge is timed on. |
| Judge | The organizers' service that checks a submission for correctness and times it against SGLang. |
| Lane | Two meanings here. In CUDA, one of the 32 threads in a warp. In the campaign, one agent's area of work, such as attention or the MoE. |
mbarrier / st.async | Hopper hardware features. An mbarrier counts arrivals and bytes in shared memory; st.async writes into another CTA's shared memory and signals that CTA's mbarrier when the data lands. |
| MoE | Mixture of experts: a router picks 8 of 256 expert networks for each token, and only those run. |
| Muse Spark Open | The open model whose decoder layer the challenge uses. |
| NVFP4 | NVIDIA's 4-bit floating-point weight format with per-block scales, used for the expert weights. |
| PDL | Programmatic dependent launch: lets the next kernel start its setup before the previous one finishes, then wait only at the point where it needs the previous kernel's output. |
| Public run | A judge run that counts on the leaderboard. 20 per problem per day; the board keeps your best. |
| PTX | NVIDIA's virtual instruction set, and the memory model that says which orderings are guaranteed. |
| QKV projection | The matrix multiply that turns the layer input into attention queries, keys and values. |
| Routing slack | Swapping a near-tie expert pick for a different expert to save time. The organizers disqualified runs that did this. |
| Row | Your entry on the leaderboard, set by your best public run. |
| SGLang | The open-source serving engine whose kernels are the baseline every submission is timed against. |
| Sliding-window / full attention | The two layer kinds in the model. A sliding-window layer attends to recent tokens only; a full layer attends to the whole context. |
| Speedup | SGLang's time divided by yours. 2.28x means the layer runs in 1/2.28 of SGLang's time. |
| Tensor / expert parallelism | Splitting a model across GPUs: tensor parallelism splits each weight matrix, expert parallelism splits the set of experts. The challenge times one GPU's slice of an 8-way split. |
| us | Microseconds. |
| Warp | 32 threads that execute together on a GPU. |