GPU Kernel Generation

Shounak Ray

Key Takeaways

Test-time strategies work great for simple problems (97% on Level 1 subset) but specialized training still dominates on complex ones (2.7× gap on Level 2).

Configuration matters more than you'd think. The best settings completely reverse between problem types and models. A one-size-fits-all approach leaves significant performance on the table.

Chain-of-thought isn't universal. The explicit reasoning that helps with math encourages over-engineering in code generation. Sometimes "just do it" beats "think carefully first."

The bottleneck is semantics, not syntax. Future work should focus on helping models understand what the code should do, not just how to write valid CUDA.

Introduction

Writing efficient GPU kernels is hard. These low-level programs need to manage thousands of parallel threads, optimize memory access patterns, and avoid subtle synchronization bugs. Recently, specialized AI models trained specifically for this task have shown promising results—Kevin-32B, for instance, achieves 65% correctness on the KernelBench benchmark after extensive reinforcement learning.

But specialized training is expensive. We wanted to know: what if we just used off-the-shelf models like GPT-4.1 and GPT-5.1, but were really clever about how we used them? Could better test-time strategies—things like iterative refinement, smarter feedback, and chain-of-thought prompting—close the gap?

We ran 952 experiments across two difficulty levels to find out.

Experimental Infrastructure

This research involved significant computational infrastructure:

• 952 kernel generation instances across two difficulty levels
• Heterogeneous GPU cluster: 4× H100, 8× L40S, 18× V100, 8× RTX 2080 Ti
• Distributed job scheduling with fault tolerance and up to 100 concurrent experiments
• Custom evaluation harness (caesar-internal) for reproducible verification
• Correctness verified over 5 random trials using torch.allclose with 1e-2 tolerance

Methods

We tested three main strategies, each with a budget of 10 LLM calls per problem:

Sequential Refinement: Generate a kernel, test it, get feedback on errors, and iterate. Simple but effective—the model keeps improving until it gets it right (or runs out of attempts).

Backtracking: Like refinement, but if the model makes things worse, we revert to the previous good version and try again with higher randomness. The idea is to escape "local minima" where the model gets stuck.

Chain-of-Thought (CoT): Prompt the model to explicitly reason through the implementation—think about memory access patterns, thread organization, CUDA optimizations—before writing code.

We also varied how we presented feedback:

• Raw code: Just show the full generated code
• AST-structured: Parse the code into a syntax tree and show what changed
• Contrastive: Show both the best attempt so far and the current failure

Figure 1: Correctness rates across KernelBench difficulty levels. Level 1 (simple operations) achieves 97% correctness with GPT-5.1, while Level 2 (fused operations) drops to 17.6%—a 2.7× gap compared to specialized models.

Results

The Good News: Simple Problems Are (Mostly) Solved

On Level 1 problems—single operations like matrix multiplication—our best configuration hit 97.1% correctness. That's GPT-5.1 with sequential refinement and raw code feedback. For straightforward kernel generation, you don't need specialized training.

(Note: This was on a subset representing approximately 20% of the full Level 1 benchmark, so interpret with appropriate caution.)

The Bad News: Complex Problems Remain Hard

Level 2 problems require fusing multiple operations together, and here we only managed 17.6% correctness. Kevin-32B achieves 48% on the same problems—a 2.7× gap. Test-time cleverness alone isn't enough for harder tasks.

The Surprising Finding: One Size Doesn't Fit All

Perhaps our most interesting discovery is that optimal configurations completely reverse depending on the problem and model:

Figure 2a: Feedback structure effectiveness reverses between difficulty levels. Raw code feedback dominates on Level 1 (+34.7pp), while AST-structured feedback wins on Level 2 (+12.6pp).

Feedback structure flips by difficulty:

Our hypothesis? Simple problems benefit from seeing the full code for pattern matching, while complex problems need the focused attention that structured diffs provide.

Contrastive feedback has opposite effects by model:

Figure 2b: Model-specific responses to contrastive feedback. GPT-4.1 improves dramatically (+19.8pp) while GPT-5.1 degrades (-13.2pp), suggesting stronger reasoning models get confused by explicit contrasts.

We think GPT-5.1's stronger reasoning gets confused by explicit contrasts, while GPT-4.1 benefits from the extra signal.

Chain-of-Thought: A Spectacular Failure

Here's the counterintuitive one. Chain-of-thought prompting—which works brilliantly for math problems—fails catastrophically for kernel generation: just 7.2% correctness with 39% of outputs failing to even compile.

Why? When you ask the model to reason step-by-step about CUDA optimizations, it over-engineers everything. A typical failure: CoT generates a 4000+ character kernel with shared memory tiling and register blocking... that has an off-by-one error. The simple 2600-character naive implementation from regular refinement just works.

Figure 3: Test-time compute strategy comparison. Chain-of-thought (CoT) catastrophically fails at 7.2% correctness with 39% compilation failures. Sequential refinement's simple approach dramatically outperforms over-engineered CoT solutions.

The lesson: for executable code, simple and correct beats sophisticated and buggy.

Most Errors Are Semantic, Not Syntactic

Only 8% of failures were compilation errors. The real problem is semantic correctness—the code compiles fine but computes the wrong answer. Off-by-one errors, incorrect thread indexing, mishandled edge cases. The models understand CUDA syntax; they struggle with CUDA semantics.

Figure 4: Error type distribution showing semantic correctness (92%) vastly outweighs compilation errors (8%). Models understand CUDA syntax but struggle with correctness—off-by-one errors, thread indexing, and edge cases.

Insights

Regression Isn't Always Bad

Sequential refinement exhibited a 91.6% regression rate—meaning most of the time, intermediate attempts were worse than previous ones. Yet it achieved the highest final correctness (53.3%). This reveals that aggressive exploration with temporary regressions outperforms conservative approaches. Sometimes you need to get worse before you get better.

Speedup Potential Varies by Complexity

Among correct solutions, speedup potential varies dramatically by problem complexity:

PyTorch's Level 1 implementations are already heavily optimized, while Level 2 fusion patterns have more room for improvement.

Conclusion

Test-time strategies can match specialized training on simple problems, but the gap remains significant for complex tasks. The real insight isn't that one approach dominates—it's that optimal configurations are context-dependent. Raw feedback works for simple kernels, structured diffs for complex ones. Contrastive examples help weaker models but confuse stronger ones. Chain-of-thought, despite its success in math, catastrophically fails for code generation.

This suggests a broader principle: inference strategies are tools, not universal solutions. The bottleneck isn't syntax but semantics—models need to understand what code should do, not just how to write it. Future work should focus on semantic understanding and adaptive configuration selection rather than one-size-fits-all prompting strategies.

Acknowledgments

This research involved running 952 experiments across a heterogeneous GPU cluster (4× H100, 8× L40S, 18× V100, 8× RTX 2080 Ti). Thanks to the instructors and peers who provided feedback on experimental design and evaluation methodology. Special appreciation for the open-source community maintaining KernelBench and the distributed systems that made this scale of experimentation possible.