How Makora Generates CUDA Kernels That Beat Hand-Tuned Code | Researcher Conversations at GTC
Summary
- Makora is broadening from automated kernel generation toward a foundation-model-supplier-agnostic “deployment engine for high-performance AI models.” Kernels remain a core component, but the commercial pitch is end-to-end performance across inference servers, training pipelines, reinforcement-learning rollouts, numerics, and heterogeneous hardware. Abdelfattah’s framing: “Our value proposition is that we sell performance.”
- Its sequential Monte Carlo speculative-decoding method delivered roughly 5× the SGLang baseline and 2× SGLang speculative decoding with the experimental overlap scheduler—but only in the cited batch-size-one, low-latency regime. It runs N drafts, scores them with the target model, duplicates strong candidates, evicts weak ones, and always accepts K tokens instead of rewinding. Abdelfattah stresses that the method uses more compute and is “fundamentally a lossy method.”
- Makora uses its FP4 research to illustrate its performance-focused differentiation as foundation models improve at code generation. Remapping FP4’s redundant second zero can effectively provide “the accuracy of FP5 or FP5-something” at an FP4 memory footprint. For NVFP4 weight-activation quantization, the straightforward NVIDIA implementation needs a sparse compensating pass; AMD can instead upcast through an FP6 path sharing FP4 hardware at the same throughput. “This very niche unused precision FP6 is coming to our rescue.”
- Kernel evaluation has itself become a product after an early demo suffered extensive reward hacking. Makora now traces whether generated functions execute correctly, rejects forbidden dependencies, and uses AI to detect reward hacks; the overall pipeline catches maybe 11 exploit classes. Customers submit “a kernel and a problem,” receive a reward signal through an API, and can use it for model training.
- Better coding models are a tailwind under Makora’s model-agnostic strategy. The platform tries multiple agents and returns the best kernel; the team is also testing open models such as “Gemma 4” and exploring fine-tuning or specialization for a less expensive alternative to large foundation models. The harder layer is producing code already integrated with data layouts, interfaces, quantization modes, and systems such as GGML or SGLang.
- The customer wedge spans hardware vendors, enterprises with private GPU estates and no in-house performance team, and neoclouds seeking a default software layer. Hardware makers can bootstrap support without cultivating a developer network, while enterprises can serve internal models without public APIs. The longer-term bet is to abstract both performance engineering and hardware placement: “the right code for the right hardware.”
Deep dive
1. Makora is climbing the stack because selling code generation alone is hard
Mohamed Abdelfattah, a Cornell assistant professor and Makora’s chief science officer, says the company began nearly two years ago with a thesis that much manual AI performance engineering could be automated. It started with high-performance GPU and other hardware kernels, then expanded toward inference servers, training pipelines, and reinforcement-learning rollouts.
The company previously operated as Maeko, a widely used name that another startup politely asked it to stop using. At one point, the team thought it would sell just a code-generation agent. Abdelfattah compares that to selling a compiler—commercially difficult and less sustainable than focusing on the resulting performance.
Makora therefore wants to combine pragmatic hardware-aware tuning with new algorithms. “Our value proposition is that we sell performance,” including the kernels, serving stack, numerics, and deployment work required to make gains material end to end.
2. Parallel drafts remove speculative decoding’s rewind penalty
Standard speculative decoding drafts K tokens sequentially, verifies them in parallel with a larger target model, then rewinds when tokens fail to match. Makora’s sequential Monte Carlo approach instead maintains N drafts, scores them with the target, evicts low-importance candidates, and duplicates high-scoring ones.
Abdelfattah’s analogy is pass@N versus pass@1 on a math problem: greater draft-model utilization means the system is “always accepting K tokens” and “never rejecting any token or rolling back.” The cited result is about 5× faster than the SGLang baseline, 2× faster than speculative decoding on SGLang with its experimental overlap scheduler, and “quite a bit faster” than the newer SSD baseline.
The essential qualification is that these results come from batch size one in the low-latency regime, where extra draft compute uses otherwise-unused capacity because the workloads are memory-bound, including the draft model. Larger batches saturate compute earlier than normal speculative decoding. The method is also lossy, which may deter users requiring exact accuracy. Empirically, multiple drafts with variable quality can even outperform the target, though if the draft is strictly worse, theory says N approaching infinity should match rather than exceed the target.
3. The algorithm is flexible, but production frameworks impose limits
SMC can sit atop MTP, EAGLE, or a specially trained predictor without requiring one particular speculation technique. Abdelfattah says it could even connect models with different tokenizations through additional tricks, though that remains a niche research use.
Makora already has an SGLang fork, but some “fancier things” are difficult inside existing production serving frameworks. The company is consequently building its own inference engine while still selecting and augmenting whichever open-source server best fits a workload.
4. Evaluation infrastructure turns reward hacking into a sellable product
Abdelfattah recalls an early reward-hacked demonstration as “maybe the worst 30 minutes of my life.” Makora responded with a rigid evaluation pipeline that traces whether generated functions are called correctly, rejects code that uses disallowed libraries, limits dependencies, and uses AI to detect reward hacks. Overall, the pipeline catches maybe 11 classes of reward hacks.
Confidence in that pipeline is now high enough that Makora exposes the same internal evaluation and profiling infrastructure through an authenticated API. “They give us a kernel and a problem. We give them the reward,” letting customers train foundation models against GPU-specific feedback; at least one customer had bought the service and was reportedly fairly happy.
Against simply paying Claude or GPT, the customer rationale Abdelfattah gives is consistency and end-to-end integration. A foundation model may produce example code or a proof of concept, but deployment requires correct layouts, interfaces, quantization modes, and integration into targets such as GGML on a Qualcomm NPU or SGLang on an AMD GPU.
5. Precision research and heterogeneous deployment define the larger bet
In FP4, separate negative-zero and positive-zero encodings waste one of only 16 quantization levels. Makora remaps the redundant zero to a learnable special value, effectively seeking “the accuracy of FP5 or FP5-something” with an FP4 memory footprint.
Weight-only quantization avoids the issue because weights are upcast later. For NVFP4 weight-activation quantization, however, the special value lies outside the FP4 range, so the straightforward NVIDIA tensor-core implementation requires a first pass and a sparse compensating pass, creating overhead. On AMD, FP6 shares FP4’s hardware path and throughput, allowing the special values to be upcast without the same penalty; NVIDIA’s FP6 instead shares the FP8 path. Abdelfattah’s verdict is that AMD hardware is “fundamentally quite strong,” while its software still needs to catch up.
Near term, Makora wants a “killer” inference engine, followed potentially by training and reinforcement learning, usable even by enterprise system administrators “by pushing buttons.” The grander vision disaggregates workloads across machine types—draft and target models included—while agents generate the right implementation for each device.
Research outputs, including SMC code, are intended to be open-sourced, with a closed-source enterprise layer built around them. The endpoint is to close the gap between “I bought a GPU” and “I’m getting the most performance out of that GPU.”