1.5-Hour Transformer Training Beats LLMs on ARC-AGI Benchmark

TL;DR

Independent researcher Mithil Vakde has trained a small transformer from scratch in just 1.5 hours on a single RTX 5090, achieving 44% on the ARC-AGI-1 benchmark for a total compute cost of 67 cents. The model outperforms many large language models and matches the performance of Tiny Recursive Models (TRM) and Hybrid Recursive Models (HRM), challenging assumptions about scale and sample efficiency in AI.

The Breakthrough: Small Model, Big Results

Vakde, an IIT Bombay graduate, trained his transformer using a novel approach that emphasizes sample efficiency over raw scale. The entire training run cost just 67 cents on cloud GPU rental, and the model also scores 7% on the harder ARC-2 benchmark. This is a significant upgrade from his previous model, which was slower, more expensive, and less accurate.

The work has already caught the attention of prominent researchers, including Lucas Beyer, Jeremy Howard, and Rohan Anil, with discussions going viral on X (formerly Twitter). The code is open source and available on GitHub, making it accessible for researchers worldwide to build upon.

How It Works: Technical Details

The approach converts each input-output pair into a sequence of tokens, which are then autoregressively trained on by a small transformer. This is done from scratch at test time on both the train and eval sets, with test labels hidden. Each puzzle receives a separate additive embedding to enable cross-task learning, and positional information is encoded using 3D RoPE embeddings.

Key architectural improvements over his previous work include:

• Modern architecture: SwiGLU activation instead of GELU, RMSNorm instead of LayerNorm
• Scaling up: 8 layers instead of 4
• Better data diversity: Improved shuffling and augmentation strategies
• NorMuon optimizer: Replacing AdamW, reducing training costs significantly
• Flash attention: With varlen training and flex attention kernels for inference

One surprising finding was that training only on output tokens (supervised style) performs slightly better than including input tokens (unsupervised), despite yielding worse test loss. Vakde admits he doesn't fully understand why, but suspects it relates to finite model capacity.

Sample Efficiency: The Core Motivation

Vakde's primary motivation is tackling what he calls "the most important problem in AI today": sample efficiency. ARC-AGI is an ideal benchmark for this because it has very few samples (only 1,000 puzzles) in a high-dimensional space, making it a true test of generalization rather than memorization.

The benchmark is also designed to be a metalearning test, where each puzzle uses a different rule but shares common concepts. This makes it incredibly easy for humans but challenging for AI systems that rely on pattern matching from vast training data.

Interesting Behavior and Ablations

The model exhibits some counterintuitive behaviors. Despite worse test loss, it scores better on the benchmark, suggesting that optimizing for validation loss may not always align with actual performance on reasoning tasks. This points to a potential failure mode in current sample efficiency research methodologies.

Ablation studies revealed that the biggest contributors to performance are the 3D RoPE positional encodings and per-task embeddings. Removing either causes the score to drop from 44% to around 24-25%. Other ablations showed that:

• Training on inputs performs slightly worse (~39%)
• Restricting training to ARC-1+ConceptARC performs about the same (~40%)
• CompressARC-style training drops performance to ~18%

Addressing Criticism and Controversy

The approach has faced criticism from some researchers, particularly regarding the use of test-time training on eval puzzle inputs. Vakde addresses these concerns directly, arguing that this is a form of transductive reasoning, a well-established concept in machine learning dating back to Vapnik.

He also clarifies that the ARC testing policy allows test-time training, and that the criticism stems from a misunderstanding of the policy. The policy's intent is to prevent human designers from using eval set information to build inductive biases, not to prevent AI systems from learning during inference.

Why It Matters

This result challenges the prevailing wisdom that solving ARC-AGI requires massive models, recursive architectures, or extensive synthetic data. Vakde's transformer, trained in under two hours for under a dollar, matches the performance of models that use significantly more complex approaches.

The implications extend beyond ARC-AGI. It suggests that sample efficiency, not just scale, could be the key to advancing AI reasoning capabilities. For an industry increasingly focused on massive training runs and enormous compute budgets, this work demonstrates that smarter approaches can achieve comparable or better results at a fraction of the cost.

What's Next

Vakde believes 65% on ARC-AGI-1 is achievable within the transformer framework, and he's inviting the research community to contribute. He suggests several avenues for improvement, including new positional encoding schemes like PoPE, further architecture modernization, and eliminating data augmentations.

The open-source nature of the work means that researchers worldwide can iterate on his approach. With compute costs this low, the barrier to entry for ARC-AGI research has dropped dramatically, potentially democratizing access to cutting-edge AI reasoning research.