Cartesia says H-Net matches a 3B model with 10x less training
The tokenizer-free architecture learns from raw bytes and reaches SmolLM3-level results with roughly one-tenth the data and compute, according to Cartesia.
Cartesia says a new version of its H-Net architecture can learn language directly from raw bytes instead of relying on a fixed tokenizer. In controlled scaling experiments, the company reports that learned tokenization gave H-Nets a three-to-fivefold compute-efficiency advantage over comparable tokenized transformers.
At larger scale, Cartesia trained a 3.67-billion-parameter byte-level H-Net and compared it with Hugging Face’s open SmolLM3-3B recipe. The H-Net reached roughly the same average benchmark score after about 1 trillion token-equivalents, versus 11.1 trillion tokens for SmolLM3. Cartesia says the models were matched on training compute per byte, so the shorter training run translates into a similar reduction in total compute.
The practical promise is broader than replacing one preprocessing step. A model that discovers useful units from raw data could handle code, multilingual text and unfamiliar formats without inheriting a tokenizer’s fixed vocabulary. Cartesia is also testing the approach on audio, DNA and images.
The results are Cartesia’s own and have not yet been independently replicated. The company also notes that fair byte-versus-token comparisons are methodologically difficult, so the headline efficiency gain should be treated as an early research result rather than a proven replacement for mainstream transformers.
Cartesia
