2026

Koda, an LLM trained from scratch

A 1.27 billion parameter language model trained from scratch, just to understand how they work inside. LLaMA-style decoder-only (24 layers, GQA, SwiGLU, RoPE), trained in JAX/Flax NNX on 2 L40S GPUs. Checkpoints published on Hugging Face, with HF, GGUF and MLX exports.

github.com huggingface.co

KodaLite-1.3B model card on Hugging Face: honest TL;DR about undertraining, benchmark table, Safetensors metadata — The KodaLite-1.3B Hugging Face model card, honest TL;DR and benchmarks included

1.27Bparameters, trained from scratch to understand

Overview

KodaLite-1.3B is a language model I trained from scratch, not for the size, but to understand the internals: a LLaMA-style decoder-only architecture (24 layers, hidden 2048, GQA 32/8, SwiGLU, RMSNorm pre-norm, RoPE), GPT-2 BPE tokenizer. The full pipeline runs in JAX + Flax NNX on 2 NVIDIA L40S GPUs in bf16: pretraining on SlimPajama (~1.6 billion tokens, ~25 hours) with a crash-recovery orchestrator, LoRA SFT on Dolly and OASST, NTK-aware context extension from 1024 to 2048 tokens. Checkpoints are published on Hugging Face with exports to Transformers, GGUF (llama.cpp, Ollama, LM Studio) and MLX, plus a homemade 8-task zero-shot benchmark.

LLaMA-style architecture

Architecture table from the Koda-v0.1 GitHub README: parameters, layers, hidden size, GQA attention, SwiGLU, RMSNorm, RoPE, context and vocab — The architecture table from the Koda-v0.1 GitHub README

A 1.27 billion parameter decoder-only model: 24 layers, hidden 2048, GQA 32/8, SwiGLU, RMSNorm pre-norm, RoPE, GPT-2 BPE tokenizer. Native context is 1024 tokens, extended to 2048 in phase 3.

Pretraining on 2 L40S

The pipeline runs in JAX + Flax NNX in bf16 on 2 NVIDIA L40S GPUs (96 GB VRAM). About 1.6 billion SlimPajama tokens in 25 hours, with a crash-recovery orchestrator so no progress is lost.

An honest benchmark

Benchmark table from the Hugging Face model card: nine models ranked by average accuracy, KodaLite-1.3B last, with per-task evaluation results on the right — The 8-task zero-shot ranking on the Hugging Face model card

A homemade 8-task zero-shot benchmark (HellaSwag, ARC, WinoGrande, PIQA, BoolQ, OpenBookQA, LAMBADA) compares KodaLite with 8 models of about 1 billion parameters. It ranks last, and the card says so.

Chinchilla scaling law

Chinchilla section of the Hugging Face card: table of tokens seen versus target, KodaLite at 6.5 %, GPT-2-XL, Pythia and TinyLlama far above — The Chinchilla table on the card: 6.5 % of the needed tokens

The card explains why a model 10 times bigger than GPT-2-124M scores below it: 1.64 billion tokens seen, 6.5 % of the Chinchilla target of about 25 billion. Tokens matter more than parameters at this budget.

HF, GGUF, MLX exports

File tree of the KodaLite-1.3B-GGUF Hugging Face repo: Q4_K_M, Q8_0 and f16 files with their sizes — The GGUF repo on Hugging Face, three published quantizations

Checkpoints are published on Hugging Face with exports to Transformers, GGUF (llama.cpp, Ollama, LM Studio) and MLX, in fp16 and 8-bit, so the model can be used outside JAX.

Challenges

Training a 1.27B parameter model on a limited GPU budget (2x L40S, 96 GB VRAM)
Sustaining a ~25 hour pretraining run without losing progress
Extending context from 1024 to 2048 tokens after pretraining
Making the model usable outside JAX (Transformers, GGUF, MLX)

Solutions

JAX + Flax NNX implementation in bf16 with a crash-recovery orchestrator
SlimPajama pretraining, then LoRA SFT (Dolly, OASST) and an EOS token fix
NTK-aware context extension without full retraining
Export pipeline to Hugging Face Transformers, GGUF and MLX (fp16 and 8-bit)

Results

KodaLite-1.3B published on Hugging Face (YoAbriel/KodaLite-1.3B, GGUF and MLX variants)
Full pretraining: ~1.6B SlimPajama tokens in ~25h on 2x L40S
Homemade 8-task zero-shot benchmark to measure what the model can actually do
Public code on GitHub (Koda-v0.1)

Technologies

JAXFlax NNXPythonLoRASlimPajamaHugging FaceGGUFMLX

Next project