tinygrad-notes

Tutorials on Tinygrad

View on Github | View on Website

Tutorials on tinygrad internals, adapted from Di Zhu’s original notes. Updated September 21, 2026 against tinygrad 8ad8f73.

The chapters now explain the current UOp, rangeification, scheduling, compilation, and runtime implementation. Existing chapter addresses are preserved; each chapter links its original version. These are independent tutorials, revised with AI assistance.

Start here

• Setup and how to use these notes
• Introduction: from Tensor values to execution
• Worked compiler trace: transpose → pad → add → reduce — captured UOps, CPU/Metal source, and an independent regression oracle
• Shapes, movement operations, and indexing — replaces the ShapeTracker chapter
• Scheduling with LINEAR, CALL, and buffer dependencies
• The current UOp IR
• Pattern matching and graph rewriting
• UOp interning and shared graphs

Computation and optimization

• Dot products
• Matrix multiplication
• Convolution windows and arange
• Operator fusion
• Following a reduction through UOps
• Reshape composition and dimension simplification
• Symbolic sizes and means
• BEAM search and kernel optimization
• Upcasting
• Index expressions after range splitting
• Code generation
• Instruction ordering and control flow
• Kernel names and axis colors

Execution and inspection

• TinyJit capture and replay
• Graph rewrites with VIZ
• Profiling compilation, execution, and replay
• Host memoryviews and Tensor storage
• Backend responsibilities
• Command queues and completion
• Sharding and multi-device execution
• Adding an accelerator

Hardware details

• Following the Metal runtime
• Tensor-core tiles and fragment layouts
• LOP3 truth tables

Runnable Python blocks are checked by the example runner. See validation coverage for the tested revision, devices, and hardware limits.