All tutorials · Updated September 21, 2026 · tinygrad 8ad8f73
A Tensor program describes values and dependencies. tinygrad turns that description into kernels, chooses an execution order, compiles the kernels for a device, and runs them. The same UOp representation participates at several stages, but the operations permitted at each stage differ.
from tinygrad import Tensor
from tinygrad.uop.ops import Ops
a = Tensor([1.0, 2.0, 3.0, 4.0]).realize()
b = Tensor([10.0, 20.0, 30.0, 40.0]).realize()
c = a + b
assert c.uop.op is Ops.ADD
assert c.tolist() == [11.0, 22.0, 33.0, 44.0]
print(c.tolist())
Before realization, c.uop describes addition. Its sources reference the input computations/storage. tolist() requests execution and obtains host-readable data. realize() requests execution but returns a Tensor.
Use initialized values when testing correctness. Tensor.empty does not promise zeros. Earlier versions of this tutorial showed zeros from newly allocated memory; that observation was allocator-dependent.
In tensor.py, Tensor.realize selects outputs that need storage and calls linear_with_vars. That method bufferizes outputs, transforms the computation into a call, updates Tensor references, and asks scheduling for a linear execution graph and bound symbolic values.
This matters when inspecting internals: schedule_linear() is not a read-only pretty printer. It participates in preparing the Tensor graph. Use a disposable computation for schedule inspection, or execute the returned schedule as described in the scheduling chapter.
The current scheduling path includes:
prepare_rangeify: prepare the tensor graph.get_kernel_graph: propagate ranges, express indexing, introduce storage where needed, and split kernels.create_schedule: order kernel calls according to dependencies.The resulting LINEAR contains CALL nodes. A call includes its body and arguments. Before compilation, a compute body can be a SINK; after compilation it becomes a PROGRAM.
engine/realize.py implements run_linear. Outside JIT replay it first compiles and links the linear graph. lower_and_compile finds bodies that still need compilation and deduplicates work using compiler cache keys.
codegen/init.py implements to_program. For a kernel sink it applies lowering/optimization, constructs program information, performs instruction selection for applicable ISA renderers, linearizes, renders, and compiles. A completed PROGRAM contains the lowered sink, instruction sequence, source representation, and binary.
Execution dispatches each call to the appropriate handler. A compiled kernel resolves input buffers and scalar values, allocates storage when needed, loads the runtime program, computes launch dimensions, and invokes it. Copy and graph calls use other handlers.
This separates three questions: which Tensor values are needed, which computations belong in a kernel, and how that kernel executes on a particular device.
Save the first example as example.py:
DEV=CPU DEBUG=2 python3 example.py
DEV=CPU DEBUG=4 python3 example.py
The first command reports execution statistics. The second also exposes generated source. CPU is useful for a reproducible starting point; choose a supported GPU backend to inspect its kernels. Counts, names, launch dimensions, and timings are observations of a particular configuration, not fixed properties of the Python expression.
Next read movement and indexing, scheduling, and pattern matching.
Original chapter by Di Zhu: historical version.