All tutorials · Updated September 21, 2026 · tinygrad 8ad8f73
Fusion lets a kernel compute intermediate values without storing each Tensor operation’s result as a separate array. The benefit can include fewer launches and less memory traffic, but the scheduler must preserve dependencies and select a workable kernel.
from tinygrad import Tensor
x = Tensor([1.0, 2.0, 3.0]).realize()
fused = ((x + 1) * 2).tolist()
intermediate = (x + 1).realize()
staged = (intermediate * 2).tolist()
assert fused == staged == [4.0, 6.0, 8.0]
print(fused)
Run with DEBUG=2 to inspect actual execution. In the fused expression, addition is available to the multiplication as graph structure. In the staged expression, realize() explicitly requests the intermediate value before constructing the second computation.
Equal outputs do not establish equal kernel counts or costs. Inputs, constants, copies, layout requirements, and optimizer choices can all affect the observed execution.
run_rangeify in indexing.py computes realization and coordinate information. get_kernel_graph performs cleanup, storage conversion, and kernel splitting. This replaces the old LazyBuffer/ScheduleItem account.
Reduction makes fusion more involved than concatenating elementwise instructions. A producer may be reused across several consumers; recomputing it can cost more than materializing it. Different consumers can also demand different iteration domains or layouts.
Check numerical behavior first. Then measure launches, intermediate storage, and synchronized runtime under identical conditions. Fewer kernels is not automatically faster: a fused kernel can increase register pressure or duplicate expensive computation.
A useful regression describes a specific dependency/layout case and checks its result. A performance test should additionally establish that the intended path ran and that the comparison used equivalent inputs and warmup.
Original chapter by Di Zhu: historical version.