All tutorials · Updated September 21, 2026 · tinygrad 8ad8f73
In tinygrad’s kernel optimizer, upcasting means expanding work over an axis so one execution lane handles several values. It does not mean converting a value to a wider dtype.
For sixteen elementwise outputs, one possible organization has sixteen lanes producing one value each. Another has four lanes producing four values each. They cover the same logical output, but differ in address arithmetic, instruction count, vectorization opportunities, and register demand.
from tinygrad.codegen.opt import Opt, OptOps
from tinygrad.uop.ops import AxisType
upcast = Opt(OptOps.SPLIT, axis=0, arg=(4, AxisType.UPCAST))
unroll = Opt(OptOps.SPLIT, axis=0, arg=(4, AxisType.UNROLL))
assert upcast != unroll
print(upcast)
print(unroll)
These are requested transformations, not executable kernels. postrange.Scheduler validates a split against its source range and allowed target axis role. An output axis and a reduction axis are not interchangeable.
The old Linearizer.upcast() and OptOps.UPCAST examples have been replaced by range roles and SPLIT. codegen’s expander builds a range map for UPCAST and UNROLL axes and replaces selected ranges with shaped constants before later lowering.
Processing several adjacent values can expose vector operations and reuse computations. Expanding too much work can increase live values, register pressure, instruction size, or compilation cost. On a GPU, it can also reduce occupancy.
The optimizer’s choices depend on target capabilities and layout. Four values per lane does not promise a single four-wide load: alignment, validity masks, memory layout, and renderer support still constrain the generated instructions.
Use BEAM or the existing heuristics to compare legal candidates. For correctness, cover tails and masked accesses as well as evenly divisible sizes; for performance, report the target and generated program.
Original chapter by Di Zhu: historical version.