All tutorials · Updated September 21, 2026 · tinygrad 8ad8f73
For matrices A[M,K] and B[K,N], output C[i,j] is the sum of A[i,k] * B[k,j] over k. tinygrad’s ordinary dot path expresses that computation through reshaping, broadcasting, multiplication, and reduction.
from tinygrad import Tensor
a = Tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]).realize()
b = Tensor([[7.0, 8.0], [9.0, 10.0], [11.0, 12.0]]).realize()
expanded = a.reshape(2, 1, 3) * b.T.reshape(1, 2, 3)
manual = expanded.sum(axis=2)
expected = [[58.0, 64.0], [139.0, 154.0]]
assert expanded.shape == (2, 2, 3)
assert manual.tolist() == expected
assert (a @ b).tolist() == expected
print(expected)
The broadcasted expression has logical shape [M,N,K]. Reducing its last dimension yields [M,N]. That does not require allocating the complete M*N*K product array: the expression remains lazy until a consumer requests values, and the compiler can combine multiplication and reduction in a kernel.
Explicitly materializing expanded changes the question. It asks tinygrad to make those intermediate values available, potentially creating storage and execution work that a fused expression avoids.
The implementation is OpMixin.dot in mixin/op.py, inherited by Tensor. It checks dimensions, reshapes the operands to insert broadcast dimensions, transposes the right operand, multiplies, reduces over the last axis, and casts to the chosen result type.
The same implementation handles vector and batched cases with different shape normalization. Do not generalize the two-dimensional reshape above to every rank; read the dx, dw, and axis_w branches.
The optional accumulation dtype also matters. Equal mathematical sums can differ after floating-point reassociation or changes in accumulation precision.
Rangeification turns movement operations into indexing. Kernel optimization can split output/reduction ranges, arrange local work, unroll work, or match tensor-core shapes where the renderer supports them.
The high-level expression alone does not establish that tensor cores were used. Inspect the lowered program and target instructions. The BEAM chapter explains measured candidate selection; the tensor-core chapter explains what evidence distinguishes a matrix expression from matrix instructions.
Run this example with DEBUG=4 to inspect source. For performance measurements use larger inputs, warm up, synchronize, and record the device, renderer, precision settings, and compiler commit.
Original chapter by Di Zhu: historical version.