A batch of 32 red-green-blue images can have shape [32, 3, 224, 224]: batch, color channel, height and width. That is one layout, not a universal image convention; some libraries put channels last. A shape error often means an operation expected a different arrangement or number of values.
Three properties help when debugging: shape describes dimensions, data type describes how each value is represented, and device describes where the data is stored, such as processor or accelerator memory. Operations may require compatible shapes, types and devices. PyTorch’s tensor tutorial demonstrates these properties and the operations on them.
This matters for memory as well as correctness. A dense float32 tensor in that example holds 4,816,896 values, taking 19,267,584 bytes for the raw values, before other training or framework allocations. Learned weights, activations and embeddings can all be tensors. The term describes their representation, not whether their values were learned.
For a language-model example, read Transformer for an illustrative attention-weighted sum and a causal-mask example.
Sources
- PyTorch: Tensors tutorial — Demonstrates tensors, shape, data type, device and tensor operations; the image-memory calculation here is illustrative.
Go deeper
- NumPy: Broadcasting docs
Work through shape compatibility and array operations, a useful foundation for tensor debugging.