4

Home

Max edited this page 2023-01-30 22:25:46 +00:00

Table of Contents

• What is Tinygrad
• Some features of tinygrad
• Speed strategies
• Optimizing operations
• Simplicity of computational accelerators
• Code
• File structure
• Important Classes

What is Tinygrad

Tinygrad is a deep learning framework, that is compact, simple, fast, and resource-efficient. At the user API level, tinygrad looks like any modern tensor library, where one works with a Tensor class to perform computation. Tinygrad has two main strategies to reach the goal of being fast and resource-efficient. The first strategy is to reduce memory access by merging and removing redundant operations done on tensors. The second strategy is to make it easy to make good computational accelerators.

Some features of tinygrad

• Tensor manipulation and fast computation, using various backends.
• Auto grading.
• Lazy evaluation.
• Compactness, Tinygrad aims to be compact (originally aiming for less than 1000 lines of code).

Speed strategies

Optimizing operations

The computation of tensors are deferred until the result is needed. This deference allows tinygrad to make a graph of what operations are done to a tensor, then optimize the graph by removing redundant operations from the graph and merging certain operations.

Simplicity of computational accelerators

Making efficient computational accelerators is complex, by keeping the interface to the computational accelerator simple, tinygrad makes it easier to get all the performance out of the accelerator hardware.

Code

File structure

├── LICENSE
├── README.md
├── accel - Accelerator implementations
├── docs - Tinygrad documentation
├── examples - Tinygrad usage examples
├── extra - Extra things that are not considered core tinygrad
├── test - Tests for tinygrad
├── tinygrad - Core tinygrad
│   ├── ast.py - Definitions for representing abstract syntax trees
│   ├── graph.py - Creating visual graphs of the Tensors dependencies.
│   ├── helpers.py - Various helper functions
│   ├── lazy.py - Things that deal with the lazy evaluation of Tensors
│   ├── llops - Definitions for low-level operations 
│   ├── mlops.py - Mid-level operations 
│   ├── nn - Neural network-level things
│   ├── ops.py - Basic definitions for Tensor operations 
│   ├── shape - Things that deal with Tensor shapes
│   └── tensor.py - Main Tensor class

Important Classes

• Tensor - Main Tensor class
• LazyOp - Class used to track operations that should be executed on a Tensor.
• DeviceBuffer - Abstract class representing an accelerator hardware backing buffer of a Tensor object
• ShapeTracker - Tracks “changes” made to a Tensor's shape. This is used to determine if a Tensor's backing memory is contiguous. Knowledge of the memory layout of a tensor allows an accelerator to load a large memory section with many values in a single memory request, rather than loading each value from memory by itself in many requests.
• View - Class that represents some way (a shape) of looking at the values of a Tensor.
• ZeroView - Class that does virtual 0 padding around the values of a Tensor.