Build a deep learning framework from nothing, then use it to understand how machines hear and speak: 23 stages in three parts, from the maths to a speech model.
What this is
A path for a developer who wants to understand deep learning by building it. You write a small PyTorch-like framework on NumPy, one module at a time, then use it to read a real speech system: how audio becomes numbers, how a model recognises a speaker, and how a text-to-speech model turns a written description into a voice.
Who it is for
You program in Python and have met machine learning before: you know roughly what a loss and a gradient are. You have not built a framework, and you have not worked with audio.
What you will build
A NumPy framework with tensors, layers, losses, a data loader, autograd, optimisers, convolutions, attention and a transformer, then profiling, quantization, a KV cache and benchmarks. On the way it runs six historical milestones, from Rosenblatt's perceptron (1958) to a transformer (2017).
What you need first
Python, NumPy and school algebra; stage 1 covers the rest of the maths, with a reading for each piece. A computer with Python 3.10 or later. TrenTorch needs no GPU. The audio stages read code in the Alaap repo; running its demo needs the environment its README describes.
How long it takes
About 91 h in all: each TrenTorch module about 1.5 hours, the rest as each stage states. At five hours a week, about 18 weeks.
How to use it
Take the stages in order. Try each exit check before you open its answer, and if you cannot answer it, stay on the stage. Mark a stage done when you pass. Your progress stays in this browser; nothing is sent anywhere.
Set up
TrenTorch is an open framework you build yourself, module by module, in NumPy: its own take on the TinyTorch curriculum. Clone it and start the first module (the README has the Windows version):
git clone https://github.com/TrenTorch/TrenTorch.git
cd TrenTorch/TrenTorch_CLI
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt && pip install -e .
tren setup
tren module start 01
Alaap is an open research system for voice design: a written character description becomes a persistent voice identity that can speak any dialogue. Part 3 quotes its study guide and its architecture diagrams, which are in the architecture tab.
The path
The whole path. Green is part 1, blue part 2, amber part 3. Select a stage to open it.
How each stage is laid out
After this you can
the capability, stated as something you can do
Why it matters
where the idea shows up in real models
TrenTorch module
what you build, its objectives, and its reflection questions
From the Alaap guide
what to read and do, quoted from Alaap's study guide; paths such as alaap/acoustics.py are files in the Alaap repo
Exit check
one question with a short model answer; if you cannot answer it, do not move on
Part 1 · Foundations
about 13 h across 3 stages
No framework yet. The maths every later stage leans on, and what sound is as numbers. The two sound stages need no PyTorch, so you can also run them alongside part 2.
Stage 1
The maths you need, and where it is used
about 8 h
After this you can Do by hand the maths the path leans on: matrix shapes, the chain rule, logarithms and softmax, cross-entropy, cosine similarity, and what an FFT bin measures.
Why it matters Every later stage uses some of it, and the table says which. With it fresh, the code reads as the maths it is.
The maths, piece by piece
Work down the table. If a row is already easy, do its task on paper and move on; if not, read the lesson beside it. Each reading is a free lesson from AI Engineering from Scratch.
Topic
Be able to
Used in stage
Read
Vectors, matrices and shapes
Multiply a (32, 80) matrix by an (80, 256) one, and say the result's shape before you compute it.
Exit check A softmax outputs p = (0.7, 0.2, 0.1) and the correct class is the first. What is the cross-entropy loss, and its gradient with respect to the logits?
Show the answer
The loss is minus ln 0.7, about 0.36. The gradient is p minus the one-hot target: (-0.3, 0.2, 0.1). It is largest where the prediction is most wrong, which is why this pairing keeps training.
Stage 2
Sound as data: sampling and Nyquist
about 2 h
Alaap guide 2.1
After this you can Say what a sample rate buys you, find the Nyquist cliff on a real spectrum, and explain why analysing audio at the wrong rate shifts every frequency.
Why it matters Alaap's audio lives at four sample rates: corpus 24 kHz, WavLM 16 kHz, MioCodec 44.1 kHz, Sarvam 22.05 kHz. Mixing them up once reversed an experiment's result.
The cliff is the filter. A sample rate represents only frequencies below half of itself. Resampling down must remove what no longer fits, or it comes back at the wrong frequency as aliasing.
From the Alaap guide, Part 2: Sound as data 12 h
After this you can read alaap/acoustics.py and say what every function measures and
how it could be wrong.
Guide 2.1 Sampling and the time domain 2 h
Sample rate, Nyquist, bit depth, why 16 kHz / 22.05 kHz / 24 kHz / 44.1 kHz all appear here.
Intuition a waveform is a very long list of air-pressure readings. Nyquist says you can
represent frequencies up to half your sample rate and nothing above.
Why it matters the Alaap project's audio lives at four sample rates, corpus 24 kHz,
WavLM 16 kHz, MioCodec 44.1 kHz, Sarvam 22.05 kHz, and analysing one at another's rate
puts every frequency off by a ratio. That happened in S21 phase B and reversed a result.
Do: load a clip at 24 kHz, resample to 16 kHz, plot both spectra. Find the cliff.
Exit check A clip recorded at 24 kHz is resampled to 16 kHz. What happens to a component at 10 kHz, and why?
Show the answer
It is removed. At 16 kHz the Nyquist limit is 8 kHz, so the resampler's low-pass filter cuts everything above it. Without the filter, the 10 kHz energy would alias down to 6 kHz. That cut is the cliff on the plot.
Stage 3
Source and filter: what describes a voice
about 3 h
Alaap guide 2.3
After this you can Separate the glottal buzz (F0) from the vocal-tract tube (formants), and put each of Alaap's measurements on one side.
Why it matters Alaap's identity axes follow this split. f0_mean measures the source; vtl_cm and the formants measure the filter. spectral_tilt was meant to measure the source's shape but partly reads pitch.
Change the buzz and it is the same person; change the tube and it is not. Every identity axis Alaap keeps sits on one side of this split. Spectral tilt straddles it, and it caused the trouble.
Guide 2.3 The source–filter model 3 h
Glottal source (pitch, F0) vs vocal-tract filter (formants). Why they are separable and why
that separation is the whole basis of describing a voice.
Intuition the vocal folds make a buzz at F0. The mouth and throat are a tube that
resonates at certain frequencies, formants. Change the buzz, same person. Change the
tube, different person.
This is the single most important idea in Part 2.f0_mean measures the source;
vtl_cm and the formants measure the filter; spectral_tilt is supposed to measure the
source's spectral shape and, as S21 found, partly measures the buzz rate instead.
Exit check Two recordings of the same person, one speaking and one singing higher. Which measurements should move, and which should not?
Show the answer
F0 goes up, because it belongs to the source. Vocal-tract length and the formant pattern should not move: the tube is the same person. A measurement of the tube that moves with F0, as spectral tilt partly did, is confounded.
Part 2 · PyTorch from scratch
about 31.5 h across 11 stages
Build a deep learning framework on NumPy, one TrenTorch module at a time: tensors, layers, losses, data, autograd, optimisers, convolutions, embeddings and attention, then the tools that make a model run fast. Halfway through, map every piece to real PyTorch.
Stage 4
Tensors, built by hand
about 1.5 h
Alaap guide 1.1
TrenTorch 01 Tensor Foundation
After this you can Build a Tensor class on NumPy and predict the shape, dtype and memory of any operation before running it.
Why it matters Every model file is tensor shapes. Once you have built a tensor yourself, you read them without guessing.
A tensor is a buffer plus a way of reading it. Shape and strides decide what an index means, so transpose and view are free, and broadcasting repeats a smaller tensor without copying it.TrenTorch 01
Build the foundational Tensor class that powers all machine learning operations.
You have built
Nothing - this is our foundation!
You build
A complete Tensor class with arithmetic, matrix operations, and shape manipulation
It enables
Foundation for activations, layers, and all future neural network components
Build ittren module start 01, work in the notebook, then tren module test 01 and tren module complete 01. It exports trentorch.core.tensor; the source is TrenTorch_CLI/data/src/01_tensor/01_tensor.py.
Learning objectives
Implement a complete Tensor class with fundamental operations
Understand tensors as the universal data structure in ML
Master broadcasting, matrix multiplication, and shape manipulation
Intuition a tensor is a numpy array that remembers where it came from. That memory is
autograd.
Exit check A (32, 1, 80) tensor is added to an (80,) tensor. What is the result's shape, and what did broadcasting do?
Show the answer
(32, 1, 80). The (80,) tensor is treated as (1, 1, 80), repeated along the first two axes without copying memory, then added element by element.
Stage 5
Activations and layers
about 3 h
Alaap guide 1.3
TrenTorch 02 Activations
TrenTorch 03 Layers
Unlocks milestone 01
Unlocks milestone 02
After this you can Build Linear, ReLU, softmax and dropout as modules, and say what each one's parameters and buffers are.
Why it matters nn.Module, train() and eval() appear in every Alaap experiment. Dropout is the smallest layer that behaves differently in the two modes.
A layer is parameters plus a rule, and some rules depend on the mode. Linear owns weights. ReLU and softmax own nothing. Dropout behaves differently under train() and eval(), so Alaap calls eval() before every measurement.TrenTorch 02
Add the secret ingredient that makes neural networks intelligent - nonlinearity.
You have built
Tensor with data manipulation and basic operations
You build
Activation functions that add nonlinearity to transformations
It enables
Neural networks with the ability to learn complex patterns
Build ittren module start 02, work in the notebook, then tren module test 02 and tren module complete 02. It exports trentorch.core.activations; the source is TrenTorch_CLI/data/src/02_activations/02_activations.py.
Computational Cost Comparison. ReLU is the most popular activation function in hidden layers. Given what you implemented, why is ReLU computationally cheaper than Sigmoid or GELU?
Numerical Stability. Look at your Softmax implementation. Why did we subtract the maximum value before computing exponentials?
Sparsity and Efficiency. ReLU creates "sparsity" by zeroing negative values. Why might having many zero activations be beneficial for computation?
Activation Selection for Different Stages. Why do we typically use different activations for hidden stages vs. output stages of a computation?
The "Dying ReLU" Problem. If ReLU outputs 0 for some inputs, those inputs get no signal passed through -- they effectively "die." What situations might cause this, and why is it a problem?
Build Linear layers and Dropout for multi-layer neural networks.
You have built
Tensor class (Module 01) with all operations and activations (Module 02)
You build
Linear layers and Dropout regularization
It enables
Multi-layer neural networks, trainable parameters, and forward passes
Build ittren module start 03, work in the notebook, then tren module test 03 and tren module complete 03. It exports trentorch.core.layers; the source is TrenTorch_CLI/data/src/03_layers/03_layers.py.
Learning objectives
Implement Linear layers with proper weight initialization
Parameter Scaling and Memory. Consider three different network architectures for MNIST (28x28 = 784 input features, 10 output classes):
Dropout Training vs Inference. You have a Dropout layer with p=0.5 in your network. During training, we scale surviving values by 1/(1-p) = 2.0.
Weight Initialization Trade-offs. We initialize weights with scale = sqrt(1/in_features) (LeCun-style). For Linear(1000, 10), how does this compare to Linear(10, 1000)?
Layer Ordering Effects. In a typical layer block, we compose: Linear -> Activation -> Dropout. What happens if you change the order to: Linear -> Dropout -> Activation?
Production Deployment Memory. You're deploying a 3-layer network (784->256->128->10) to a mobile device with 10MB free memory.
Unlocks TrenTorch milestone 01 · Perceptron (1958) Build Rosenblatt's first neural network (forward pass). Needs modules 01-03. Run it with tren milestone run 01.
Unlocks TrenTorch milestone 02 · XOR Crisis (1969) Experience the XOR limitation that triggered AI Winter. Needs modules 01-03. Run it with tren milestone run 02.
Guide 1.3nn.Module
__init__ vs forward, parameters vs buffers, train()/eval(), state_dict.
Why it matters heremodel.eval() and torch.no_grad() appear in every experiment in
the Alaap repo. Know exactly what each turns off, they are not the same thing.
Exit check Dropout with p = 0.5 scales surviving activations by 2 during training. What does it do at eval time, and why the scaling?
Show the answer
Nothing: at eval every unit passes through unscaled. Scaling by 1/(1-p) during training keeps the expected activation the same in both modes, so the next layer sees the same magnitudes whether or not dropout is on.
Stage 6
Losses: measuring how wrong
about 1.5 h
TrenTorch 04 Losses
After this you can Build MSE and cross-entropy, compute softmax stably, and say why cross-entropy's gradient survives where squared error's saturates.
Why it matters The mixture density network in stage 20 trains on negative log-likelihood; cross-entropy is its one-component version.
Cross-entropy's gradient survives saturation. Paired with softmax, the log cancels the exponential and the gradient is p minus the target.TrenTorch 04
Build ittren module start 04, work in the notebook, then tren module test 04 and tren module complete 04. It exports trentorch.core.losses; the source is TrenTorch_CLI/data/src/04_losses/04_losses.py.
Learning objectives
Implement MSELoss for regression problems
Implement CrossEntropyLoss for classification problems
Implement BinaryCrossEntropyLoss for binary classification
Understand numerical stability in loss computation
Memory and Performance. Loss Function Selection for Large Vocabulary
Loss Function Performance Bottleneck. Performance Analysis
Numerical Stability. Debugging Exploding Loss
Production Considerations. Real-Time Inference Latency
Class Imbalance in Medical Diagnosis. Handling Class Imbalance
Batch Size and Loss Computation. Systems Thinking
Exit check With softmax and cross-entropy together, what is the gradient with respect to the logits?
Show the answer
p minus the one-hot target. It stays large while the prediction is wrong, however confident, so this pairing does not stall the way sigmoid plus squared error does.
Stage 7
The data pipeline
about 1.5 h
TrenTorch 05 DataLoader
After this you can Build a Dataset and a DataLoader that shuffle, batch and collate, and reason about what a batch costs in memory.
Why it matters Alaap streams clips from corpora it never downloads whole. How data reaches the model decides what an experiment can afford.
Shuffle the indices, not the data. A DataLoader permutes a list of integers, fetches only the batch it needs, and stacks it. The whole dataset never has to fit in memory.TrenTorch 05
Build data loading infrastructure for efficient ML training.
You have built
Tensor operations, activations, layers, and losses
You build
Dataset abstraction, DataLoader with batching/shuffling, and real dataset support
It enables
Efficient data pipelines that will feed hungry neural networks with properly formatted batches
Build ittren module start 05, work in the notebook, then tren module test 05 and tren module complete 05. It exports trentorch.core.dataloader; the source is TrenTorch_CLI/data/src/05_dataloader/05_dataloader.py.
Learning objectives
Understand the data pipeline: individual samples → batches → training
Implement Dataset abstraction and TensorDataset for tensor-based data
Build DataLoader with intelligent batching, shuffling, and memory-efficient iteration
Experience data pipeline performance characteristics firsthand
Create download functions for real computer vision datasets
The Batch Memory Budget. You're loading a large image dataset. Each image is a tensor of shape (3, 224, 224) stored as float32 (4 bytes per value). Your batch size is 256.
To Shuffle or Not to Shuffle?. You're training on a medical dataset where samples are ordered by patient (first 1000 samples = Patient A, next 1000 = Patient B, etc.). Consider these scenarios:
Data Loading Bottlenecks. Your program reports these timings per batch:
Memory Explosion with Large Datasets. You're training on 100GB of high-resolution medical scans. Your DataLoader code:
The Shuffle Memory Trap. You implement shuffling like this:
Exit check Samples are stored ordered by speaker. What goes wrong if you train without shuffling?
Show the answer
Each batch holds one speaker, so gradients chase that speaker and then the next: training oscillates, and the model forgets earlier speakers. Shuffling makes each batch a fair sample of the whole set.
Stage 8
Autograd: the gradient engine
about 1.5 h
Alaap guide 1.2
TrenTorch 06 Autograd
After this you can Build reverse-mode automatic differentiation: record the forward pass as a graph, then walk it backward with each operation's gradient rule.
Why it matters The Alaap guide's Part 1 promises you will know what backward() does, and what no_grad() and detach() switch off. You learn it here by building it.
The graph is a tape recorded on the way forward. Each operation saves what its gradient rule needs; backward() walks the tape in reverse and multiplies by each local derivative. no_grad() stops the recording, which is why it saves memory.TrenTorch 06
Build the autograd system that computes gradients automatically.
You have built
Tensor operations, activations, layers, losses, and DataLoader
You build
The autograd system that computes gradients automatically
It enables
The ability to optimize neural networks.
Build ittren module start 06, work in the notebook, then tren module test 06 and tren module complete 06. It exports trentorch.core.autograd; the source is TrenTorch_CLI/data/src/06_autograd/06_autograd.py.
Learning objectives
Enhance Tensor with automatic differentiation capabilities
Build computation graphs that track operations for gradient flow
Implement backward() method for reverse-mode differentiation
Create Function classes for operation-specific gradient rules
Test gradient correctness with mathematical validation
Inside the module
The Challenge
The Solution: Computational Graphs
Computational Graph Example
Memory Layout During Backpropagation
The Function Architecture
Function Base Class - The Foundation of Autograd
Operation Functions - Implementing Gradient Rules
Understanding Broadcasting in Gradients
AddBackward - Gradient Rules for Addition
MulBackward - Gradient Rules for Element-wise Multiplication
SubBackward - Gradient Rules for Subtraction
DivBackward - Gradient Rules for Division
MatmulBackward - Gradient Rules for Matrix Multiplication
SumBackward - Gradient Rules for Reduction Operations
Computational Graph Memory. A 10-layer neural network processes a single sample. Each layer performs matrix multiplication (matmul) and addition (bias). How much memory does the computation graph use compared to just storing the weights?
Gradient Accumulation. A weight matrix is shared between two computation paths in a network (like a tied-weights architecture). Why does gradient accumulation (`grad = grad + new_grad`) save memory during training? What's the trade-off?
Backward Pass Cost. A forward pass through a 3-layer MLP takes 10ms. Is the backward pass faster, slower, or the same speed as the forward pass? Why?
Graph Retention. You're training a language model that processes sequences of varying lengths. When should you call `.zero_grad()`? What happens if you forget?
Production Pattern. PyTorch and TensorFlow use `requires_grad` flags instead of always tracking gradients for every tensor. Why? What's the performance benefit of making gradient tracking opt-in?
Guide 1.2 Autograd
requires_grad, the graph, backward(), torch.no_grad(), why .detach() exists.
Intuition the graph is built forward as a side effect of computing, then walked
backward. Nothing is symbolic; it is a recorded tape.
Exit check y = x @ W. Given the gradient of the loss with respect to y, what are the gradients for x and W?
Show the answer
grad_x = grad_y @ W.T and grad_W = x.T @ grad_y. Two matmuls backward for one forward, so the backward pass costs about twice the forward.
Stage 9
Optimisers and the training loop
about 3 h
Alaap guide 1.4
TrenTorch 07 Optimizers
TrenTorch 08 Training
Unlocks milestone 03
After this you can Build SGD and Adam, write the full loop (zero_grad, forward, loss, backward, step), and train a network from scratch.
Why it matters This is the first real checkpoint: a small network trained from scratch, with every line and every gradient explained.
Every training script is these five lines. Adam keeps two running averages per parameter, so with the weights it holds three copies, against two for SGD with momentum. Evaluation switches both the layer mode and gradient recording off.TrenTorch 07
Build SGD, Adam, and AdamW optimizers for neural network training.
You have built
Tensor with gradients (Modules 01-06)
You build
SGD, Adam, and AdamW optimizers with sophisticated momentum and adaptive learning
It enables
Modern optimization algorithms that power state-of-the-art neural networks
Build ittren module start 07, work in the notebook, then tren module test 07 and tren module complete 07. It exports trentorch.core.optimizers; the source is TrenTorch_CLI/data/src/07_optimizers/07_optimizers.py.
Learning objectives
Implement SGD with momentum for stable gradient descent
Build Adam optimizer with adaptive learning rates
Create AdamW optimizer with decoupled weight decay
Understand memory and computational trade-offs in optimization algorithms
Inside the module
The Optimization Challenge
Our Optimizer Toolkit
The Mathematics Behind Movement
Understanding Momentum: The Physics of Optimization
Adam: Adaptive Learning for Each Parameter
AdamW: Fixing Weight Decay
Implementation Strategy
Gradient Extraction - Handling Tensor vs NumPy Gradients
Build complete training loops with checkpointing and scheduling.
You have built
Tensors, activations, layers, losses, DataLoader, gradients, and optimizers
You build
Complete training loops with checkpointing, scheduling, and gradient management
It enables
Full model training pipeline for the MLP milestone
Build ittren module start 08, work in the notebook, then tren module test 08 and tren module complete 08. It exports trentorch.core.training; the source is TrenTorch_CLI/data/src/08_training/08_training.py.
Learning objectives
Implement a complete Trainer class with train/eval modes
Build learning rate scheduling and gradient clipping
Create checkpointing for model persistence
Test training loops with immediate validation
Understand gradient accumulation patterns
Inside the module
Training Loop Mathematics
Train vs Eval Modes
Learning Rate Scheduling: Adaptive Training Speed
Gradient Clipping: Preventing Training Explosions
The Trainer Class: Orchestrating Complete Training
Trainer.__init__ - Setting Up the Training System
Trainer.train_epoch - The Core Learning Loop
Trainer.evaluate - Measuring Model Performance
Trainer.save_checkpoint - Persisting Training State
Memory Trade-offs. If you have a model with 1 million parameters and use Adam optimizer, what's the total training memory required?
Gradient Clipping. Why do we clip gradients by *global norm* rather than clipping each gradient independently?
Learning Rate Scheduling. Why does cosine annealing start with high learning rate and end with low learning rate?
Checkpointing Strategy. You're training for 100 epochs with 1GB checkpoints. How often should you save?
Train vs Eval Modes. Why is it crucial to set model.training = False during evaluation?
Unlocks TrenTorch milestone 03 · MLP Revival (1986) Train MLPs to solve XOR + recognize digits. Needs modules 01-08. Run it with tren milestone run 03.
Guide 1.4 The training loop
Dataset/DataLoader, loss, optimiser, zero_grad, LR schedules.
Exit check What happens if you forget zero_grad() between steps?
Show the answer
Gradients accumulate, so each update applies the sum of every earlier gradient. Steps grow and training diverges. Accumulating on purpose gives a large effective batch, as long as you zero every N steps.
Stage 10
The same things, in real PyTorch
about 3 h
Alaap guide 1.5
After this you can Map every TrenTorch piece to its PyTorch name, and pass Part 1's exit checklist on a real Alaap model: WavLM's hidden states.
Why it matters TrenTorch taught the mechanism, and Alaap is written in PyTorch. This stage closes the gap. It also covers out-of-memory that is system RAM, not VRAM.
Every PyTorch line in Alaap maps to something you built. The differences are devices, dtypes and scale; the mechanism is the one you wrote on NumPy.
Guide 1.5 The things that bite
CPU vs CUDA, and .to(device) on both model and data
memory: activations dominate, not weights
float32 vs bfloat16
why a model that "fits in VRAM" still OOMs
The Alaap repo's own version of that lesson: loading two models in one process segfaults on
a 15.7 GB box while the 6 GB GPU sits at 614 MiB. It is system RAM, not VRAM. See
HANDOFF.md §8b.
Do in the Alaap repo, no training required:
import torch, torchaudio
m = torchaudio.pipelines.WAVLM_BASE_PLUS.get_model().eval()
x = torch.randn(1, 16000) # 1 second at 16 kHz
with torch.no_grad():
hs, _ = m.extract_features(x, num_layers=12)
print(len(hs), hs[0].shape)
Then explain, out loud: why 12 tensors? What are the three dimensions of each? Why
no_grad()?
Check Why does hs[0].shape[1] change when you change the input length, and what is the
ratio? (This is the frame rate, and it is the first properly "audio" idea.)
Exit check model.eval() and torch.no_grad() are both used in every Alaap experiment. What does each switch off?
Show the answer
eval() changes layer behaviour: dropout stops dropping and batch norm uses its running statistics. no_grad() stops recording the graph, so no activations are kept for backward and memory falls. They are independent, and inference wants both.
Stage 11
Convolutions, and batch normalisation
about 1.5 h
TrenTorch 09 Convolutions
Unlocks milestone 04
After this you can Build Conv2d, pooling and BatchNorm2d, and see a convolution as one small filter slid along an axis.
Why it matters ECAPA-TDNN, Alaap's independent identity judge, is a 1-D convolution along time over audio frames, with dilation, then attentive pooling. This module teaches that mechanism.
ECAPA-TDNN is convolution along time. The same small filter slides over frames of audio, dilation widens what it sees, and pooling over time turns an utterance of any length into one vector.TrenTorch 09
Convolutions
Processing Images with Spatial OperationsModule source
Implement Conv2d, MaxPool2d, and AvgPool2d for image processing.
You have built
Complete training pipeline with MLPs, optimizers, and data loaders
You build
Spatial operations - Conv2d, MaxPool2d, AvgPool2d for image processing
It enables
Convolutional Neural Networks (CNNs) for computer vision
Build ittren module start 09, work in the notebook, then tren module test 09 and tren module complete 09. It exports trentorch.core.spatial; the source is TrenTorch_CLI/data/src/09_convolutions/09_convolutions.py.
Learning objectives
Implement Conv2d with explicit loops to understand O(N²M²K²) complexity
Build pooling operations (Max and Average) for spatial reduction
Understand receptive fields and spatial feature extraction
Analyze memory vs computation trade-offs in spatial operations
Inside the module
The Two Core Spatial Operations
Visual Example: How Convolution Works
Why Spatial Operations Transform ML
Understanding Convolution Step by Step
The Mathematical Formula
Pooling: Spatial Summarization
Why This Complexity Matters
Key Properties That Enable Deep Learning
Conv2d: Detecting Patterns with Sliding Windows
The Six Nested Loops of Convolution
Shared Input Validation
Conv2d Implementation - Building the Core of Computer Vision
Mobile ML Deployment. Why do mobile ML models prefer depthwise-separable convolutions over standard Conv2d?
Unlocks TrenTorch milestone 04 · CNN Revolution (1998) Build LeNet for image recognition. Needs modules 01-09. Run it with tren milestone run 04.
Exit check Batch normalisation has two separate effects. Name them.
Show the answer
First, it normalises each channel to zero mean and unit variance over the batch, which steadies and speeds up training. Second, batch statistics are noisy, so it adds regularising noise. At eval it uses running statistics instead of batch ones, which is why eval() matters.
Stage 12
Tokens and embeddings
about 3 h
TrenTorch 10 Tokenization
TrenTorch 11 Embeddings
After this you can Build a character and a BPE tokenizer, an embedding table, and positional encodings, and say where the embedding numbers physically come from.
Why it matters Modern TTS turns audio into tokens with a codec and generates them with a language model. The tokens differ from text tokens; the machinery is the same.
An embedding is a row of a weight matrix. A token id picks the row; gradient descent wrote the numbers. A codec does for audio what the tokenizer does for text, which is why TTS can be a language model.TrenTorch 10
Build tokenization systems that convert text to numerical representations.
You have built
Neural networks, layers, training loops, and data loading
You build
Text tokenization systems (character and BPE-based)
It enables
Text processing for language models and NLP tasks
Build ittren module start 10, work in the notebook, then tren module test 10 and tren module complete 10. It exports trentorch.core.tokenization; the source is TrenTorch_CLI/data/src/10_tokenization/10_tokenization.py.
Learning objectives
Implement character-based tokenization for simple text processing
Build a BPE (Byte Pair Encoding) tokenizer for efficient text representation
Understand vocabulary management and encoding/decoding operations
Create the foundation for text processing in neural networks
Vocabulary Size and Storage. You implemented tokenizers with different vocabulary sizes.
Sequence Length Trade-offs. Your character tokenizer produces longer sequences than BPE. For the text "machine learning" (16 characters):
Tokenization Coverage and Robustness. Your BPE tokenizer handles unknown words by decomposing into subwords.
Production Scale Considerations. A production language model serves 1 million requests per day, each with average 500 tokens.
TrenTorch 11
Embeddings
Converting Tokens to Learnable RepresentationsModule source
Build embedding layers that convert tokens into dense, learnable vectors.
You have built
Tensors, layers, tokenization (discrete text processing)
You build
Embedding lookups and positional encodings for sequence modeling
It enables
Foundation for attention mechanisms and transformer architectures
Build ittren module start 11, work in the notebook, then tren module test 11 and tren module complete 11. It exports trentorch.core.embeddings; the source is TrenTorch_CLI/data/src/11_embeddings/11_embeddings.py.
Learning objectives
Implement embedding layers for token-to-vector conversion
Understand learnable vs fixed positional encodings
Build both sinusoidal and learned position encodings
Analyze embedding memory requirements and lookup performance
Memory Scaling. You implemented an embedding layer with vocab_size=50,000 and embed_dim=512.
Lookup Complexity. Your embedding layer performs table lookups for token indices.
Positional Encoding Trade-offs. You implemented both learned and sinusoidal positional encodings.
Production Implications. Your complete EmbeddingLayer combines token and positional embeddings.
Exit check Where do the numbers in an embedding table come from?
Show the answer
They are ordinary weights: a (vocabulary x dimension) matrix, initialised randomly and updated by gradient descent. Looking up a token is selecting a row, which is the same as multiplying a one-hot vector by the matrix.
Stage 13
Attention and transformers
about 3 h
TrenTorch 12 Attention
TrenTorch 13 Transformers
Unlocks milestone 05
After this you can Build scaled dot-product and multi-head attention, then a full transformer block, and generate text with it.
Why it matters WavLM and Alaap's acoustic language model are transformers. Their layers encode different things, so the WavLM layer you pool changes what you measure.
Every token scores every other token. The n by n score matrix is where attention's quadratic memory comes from, and the square root of d_k keeps those scores out of softmax's flat tails.TrenTorch 12
Scaled dot-product attention and multi-head attention mechanisms
It enables
Transformer architectures, GPT-style language models, and sequence-to-sequence processing
Build ittren module start 12, work in the notebook, then tren module test 12 and tren module complete 12. It exports trentorch.core.attention; the source is TrenTorch_CLI/data/src/12_attention/12_attention.py.
Learning objectives
Implement scaled dot-product attention with explicit O(n²) complexity
Build multi-head attention for parallel processing streams
Understand attention weight computation and interpretation
Build TransformerBlock, GPT architecture, and autoregressive generation.
You have built
Tokenization, embeddings, attention mechanisms, and all foundational components
You build
TransformerBlock, complete GPT architecture, and autoregressive generation
It enables
Full language model training and text generation capabilities
Build ittren module start 13, work in the notebook, then tren module test 13 and tren module complete 13. It exports trentorch.core.transformers; the source is TrenTorch_CLI/data/src/13_transformers/13_transformers.py.
Learning objectives
Implement complete TransformerBlock with attention, MLP, and layer normalization
Build full GPT architecture with multiple transformer blocks
Add autoregressive text generation capability
Understand parameter scaling in large language models
Test transformer components and generation pipeline
Attention Memory Complexity. You implemented multi-head attention that computes attention matrices of size (batch, heads, seq_len, seq_len).
Residual Connection Benefits. Your TransformerBlock uses residual connections (x + attention_output, x + mlp_output).
Parameter Scaling Analysis. Your GPT model combines embeddings, transformer blocks, and output projection.
Autoregressive Generation Efficiency. Your generate() method processes the full sequence for each new token.
Unlocks TrenTorch milestone 05 · Transformer Era (2017) Prove attention with reversal, copy, and mixed sequence tasks. Needs modules 01-08, 11-13. Run it with tren milestone run 05.
Exit check Why divide QK^T by the square root of d_k?
Show the answer
With unit-variance entries, a dot product of d_k terms has variance d_k, so scores grow with dimension. Large scores push softmax into saturation where its gradients vanish. Dividing by the square root of d_k keeps the variance near one.
Stage 14
Making it run: inference, measured
about 9 h
TrenTorch 14 Profiling
TrenTorch 15 Quantization
TrenTorch 16 Compression
TrenTorch 17 Acceleration
TrenTorch 18 Memoization
TrenTorch 19 Benchmarking
Unlocks milestone 06
After this you can Profile a model, quantize and prune it, fuse and vectorise its hot paths, cache attention keys and values, and benchmark the result with confidence intervals.
Why it matters A trained model is only useful once it runs fast and cheaply enough. A speech service is judged on p95 latency and cost per minute under load, and these modules teach you to measure and move both.
Measure, shrink, speed up, prove. A speech service is judged on its p95 latency and cost per minute under load, so every change to the model is measured before and after.TrenTorch 14
Build profiling tools for parameters, FLOPs, memory, and latency.
You have built
Complete ML stack from tensors to transformers (Modules 01-13)
You build
Comprehensive profiling system for parameters, FLOPs, memory, and latency
It enables
Data-driven optimization decisions and performance analysis
Build ittren module start 14, work in the notebook, then tren module test 14 and tren module complete 14. It exports trentorch.perf.profiling; the source is TrenTorch_CLI/data/src/14_profiling/14_profiling.py.
Learning objectives
Implement a complete Profiler class for model analysis
Count parameters and FLOPs accurately for different architectures
Measure memory usage and latency with statistical rigor
From Implementation to Optimization: The Profiling Foundation
Parameter Counting: Model Size Detective Work
FLOP Counting: Computational Cost Analysis
Memory Profiling: The Three Types of Memory
Latency Measurement: Dealing with Reality
The Profiler Architecture
Why Helper Functions Matter
Why Parameter Counting Matters
FLOP Counting Visualization
FLOP Counting Strategy
Memory Usage Breakdown
Memory Measurement Strategy
Latency Measurement Challenges
All Modules (01-13) → Profiling (14)
(implementations) (measurement)
Reflection questions from the module (4)
FLOP Analysis. You implemented a profiler that counts FLOPs for different operations. For a Linear layer with 1000 input features and 500 output features:
Memory Scaling. Your profiler measures memory usage for models and activations. A transformer model has 125M parameters (500MB at FP32). During training with batch size 16:
Performance Bottlenecks. You built tools to identify compute vs memory bottlenecks. A model achieves 10 GFLOP/s on hardware with 100 GFLOP/s peak.
Profiling Trade-offs. Your profiler adds measurement overhead to understand performance. If profiling adds 5x overhead but reveals a 50% speedup opportunity:
Reduce model precision from FP32 to INT8 while preserving accuracy.
You have built
Complete ML pipeline with profiling (Module 14)
You build
INT8 quantization system with calibration and memory savings
It enables
4x memory reduction and 2-4x speedup for production deployment
Build ittren module start 15, work in the notebook, then tren module test 15 and tren module complete 15. It exports trentorch.perf.quantization; the source is TrenTorch_CLI/data/src/15_quantization/15_quantization.py.
Learning objectives
Implement INT8 quantization with proper scaling
Build quantization-aware training for minimal accuracy loss
Apply post-training quantization to existing models
Measure actual memory and compute savings
Understand quantization error and mitigation strategies
Build pruning, knowledge distillation, and low-rank approximation techniques.
You have built
Complete optimization pipeline with profiling (14) and quantization (15)
You build
Pruning (magnitude & structured), knowledge distillation, and low-rank approximation
It enables
Compressed models that maintain accuracy while using dramatically less storage and memory
Build ittren module start 16, work in the notebook, then tren module test 16 and tren module complete 16. It exports trentorch.perf.compression; the source is TrenTorch_CLI/data/src/16_compression/16_compression.py.
Learning objectives
Implement magnitude-based and structured pruning
Build knowledge distillation for model compression
Create low-rank approximations of weight matrices
Measure compression ratios and sparsity levels
Understand structured vs unstructured sparsity trade-offs
Master vectorization and kernel fusion for hardware-efficient execution.
You have built
Complete neural network foundation with tensors (01), layers (03), autograd (06), training (08), and CNNs (09)
You build
Acceleration techniques including vectorization and operation fusion
It enables
Hardware-efficient execution for production deployment
Build ittren module start 17, work in the notebook, then tren module test 17 and tren module complete 17. It exports trentorch.perf.acceleration; the source is TrenTorch_CLI/data/src/17_acceleration/17_acceleration.py.
Learning objectives
Implement vectorized operations for maximum throughput
Create fused operations to reduce memory bandwidth
Understand the relationship between compute and memory bandwidth
Analyze acceleration trade-offs in production systems
Inside the module
The Two Enemies of Performance
The Roofline Model - Your Performance Compass
Why This Module Matters
The SIMD Revolution
Memory Access Patterns: The Hidden Performance Killer
Implement KV caching for 10-15x faster transformer text generation.
You have built
Complete transformer architecture (Module 13) and profiling tools (Module 14)
You build
Memoization system that eliminates redundant computation through caching
It enables
Production-grade inference optimization using computational reuse
Build ittren module start 18, work in the notebook, then tren module test 18 and tren module complete 18. It exports trentorch.perf.memoization; the source is TrenTorch_CLI/data/src/18_memoization/18_memoization.py.
Learning objectives
Understand memoization as a general optimization pattern (cache results, avoid recomputation)
Apply memoization to transformers through KV caching
Implement KVCache with efficient memory management and O(1) updates
Build cache-aware attention that reuses previously computed keys and values
Measure dramatic speedup gains (10-15x) and understand memory trade-offs
Cache Size Calculation. A 12-layer transformer has 12 attention heads per layer, 64-dimensional embeddings per head,
Speed vs Memory Trade-off. Your KVCache makes generation 10× faster but uses several GB of RAM.
Batch Inference Scaling. Batch size 1 generates at 500 tokens/sec, using 50 MB cache. With KV cache, each sequence in a batch gets its own cache storage.
Cache Eviction for Long Conversations. Your `KVCache` has `max_seq_len=2048`. A chatbot conversation reaches 2048 tokens - the cache is full!
Production Reality - Multi-User Serving. ChatGPT serves millions of users. Each user's conversation needs its own KV cache.
Build the benchmarking infrastructure for the capstone competition.
You have built
Complete ML framework with profiling, acceleration, quantization, and compression
You build
TorchPerf benchmarking system for fair model comparison and performance evaluation
It enables
Systematic optimization combination and competitive performance evaluation
Build ittren module start 19, work in the notebook, then tren module test 19 and tren module complete 19. It exports trentorch.perf.benchmarking; the source is TrenTorch_CLI/data/src/19_benchmarking/19_benchmarking.py.
Learning objectives
Implement professional benchmarking infrastructure with statistical rigor
Learn to combine optimization techniques strategically (order matters!)
Build the TorchPerf class - a standardized performance evaluation framework
Understand ablation studies and systematic performance evaluation
Statistical Confidence in Measurements. You implemented BenchmarkResult with confidence intervals for measurements.
Measurement Overhead Analysis. Your precise_timer context manager has microsecond precision, but models run for milliseconds.
Benchmark Configuration Trade-offs. The BenchmarkSuite class uses configurable warmup_runs and measurement_runs parameters
MLPerf Compliance Metrics. You implemented MLPerf-style standardized benchmarks with target thresholds.
Optimization Comparison Analysis. Your analyze_optimization_techniques() generates recommendations for different use cases.
Unlocks TrenTorch milestone 06 · MLPerf Benchmarks (2018) Optimize and benchmark your neural networks. Needs modules 01-08, 14-19. Run it with tren milestone run 06.
Exit check Why does a KV cache make autoregressive generation faster, and what does it cost?
Show the answer
Without it, each new token recomputes keys and values for the whole prefix, so generation is quadratic. The cache stores them and computes only the new token's, making each step linear. The cost is memory per sequence, which grows with length and limits how many users one GPU serves.
Part 3 · Audio ML
about 46.5 h across 9 stages
Use what you built to read a real speech system, Alaap: spectrograms, speaker embeddings, neural codecs, text-to-speech as a language model, and evaluation that does not fool you. The text is quoted from Alaap's own study guide and architecture.
Stage 15
A speech system, end to end: what Alaap does
about 2 h
After this you can Say what Alaap does in one sentence, and trace one written description through its architecture to audio.
Why it matters Every later stage fills in one idea: a frozen TTS model driven by a learned speaker vector. Learn that shape first.
From the Alaap guide, Part 0: Orientation 2 hours, do this first
After this you can describe what Alaap does, and run it.
The product promise in one picture. A person writes a sentence describing a
character; a voice that has never existed comes back, and it stays that same voice across
every line of a script.
The frozen box is the whole idea. The TTS model is never trained. All the
learning lives in the mapper, which produces a speaker vector the frozen model already knows
how to consume. Watermarking is not a launch task, it happens on render #1
(invariant I7, EU Art. 50(2)).
Exit check Why is the TTS model never trained, and what would break if it were?
Show the answer
All the learning is in the mapper, which produces a vector the frozen model already knows how to consume. Train the TTS and the vector space moves under every stored identity: every minted voice drifts, and the independent drift and consistency judges stop meaning anything.
Stage 16
The rest of sound: frequency, and how estimators lie
about 7 h
Alaap guide 2.2
Alaap guide 2.4
After this you can Read a spectrogram's time and frequency resolution from its parameters, and read alaap/acoustics.py, saying what each function measures and how it can be wrong.
Why it matters These estimators produce every attribute Alaap captions. If measure() is wrong, the caption, the mapper and the adherence score are wrong together, and nothing complains.
The STFT is an FFT on a sliding window. Every parameter is a trade between time and frequency resolution, and both numbers appear inside spectral_tilt.
Guide 2.2 The frequency domain 3 h
DFT/FFT, window functions, the STFT, the spectrogram, mel scale, MFCCs.
Intuition the STFT is "run an FFT on a sliding window". Every parameter is a trade:
a long window gives fine frequency resolution and blurry time; a short one, the reverse.
Read any good DSP intro; Smith's Mathematics of the DFT if you want rigour.
Do in the Alaap repo:
import librosa, numpy as np
S = np.abs(librosa.stft(wav, n_fft=2048, hop_length=512))
n_fft=2048 at 24 kHz gives what frequency resolution? What time resolution? Both numbers
appear in spectral_tilt and both matter.
Guide 2.4 The estimators, and how each one lies 4 h
Read alaap/acoustics.py top to bottom. Every docstring is a lesson someone paid for.
function
the trap
f0_track
pYIN; octave errors on creaky voice
voiced_mask
an energy+ZCR VAD, not a periodicity detector; structurally capped near 0.60 and bimodal
formants
LPC order and analysis rate; run at ~2× max formant, not native
vocal_tract_length
never average through F2, it is vowel-dependent
spectral_tilt
OLS over linear FFT bins puts 43% of leverage on 27 bins
speaking_rate
needs text; silently NaN without it; g2p-en does not raise on Devanagari
Read the project skill .claude/skills/speech-feature-pitfalls/SKILL.md, it is the
compressed version of everything above, written from real failures here.
Check Why does a 250 Hz voice read a flatter spectral tilt than a 90 Hz voice with an
identical spectral envelope? (Answer in experiments/S21-spectral-tilt/RESULTS.md.)
Exit check At 24 kHz with n_fft = 2048 and hop 512, what are the frequency and time resolutions?
Show the answer
Frequency: 24000 / 2048, about 11.7 Hz per bin. Time: 512 / 24000, about 21 ms between frames, with each window 85 ms long. Finer frequency costs blurrier time, and the reverse.
Stage 17
Speaker identity: embeddings that recognise people
about 5 h
Alaap guide 3.1
Alaap guide 3.2
After this you can Explain how a speaker embedding is made, why it is invariant to channel, and why verification thresholds must be fitted in-domain.
Why it matters ECAPA is trained to ignore channel and quality, so it recognises you down a bad line. For the same reason Alaap uses it as the independent judge of identity, never of quality.
Train to tell speakers apart, then drop the classifier. The layer before it is the embedding, and it inherits whatever training made it ignore: channel and quality. That is why it judges identity and never quality.
From the Alaap guide, Part 3: Speaker identity 10 h
After this you can read the three papers the whole architecture rests on, and explain
why a speaker embedding is not just "an embedding".
Guide 3.1 From i-vectors to ECAPA 3 h
Read
Snyder et al., "X-Vectors: Robust DNN Embeddings for Speaker Recognition", ICASSP 2018
Desplanques et al., "ECAPA-TDNN: Emphasized Channel Attention, Propagation and
Aggregation in TDNN Based Speaker Verification", Interspeech 2020
Intuition train a network to tell speakers apart; throw away the classifier; the
penultimate layer is now a speaker embedding. Everything downstream inherits whatever that
training made it invariant to.
Why it matters here, and it is a rule, not a detail
ECAPA is trained to be INVARIANT to channel and quality, so it can recognise you down a
bad phone line. That is exactly why it must never be used to measure quality. It is the
project's independent judge for identity, and nothing else.
Guide 3.2 Cosine, EER, and calibration 2 h
Cosine similarity, thresholds, EER, and why a threshold from a paper does not transfer.
Do read alaap/metrics.py's CALIBRATION and E4's results. C_same = 0.6988,
C_diff = 0.2011. Then answer: why must these be fit in-domain? (E4 measured that doing
so halves EER.)
Exit check Why does fitting C_same and C_diff in-domain halve the equal error rate?
Show the answer
Cosine scores depend on corpus, channel and language. A threshold taken from a paper sits in the wrong place for Alaap's audio and trades errors badly. Fitted on Alaap's own same-speaker and different-speaker pairs, it sits where the two distributions cross.
Stage 18
Is the space navigable?
about 5 h
Alaap guide 3.3
Alaap guide 3.4
After this you can Argue, from three papers, that a speaker embedding space can be sampled and interpolated, and say what goes wrong in sparse regions and past the shell.
Why it matters The project rests on this: if the space could not be navigated, Tower A would not be worth training.
Guide 3.3 Is the space navigable? 3 h
This is the intellectual core of the project. Three papers, all in SOURCES.md:
the collapse and its fix. Per-dim rescaling: GVD −6.50 → −0.14. Dimension ranges span three orders of magnitude.
Intuition for the third one: if one dimension ranges over ±80 and another over ±0.5,
then an unweighted MSE loss is entirely about the first dimension. Every voice collapses
onto the same one. Rescale per dimension or the model learns nothing.
Guide 3.4 Moving through the space 2 h
Interpolation vs extrapolation; SLERP vs LERP; sparse regions.
Check Why does mint(novelty=0.7) produce voices closer to the centre of the space,
not further from it? (S9. It is not obvious, and the experiment retracted a claim over it.)
You cannot treat a speaker embedding space as isotropic. This is invariant
I1, and ignoring it collapses every identity onto one voice.
Interpolation between real speakers is well supported, 1,225 pairwise
interpolations held WER at 6.42–7.35% (Interspeech 2021). Extrapolation is not. That is why
novelty defaults to 0.0 and why the mapper
blends anchors rather than inventing points.
Involves
raw vectors, mean-centring, per-dimension rescale, PCA basis, the shell, SLERP
Exit check Why SLERP rather than linear interpolation between two speaker vectors?
Show the answer
Real speakers sit on a shell at roughly constant norm. A straight line between two points on a sphere cuts inside it, where the norm shrinks and the voice drifts toward the average. SLERP stays on the shell.
Stage 19
How a TTS model actually works
about 10 h
Alaap guide 4.1
Alaap guide 4.2
Alaap guide 4.3
Alaap guide 4.4
After this you can Read the model card of any open TTS system and predict whether it can serve Alaap's architecture.
Why it matters Codecs turn audio into integers. That is why modern TTS is a language model, and why Alaap can decode forty voices for the price of one.
Once audio is integers, a language model can write it. Content tokens are generated once and decoded through any speaker vector, which is why forty voices cost the same language-model time as one.
From the Alaap guide, Part 4: How a TTS model actually works 10 h
After this you can read the model card of any open TTS system and predict whether it
can serve this architecture.
Guide 4.1 The shape of the field 2 h
Autoregressive vs non-autoregressive; two-stage (text→tokens→waveform) vs end-to-end.
Read Kim et al., "VITS: Conditional Variational Autoencoder with Adversarial Learning
for End-to-End Text-to-Speech", 2021. It is the clean end-to-end baseline everything else
is a reaction to.
Guide 4.2 Neural audio codecs 3 hthe key unlock
Read
Zeghidour et al., "SoundStream: An End-to-End Neural Audio Codec", 2021
Défossez et al., "High Fidelity Neural Audio Compression" (EnCodec), 2022
Mentzer et al., "Finite Scalar Quantization: VQ-VAE Made Simple", 2023, FSQ, which
is what MioCodec uses
Intuition a codec learns to turn a waveform into a short sequence of integers and back.
Once audio is integers, a language model can generate it. That is why modern TTS is a
language model.
Why FSQ specifically: it replaces a learned codebook with a fixed grid, quantise each
of 5 dimensions to [8,8,8,5,5] levels, giving 12 800 combinations, no codebook collapse,
no auxiliary losses.
And the trap, which cost the Alaap project a week: two FSQ codecs with the same levels
share no codebook. Every index is valid in both. Nothing raises. You get fluent speech
saying different words. Diagram 10 in architecture.html.
Guide 4.3 Self-supervised speech representations 2 h
Read Chen et al., "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack
Speech Processing", 2022. HuBERT if you want the predecessor.
Intuition train on unlabelled audio by masking and predicting. Different layers end up
encoding different things, early layers pitch and speaker, middle layers phonetic content.
Why it matters here the naturalness gate is distance to the real-speech manifold in
WavLM features, and which layer you pool changes what you measure. Layer 0 correlates
with F0 at −0.578, which is the DNSMOS trap in disguise. See S20b and S23.
Guide 4.4 Flow matching and diffusion, enough to argue 3 h
Read Lipman et al., "Flow Matching for Generative Modeling", 2023. Skim the maths;
get the picture.
Intuition diffusion learns to reverse noise step by step. Flow matching learns a
velocity field that transports noise to data along a straight-ish path, fewer steps, same
destination.
The one A/B that matters here, from SOURCES.md:
arXiv:2406.08812, Interspeech 2024, flow matching wins
fidelity (FAD 3.559 vs 5.244) and loses adherence (SRCC 0.60 vs 0.74). A hybrid is
best. The trade is bidirectional, which is why the Alaap project's S3 plans MDN first.
This split is what makes the Indic experiments cheap. It is also where the
project's most expensive bug lived.
Why forty voices cost the same LM time as one. Content tokens are generated
once and decoded through every speaker vector: L LM
generations + N×L codec decodes, not N×L
LM generations. That is the two-tower split being real in the weights, and it makes voice
comparisons exact, since every voice says an identical waveform of content.
Involves
MioCodec encode, content tokens at 25 Hz, global embedding (128), decode
Exit check Two FSQ codecs both use levels [8, 8, 8, 5, 5]. Why can you not feed one's tokens to the other's decoder?
Show the answer
Equal levels give the same 12,800 valid indices, but each codec learned its own mapping from index to sound. Every index is accepted and nothing raises, so you get fluent speech saying different words.
Stage 20
Description to voice
about 5 h
Alaap guide 5.1
Alaap guide 5.2
Alaap guide 5.3
After this you can Read the mapper code and the paper it copies, and explain why the mapper blends top_k = 2 anchors.
Why it matters One description fits many valid voices. A mixture density network predicts a family of them instead of one point.
One description, many valid voices. A mixture density network outputs a distribution over speaker vectors and is trained by the likelihood of the real one; a regressor would average them into one central voice.
From the Alaap guide, Part 5: Description → voice 5 h
After this you can read the mapper code and the paper it copies, and explain the design
choice in top_k=2.
Guide 5.1 Mixture Density Networks 2 h
Read Bishop, "Mixture Density Networks", 1994 (tech report, ~20 pages, very readable).
Intuition a normal regressor predicts one output per input. But one description
("a warm low voice") corresponds to many valid voices. An MDN predicts a mixture of
Gaussians instead of a point, so the model can say "any of these".
Why it matters it is the difference between one voice per description and a family. The
reference implementation is MDNLayer(256, 256, num_gaussians=10, dim_wise=True).
Read Shimizu et al., "PromptTTS++: Controlling Speaker Identity in Prompt-Based
Text-to-Speech Using Natural Language Descriptions", 2024. Then the code:
line/promptttspp, Apache-2.0, weights downloadable.
This is the only open working implementation of exactly the contract Alaap needs:
natural-language description → speaker embedding → frozen TTS.
Guide 5.3 Data-Speech and measure-first captions 1 h
Read Lyth & King, "Natural language guidance of high-fidelity text-to-speech with
synthetic annotations", 2024 (the Parler-TTS / Data-Speech line).
Do compare against alaap/acoustics.py: Data-Speech computes 9 columns from 5
tools, equal-width bins. Alaap uses percentile bins and adds six attributes Data-Speech
has not got. Read Binner's docstring for why percentile, it is a real finding about
skewed corpora.
Check Why does Binner regress HNR on log-F0 before binning it? (39.9% of HNR
variance was pitch, a measurement artefact, and it made "a high, harsh voice" impossible to
satisfy.)
Every box here is real code in alaap/mapper.py and
alaap/service.py. This is the path a description takes to become a
stored identity.
Two things here are counter-intuitive and both were measured.top_k = 2 rather than 1: blending two anchors beats pure retrieval,
but blending four contracts toward the centroid and loses diversity.
novelty = 0.0: pushing minted vectors away from real anchors makes
them more central, not less (S9). And the store keeps a seed clip as well as a
vector, invariant I2, because an encoder change orphaned every
stored vector once already.
Involves
bin labels from text, MiniLM text encoder, hybrid retrieval, top_k = 2 SLERP blend, novelty, four gates, IdentityStore
Exit check Why an MDN rather than a regressor that predicts one speaker vector?
Show the answer
Trained with squared error on a one-to-many mapping, a regressor predicts the average of every valid voice, which is a blurred central voice. An MDN predicts mixture weights, means and spreads, so sampling gives distinct plausible voices.
Stage 21
Evaluation, and why most of it is wrong
about 5 h
Alaap guide 6.1
Alaap guide 6.2
Alaap guide 6.3
Alaap guide 6.4
After this you can Design a measurement that will not fool you: the right diversity statistic, the right judge, and a listening test with its discard rule written first.
Why it matters This part has more hard-won knowledge, and less published literature behind it, than any other part of the project.
Vendi counts distinct things. Forty voices that are really twenty distinct ones score about twenty. Normalised Vendi divides by n, so the same fraction over different set sizes means different counts.
From the Alaap guide, Part 6: Evaluation, and why most of it is wrong 5 h
After this you can design a measurement that will not fool you. This is the part of the
project with the highest ratio of hard-won knowledge to published literature.
Guide 6.1 The MOS-predictor trap 1.5 h
Read Reddy et al. DNSMOS (2021) and Saeki et al. UTMOS (2022), then
RESEARCH/06-evaluation-harness.md §5.2.
correlation with mean F0
humans
−0.059
DNSMOS
−0.788
UTMOSv2
−0.722
They also move less than 0.1 on prosodic corruption that costs humans 1.84 MOS points.
For a project whose point is spanning the pitch range, a MOS gate systematically rejects
high-pitched voices for a reason humans do not share. Invariant I9.
Guide 6.2 Diversity: the Vendi score 1 h
Read Friedman & Dieng, "The Vendi Score: A Diversity Evaluation Metric for Machine
Learning", 2023.
Intuition the exponential of the entropy of the eigenvalues of a similarity matrix. If
you have 40 items that are really 20 distinct things, Vendi × n ≈ 20, "effective voices".
The trap the Alaap repo hit: normalised Vendi is a fraction of the maximum for a set of
that size. 0.361 over 95 items and 0.362 over 38 are not the same thing, they are ~34
and ~14 effective voices. S9 retracted a published claim over this.
Guide 6.3 Intelligibility 0.5 h
WER/CER, ASR as a judge, and why CER not WER for Indic scripts.
Guide 6.4 Listening tests 2 hread the skill, it is better than the literature
Read.claude/skills/listening-tests/SKILL.md, then
experiments/S14-rate-control/RESULTS-S14c.md as a worked failure.
The compressed lessons:
Comparative, not single-stimulus, when the baseline is itself imperfect
Positive and negative controls, always
Write the discard condition down before sending the set, then honour it
Compute the false-positive rate of that condition. S14c's fired on a coin flip
Ask "does this say X?", never "does this sound right?"
Check Design a test for "does the rate knob sound processed?" without letting the
listener answer on clip length. (There is a real tension here; S14b and S14c both failed it
in opposite ways.)
Five gates, four of them calibrated on a corpus rather than chosen. Each
answers a different question, and the project spent nineteen experiments not noticing that
none of them asked about naturalness.
The naturalness gate is a proposal, not a gate. It has never rejected
anything, the floor of 75 was chosen, not measured. Everything to its left has
blocked renders. Note also that drift/consistency/uniqueness/CER all pass on audio a listener
calls synthetic (S19): they are identity and intelligibility gates, and the absence of a
quality gate was invisible for nineteen experiments precisely because nothing was
named one.
Exit check Why would a MOS predictor make a bad naturalness gate for Alaap?
Show the answer
DNSMOS correlates with mean pitch at r = -0.788; human ratings correlate at -0.059. It rejects high-pitched voices for a reason listeners do not share, and works against the diversity the project exists to create.
Stage 22
The roads not taken
about 6 h
After this you can Argue the backend choice instead of inheriting it: what each rejected system is, why it was rejected, and what would change the answer.
Why it matters Licences and model shapes decide more of this project than benchmarks do. Five licence chains checked, five mis-declared.
From the Alaap guide, Part 7: The roads not taken 6 h
After this you can argue about backend choice rather than inherit it.
For each: what it is, why it was rejected, and what would change the answer.
system
why not
what would change it
CosyVoice 2/3
speaker vector exists but use_spk_embedding: False, no spk2info.pt shipped. Best-shaped control surface though, separate instruct_text, speed, <strong>, [laughter]
it is the documented fallback if the Direction channel had failed
F5-TTS
flow-matching, strong; but IndicF5 is a fine-tune of it under CC-BY-NC-4.0 despite an MIT card
a clean-licence Indic fine-tune
Zonos / ZONOS2
v0.1 encoder derives from VoxBlink2 (CC-BY-NC-SA); ZONOS2 is 15.34 GB bf16, does not fit 12 GB. Ships SLERP blending + additive emotion vectors, the same ideas as here
more VRAM, and a licence trace
IndexTTS-2
licence bars using it or its outputs to improve any AI model; "Use" includes running
nothing, it is structurally incompatible with a training project
XTTS-v2
code MPL-2.0, weights CPML; Coqui defunct, the fork cannot relicense
nothing
Indic Parler-TTS
zero speaker-embedding code. Identity is a name token in the same string as style
a fork that exposes a vector
Kokoro-82M
tiny and good; ref_s[:,:128]→decoder, [128:]→prosody is an undocumented free timbre/prosody split
worth revisiting for a small/fast tier
ReadRESEARCH/03-tts-backends-english.md and RESEARCH/04-indic-track.md in full.
They are the most reusable thing in the repo: primary-source verification of a dozen systems.
Do pick one rejected system and write a one-page case for reopening it. Then check
whether RESEARCH/08-licensing-propagation.md already kills your case.
Check Why does invariant I4 say a licence declaration is "a claim, not evidence"?
How many chains were mis-declared out of how many checked?
Exit check Why does invariant I4 call a licence declaration a claim, not evidence?
Show the answer
Because derived models inherit their parents' terms, and the declaration on the derived model is often wrong: every one of the five Indic chains checked was mis-declared upstream. Only tracing the chain to the training data is evidence.
Stage 23
Capstone: Alaap, from scratch
about 1.5 h
Alaap guide, closing ideas
TrenTorch 20 Capstone
After this you can Put the whole stack together: a model you trained, served and benchmarked, and the five ideas that cost the project longest to learn.
Why it matters Alaap becomes a system you trained, instead of an assembly of other people's models.
Every box, and the stage that taught it. The system context from stage 15 again. Now you can say how each part works, because you built or read every one.TrenTorch 20
Build ittren module start 20, work in the notebook, then tren module test 20 and tren module complete 20. It exports trentorch.olympics; the source is TrenTorch_CLI/data/src/20_capstone/20_capstone.py.
Learning objectives
Use Module 19's benchmarking tools to measure model performance comprehensively
Apply optimization techniques from Modules 14-18 to improve baseline models
Generate standardized JSON submissions following industry best practices
Validate submissions against a schema for reproducibility
Compare baseline vs. optimized models with quantitative metrics
Share your results with the TrenTorch community in a professional format
The five ideas that took the Alaap project longest to learn
If you remember nothing else:
A suspiciously good or suspiciously bad number is a bug report about your setup.
Recorded ~17 times here. Every single time.
State the property before you read the result. Otherwise you will rationalise.
An axis that fails to separate is a bug in the estimator until proven otherwise.
Two "noise" axes turned out to be broken measurements.
Effect size is not significance. A gap of 8 points with a standard error of 6 is
nothing, and it will look like a finding.
A metric is validated on a corpus, not in the abstract. Re-run the validation when the
corpus, model, or codec changes.
These are the three project skills in .claude/skills/, statistical-claims,
speech-feature-pitfalls, listening-tests. Read them at week 1 and again at week 13; they
will mean different things.
Exit check A new result is suspiciously good. What do you do first?
Show the answer
Treat it as a bug report about the setup. Write down the property you expected before reading further, then check the measurement, the corpus and the comparison. The project recorded this about seventeen times, and each time it was the setup.
Alaap, how the whole thing fits together
Each diagram lists what it involves, its topics, and the stages of the path that teach it.
1 · System context
The product promise in one picture. A person writes a sentence describing a
character; a voice that has never existed comes back, and it stays that same voice across
every line of a script.
The frozen box is the whole idea. The TTS model is never trained. All the
learning lives in the mapper, which produces a speaker vector the frozen model already knows
how to consume. Watermarking is not a launch task, it happens on render #1
(invariant I7, EU Art. 50(2)).
Why this architecture at all, and the one experiment that had to pass before
anything else was worth building.
E1 was the gate. If the speaker space were a discriminative
d-vector space it would resist synthesised priors, TacoSpawn measured
g2s 0.35 vs s2s 0.20
(arXiv:2111.05095). E1 returned
g2s ≈ s2s, so the space is navigable and Tower A is worth training.
The dashed return path is why drift is a gate: the vector that goes
in is not the vector that comes out (Panariello et al., Interspeech 2023).
Every box here is real code in alaap/mapper.py and
alaap/service.py. This is the path a description takes to become a
stored identity.
Two things here are counter-intuitive and both were measured.top_k = 2 rather than 1: blending two anchors beats pure retrieval,
but blending four contracts toward the centroid and loses diversity.
novelty = 0.0: pushing minted vectors away from real anchors makes
them more central, not less (S9). And the store keeps a seed clip as well as a
vector, invariant I2, because an encoder change orphaned every
stored vector once already.
Involves
bin labels from text, MiniLM text encoder, hybrid retrieval, top_k = 2 SLERP blend, novelty, four gates, IdentityStore
Identity is fixed; performance is per line and never enters the identity
record. That separation is invariant I5.
Why the phase vocoder and not resampling. Naive resampling shifts F0 by
−54.7/+91.3 Hz and collapses speaker identity to ECAPA 0.10, a different person. A phase
vocoder changes duration while preserving pitch. Why OGG and not Opus: Opus removes the
AudioSeal watermark; Vorbis survives at 0.95 detection, and is the Unity/FMOD/Wwise
default.
Five gates, four of them calibrated on a corpus rather than chosen. Each
answers a different question, and the project spent nineteen experiments not noticing that
none of them asked about naturalness.
The naturalness gate is a proposal, not a gate. It has never rejected
anything, the floor of 75 was chosen, not measured. Everything to its left has
blocked renders. Note also that drift/consistency/uniqueness/CER all pass on audio a listener
calls synthetic (S19): they are identity and intelligibility gates, and the absence of a
quality gate was invisible for nineteen experiments precisely because nothing was
named one.
6 · Reversibility, the property that makes captions checkable
Most caption datasets ask an audio model to describe a voice. You
cannot verify that. Alaap measures first and writes prose from the measurements, so a claim
can be checked by re-measuring the render.
This loop is why the estimator audits mattered. If
measure() is wrong, the caption is wrong, the mapper learns the wrong
association, and adherence scores it against the same wrong measurement, so nothing ever
complains. S17 found vtl_cm broken this way: it had been silently
dropped as "noise" by two experiments before anyone checked the formula.
Involves
measure() on 17 attributes, percentile Binner, captions from bins, re-measurement, adherence
You cannot treat a speaker embedding space as isotropic. This is invariant
I1, and ignoring it collapses every identity onto one voice.
Interpolation between real speakers is well supported, 1,225 pairwise
interpolations held WER at 6.42–7.35% (Interspeech 2021). Extrapolation is not. That is why
novelty defaults to 0.0 and why the mapper
blends anchors rather than inventing points.
Involves
raw vectors, mean-centring, per-dimension rescale, PCA basis, the shell, SLERP
Seventeen things are measured. Only three describe a voice, and working out
which three took five experiments.
Describable-space size is not the constraint. S18 put every arm on one
axis: 125 describable cells give 22.3 effective voices; 15,625 give 19.6. 125× more
description moved capacity by −2.7. What limits the catalogue is the number of
strong axes, and no new axis has ever helped, while two repaired ones have.
Involves
f0_mean, spectral_tilt, vtl_cm, dropped and unused attributes
English and Indic differ in model, codec, sample rate and vector width. The
renderer adapter has carried an Indic implementation since S6, because if the interface
cannot express Indic, the interface is wrong (ADR-003, guard 2).
Invariant I4, five for five. Every Indic licence chain checked was
mis-declared upstream: IndicF5→F5-TTS, SPRING_F5→F5-TTS, VoiceSculptor→Llasa-3B,
Zonos→VoxBlink2, Indic-Mio→Emilia/Expresso. A licence declaration on a derived artefact is a
claim, not evidence.
Involves
Renderer protocol, Qwen3-TTS 1.7B (English), Indic-Mio with MioCodec, vector widths 2048 and 128
10 · Inside MioCodec, the content / identity split
This split is what makes the Indic experiments cheap. It is also where the
project's most expensive bug lived.
Why forty voices cost the same LM time as one. Content tokens are generated
once and decoded through every speaker vector: L LM
generations + N×L codec decodes, not N×L
LM generations. That is the two-tower split being real in the weights, and it makes voice
comparisons exact, since every voice says an identical waveform of content.
Involves
MioCodec encode, content tokens at 25 Hz, global embedding (128), decode
Per-line performance on a backend with no instruct
input. Three mechanisms were tried; one works, one is bounded, one is inert.
The text channel carries no direction at all (S13/S13b), filed upstream as
Report 4. The listener's verdict on tagged renders was blunt: "they are all the same
speaker, all warm but no distinct feeling at all." What survives is a genuinely useful
property: identity is provably robust to whatever the text says.
Involves
emotion as alpha times tau on the vector, rate by phase vocoder, text tags (inert)
Sixteen modules in alaap/. Arrows are imports, so
they point from a module to the one it depends on. Nothing below the line touches a model.
The one structural rule worth copying.encoder.py
(ECAPA) is deliberately never the thing being optimised, it is the independent judge. Drift
and consistency are scored by a model that has never seen MioCodec, so a good score cannot be
an artefact of the generator agreeing with itself.
Thirty-five experiments. What matters is not the count but which ones
changed a decision, and how many later ones exist only to check an earlier one.
Read left to right, then down. The top row is a working system built in
about four days. Everything below it is one question, "why only 22 voices?", answered four
different ways, all negative, plus an instrument track that grew out of distrusting the
answers. The right-hand box holds the finding that should have redirected the project:
retrieval reaches 87% where minting reaches 58%.
Involves
E0 to E15, S2 to S24, the ~22-voice wall, four failed levers
