Generative Modeling: A Gentle Journey with Distribution Matching¶

Xing Mei · 08-27-2026

Modern generative modeling has accumulated a lot of diverse concepts and ideas: pretraining has adversarial learning, sample divergences, denoising diffusion, and flow matching; post-training has policy gradients and preference losses; deployment has distillation and quantization. Each stage arrives with its own vocabulary and its own folklore. For an engineer with a basic ML background, that sprawl makes the underlying mechanism hard to understand and implement.

This journey focuses on continuous-valued generative modeling: distributions on dequantized pixel or continuous latent spaces and paths that move continuously through them. Discrete data and token-generation models are outside its scope.

This journey offers a practical, unified methodology for understanding and implementing these stages. The organizing idea is simple. Start from a source random variable

$$ Z\sim P_Z, $$

push it through a parameterized generator,

$$ X=G_\theta(Z), \qquad X\sim Q_\theta=(G_\theta)_\#P_Z, $$

and train the induced model distribution by matching it to a target distribution $P$:

$$ \min_\theta D\!\left( (G_\theta)_\#P_Z,\, P \right) = \min_\theta D(Q_\theta,P) $$

Here $P_Z$ is the easy-to-sample source distribution; $G_\theta$ is the learnable generator; $Q_\theta$ is the distribution produced by that generator; $P$ is the target distribution we want; and $D$ says what it means for the two distributions to agree. Each stage of generative modeling can be formulated in this same form, but it places different constraints on $G_\theta$, $Q_\theta$, $P$, and $D$. We may be able to draw samples but not evaluate a density, differentiate through the generator but not the target, or observe only a reward or a teacher trajectory. Finding a computable form of $D$ while walking around these constraints is the main art of generative modeling. We derive everything in this journey from distribution matching. If distribution matching becomes the first phrase that comes to mind whenever you think about generative modeling, this note has succeeded.

We will keep adding material as the journey continues through pretraining, sampling, conditioning, RL post-training, distillation, and quantization. Comments and advice are welcome.

Edit History:

08-28-2026: add Chapter 1, The Language of Distributions
08-27-2026: add the main journey and the pretraining table of contents

Table of contents¶

Each chapter is a short, independently executable notebook. It introduces the notation and examples it needs, so no chapter requires another notebook to have been run first. The content may evolve as the journey continues.

Part	Chapter	What it establishes
I · Foundations	1. The Language of Distributions [colab]	Continuous laws, samples, densities, pushforwards, exposed operations, joint laws, and conditional expectations.
	2. A Practical Divergence Toolkit	What each divergence compares, what information it requires, and how it becomes computable.
	3. The Constraint Matrix	Turn model, target, divergence, and constraints into one reusable recipe.
II · Pretraining	4. Strategy A: Manufacture a Density	Normalizing flows, variational autoencoders, and autoregressive models.
	5. Strategy B: Compare Samples Directly	GANs, MMD, sliced Wasserstein, and Sinkhorn divergence.
	6. From Sample Divergences to Drifting Fields	Energy fields, Sinkhorn drifting, and the three-body scattering rule.
	7. The Fixed-K Sampler as One Generator	Why endpoint matching works, what unrolled training costs, and what it leaves behind.
	8. Flow Matching from Distribution Matching	Introduce continuous paths and endpoint couplings, then derive the local velocity regression and its marginal motion.
	9. One Flow Field, ODE and SDE Samplers	Build the differential toolkit, read a score from the velocity, and preserve the same marginal path with or without noise.
	10. DDPM Path Matching and DDIM Sampling	Derive DDPM from path-law matching, then reuse its slice-wise predictor for DDIM jumps.
	11. What Pretraining Leaves Behind	Compare the generator zoo and explain why reusable fields become the substrate for later stages.
III · Sampling and conditioning	12. The Sampling Operator	How solver, step budget, stochasticity, and guidance change the law produced by fixed weights.
	13. Couplings and Optimal Transport	How pairing choices reshape paths and make few-step generation easier.
	14. Conditional Generation and Classifier-Free Guidance	Turn one target into a family and understand guidance as a sampling-time tilt.
IV · Post-training	15. RL Post-Training	Match a reward-tilted target that can be scored but cannot be sampled directly.
	16. Step Distillation	Train a fast student to match the distribution produced by a slower teacher.
	17. Quantization Recovery	Recover the full-precision model distribution inside a low-bit hypothesis class.
V · Synthesis	18. The Complete Constraint Matrix	Place the full generative-model lifecycle in one model-target-divergence table.
	19. When Are Two Methods the Same?	Separate equivalence of targets, objectives, training rules, and generated laws.
	20. Recap and Limits	Collect the reusable method and state honestly where the distribution-matching lens strains.

Citation¶

@misc{mei2026generativejourney,
  author = {Xing Mei},
  title = {Generative Modeling: A Gentle Journey with Distribution Matching},
  year = {2026},
  url = {https://xingmei.io/notes/generative_modeling_journey.html}
}