A Markov Chain is a mathematical model used to represent a sequence of events or states where the probability of each event depends only on the state of the previous event. Here's a concise explanation of the key concepts:

1. States: A Markov Chain consists of a set of possible states that a system can be in.
2. Transitions: The chain moves from one state to another over time.
3. Markov Property: The crucial characteristic is that the probability of moving to any particular state depends only on the current state, not on the sequence of states that preceded it.
4. Transition Probabilities: These are the probabilities of moving from one state to another, usually represented in a matrix.
5. Memoryless: The process is "memoryless" - the future state depends only on the present state, not the past states.

A simple example to illustrate the concept of a Markov Chain is weather prediction:

• States: Sunny, Rainy, Cloudy
• If it's Sunny today, there might be a 70% chance it's Sunny tomorrow, 20% chance it's Cloudy, and 10% chance it's Rainy.
• These probabilities only depend on today's weather, not on what the weather was like in previous days.

The formula is:

$P(A|B) = P(B|A) * P(A) / P(B)$

Where:

• P(A|B) is the posterior probability of A given B
• P(B|A) is the likelihood of B given A
• P(A) is the prior probability of A
• P(B) is the marginal probability of B

In the context of diffusion models, Bayes Rule plays a crucial role in the denoising process. Here's how it applies:

a) Prior:

• This represents our initial belief about the distribution of clean images.
• In diffusion models, this is typically a learned distribution.

b) Likelihood:

• This represents the probability of observing a noisy image given a clean image.
• In diffusion models, this is defined by the forward noising process.

c) Posterior:

• This is what we're trying to estimate: the probability of a clean image given a noisy observation.
• This is used in the denoising process to gradually reconstruct the image.

d) Denoising as Bayesian Inference:

• The denoising process in diffusion models can be viewed as a series of Bayesian inference steps.
• At each step, we're using Bayes Rule to estimate the most likely less-noisy image given the current noisy image.

e) Reverse Process as Bayesian Inference:

• The reverse process in diffusion models, which generates images from noise, can be seen as iterative application of Bayes Rule.
• We start with a very noisy image (almost pure noise) and gradually apply Bayes Rule to estimate less noisy versions.

In the context of diffusion models, μ and ε refer to different aspects of the noise prediction process:

1. ε (epsilon):
• ε represents the noise that was added to the image at a particular timestep.
• The model is typically trained to predict this noise.
• By learning to predict the noise, the model can later reverse the process during generation.
2. μ (mu):
• μ represents the mean of the distribution from which the next (less noisy) image is sampled.
• It can be derived from the predicted noise (ε) and the current noisy image.

The relationship between predicting ε and μ:

1. Direct noise prediction:
• The model is trained to predict ε directly.
• This prediction tells us what noise was likely added to get from the original image to the current noisy version.
2. Deriving μ from ε:
• Once we have the predicted ε, we can use it to calculate μ.
• μ essentially represents our best guess at what the less noisy image should look like.
3. Mathematical relationship:
• $μ = (x_t - √(1-α_t) * ε_θ(x_t, t)) / √α_t$
• Where $x_t$ is the noisy image at timestep $t$, $α_t$ is a scheduling parameter, and $ε_θ$ is the predicted noise.

The advantage of predicting ε instead of μ directly:

• It's often easier for the model to learn to predict the noise rather than the denoised image directly.
• Predicting ε allows for more stable training and better results in practice.

During the generation process, the model uses these predictions to gradually denoise random noise into a coherent image.

Improving the log-likelihood during the training of a diffusion model can have significant impacts on the model's performance and capabilities. Let's break this down:

• The log-likelihood is a measure of how well the model fits the training data.
• Higher log-likelihood indicates that the model assigns higher probability to the observed data.

Impacts of Improved Log-Likelihood include better data representation, more efficient sampling, and better compression, but do not always represent a perceptual improvement in output quality. More interesting are the benefits to downstream tasks:

• An improved log-likelihood suggests that the model has learned a more accurate probability distribution of the data.
• In some cases, improved log-likelihood can lead to faster sampling or the ability to use fewer denoising steps while maintaining quality.
• For tasks that use the diffusion model as a component (e.g., image inpainting, super-resolution), a better log-likelihood often translates to improved performance.

Of course, every engineering and business decision also imposes trade-offs:

• Some models might have high log-likelihood but still produce less visually appealing samples.
• Extremely high log-likelihood on the training set without corresponding improvement on a validation set might indicate overfitting.
• Improving log-likelihood often requires more complex models or longer training times, which comes with computational costs.

• Positional Embeddings
• ResNet Blocks
• ConvNext Blocks
• Attention Modules (which you may be familiar with from Transformers)

…and a lot more things I don’t have time to explain here. Most important takeaway is that an older architecture from an adjacent discipline was repurposed to orchestrate an ensemble method that works better.