For any kernel $k : \X \times \X \to \R$, there exists a feature map $\varphi : \X \to \mathcal F$, where $\mathcal F$ is a special Hilbert space called the reproducing kernel Hilbert spacereproducing kernel Hilbert space (RKHS) corresponding to $k$. This is a space of functions, $f : \X \to \R$. These spaces satisfy a special key condition, called the reproducing property: $\langle f, \varphi(x) \rangle_\F = f(x)$ for any $f \in \F$.
For any kernel $k : \X \times \X \to \R$, there exists a feature map $\varphi : \X \to \mathcal F$, where $\mathcal F$ is a special Hilbert space called the reproducing kernel Hilbert space (RKHS) corresponding to $k$. This is a space of functions, $f : \X \to \R$. These spaces satisfy a special key condition, called the reproducing property: $\langle f, \varphi(x) \rangle_\F = f(x)$ for any $f \in \F$.
For any kernel $k : \X \times \X \to \R$, there exists a feature map $\varphi : \X \to \mathcal F$, where $\mathcal F$ is a special Hilbert space called the reproducing kernel Hilbert space (RKHS) corresponding to $k$. This is a space of functions, $f : \X \to \R$. These spaces satisfy a special key condition, called the reproducing property: $\langle f, \varphi(x) \rangle_\F = f(x)$ for any $f \in \F$.
It might help to give slightly more of an overview of MMD.$\DeclareMathOperator{\E}{\mathbb E}\newcommand{\R}{\mathbb R}\newcommand{\X}{\mathcal X}\newcommand{\h}{\mathcal H}\DeclareMathOperator{\MMD}{MMD}$$\DeclareMathOperator{\E}{\mathbb E}\newcommand{\R}{\mathbb R}\newcommand{\X}{\mathcal X}\newcommand{\h}{\mathcal H}\newcommand{\F}{\mathcal F}\DeclareMathOperator{\MMD}{MMD}$
In general, MMD is defined by the idea of representing distances between distributions as distances between mean embeddings of features. That is, say we have distributions $P$ and $Q$ over a set $\X$.
The MMD is defined bybased on a feature map $\varphi : \X \to \h$, where $\mathcal H$$\h$ is what's called a reproducing kernelsome Hilbert spacespace; this corresponds to a kernel (as in SVMs, not KDE) by $k(x, y) = \langle \varphi(x), \varphi(y) \rangle_\h$. In In general, the MMD is $$ \MMD(P, Q) = \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h .$$$$ \MMD(P, Q) = \big\lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \big\rVert_\h .$$
As one example, we might have $\X = \h = \R^d$ and $\varphi(x) = x$, corresponding to a linear kernel. In that case: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ X ] - \E_{Y \sim Q}[ Y ] \rVert_{\R^d} \\&= \lVert \mu_P - \mu_Q \rVert_{\R^d} ,\end{align}\begin{align} \MMD(P, Q) &= \bigl\lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \bigr\rVert_\h \\&= \bigl\lVert \E_{X \sim P}[ X ] - \E_{Y \sim Q}[ Y ] \bigr\rVert_{\R^d} \\&= \bigl\lVert \mu_P - \mu_Q \bigr\rVert_{\R^d} ,\end{align} so this MMD is just the distance between the means of the two distributions. Matching distributions like this will match their means, though they might differ in their variance or in other ways.
Your case is slightly different: we have $\mathcal X = \mathbb R^d$ and $\mathcal H = \mathbb R^p$, with $\varphi(x) = A' x$, where $A$ is a $d \times p$ matrix. So we have \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ A' X ] - \E_{Y \sim Q}[ A' Y ] \rVert_{\R^p} \\&= \lVert A' \E_{X \sim P}[ X ] - A' \E_{Y \sim Q}[ Y ] \rVert_{\R^p} \\&= \lVert A'( \mu_P - \mu_Q ) \rVert_{\R^p} .\end{align}\begin{align} \MMD(P, Q) &= \bigl\lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \bigr\rVert_\h \\&= \bigl\lVert \E_{X \sim P}[ A' X ] - \E_{Y \sim Q}[ A' Y ] \bigr\rVert_{\R^p} \\&= \bigl\lVert A' \E_{X \sim P}[ X ] - A' \E_{Y \sim Q}[ Y ] \bigr\rVert_{\R^p} \\&= \bigl\lVert A'( \mu_P - \mu_Q ) \bigr\rVert_{\R^p} .\end{align} This MMD is the difference between two different projections of the mean. If $p < d$ or the mapping $A'$ otherwise isn't invertible, then this MMD is weaker than the previous one: it doesn't distinguish between some distributions that the previous one does.
You can also construct stronger distances. For example, if $\X = \R$ and you use $\varphi(x) = (x, x^2)$ (giving a particular quadratic kernel), then the MMD becomes $\sqrt{(\E X - \E Y)^2 + (\E X^2 - \E Y^2)^2}$, and can distinguish not only distributions with different means but with different variances as well.
And you can get much stronger than that: if $\varphi$ maps to afor general reproducingchoices of kernel Hilbert space, then you can applyuse the kernel trick to compute the MMD: \begin{align} \MMD^2(P, Q) &= \bigl\lVert \E_{X \sim P} \varphi(X) - \E_{Y \sim Q} \varphi(Y) \bigr\rVert_\h^2 \\&= \langle \E_{X \sim P} \varphi(X), \E_{X' \sim P} \varphi(X') \rangle_\h + \langle \E_{Y \sim Q} \varphi(Y), \E_{Y' \sim Q} \varphi(Y') \rangle_\h - 2 \langle \E_{X \sim P} \varphi(X), \E_{Y \sim Q} \varphi(Y) \rangle_\h \\&= \E_{X, X' \sim P} k(X, X') + \E_{Y, Y' \sim Q} k(Y, Y') - 2 \E_{X \sim P, Y \sim Q} k(X, Y) .\end{align} It's then straightforward to estimate this with samples, and it turns out that many kernelsfor any kernel function $k$ -- even ones where $\varphi$ is infinite-dimensional, includinglike the Gaussian kernel (also called "squared exponential" or "exponentiated quadratic") $k(x, y) = \exp\left( -\frac{1}{2\sigma^2} \lVert x - y \rVert^2 \right)$.
If your choice of $k$ is "characteristic, lead to" then the MMD beingbecomes a proper metric on distributions: it's zero if and only if the two distributions are identicalthe same.
Specifically, letting $k(x, y) = \langle \varphi(x), \varphi(y) \rangle_\h$, you get \begin{align} \MMD^2(P, Q) &= \lVert \E_{X \sim P} \varphi(X) - \E_{Y \sim Q} \varphi(Y) \rVert_\h^2 \\&= \langle \E_{X \sim P} \varphi(X), \E_{X' \sim P} \varphi(X') \rangle_\h + \langle \E_{Y \sim Q} \varphi(Y), \E_{Y' \sim Q} \varphi(Y') \rangle_\h - 2 \langle \E_{X \sim P} \varphi(X), \E_{Y \sim Q} \varphi(Y) \rangle_\h \\&= \E_{X, X' \sim P} k(X, X') + \E_{Y, Y' \sim Q} k(Y, Y') - 2 \E_{X \sim P, Y \sim Q} k(X, Y) \end{align} which(This is unlike when you can straightforwardly estimateuse, say, a linear kernel, where two distributions with samplesthe same mean have zero linear-kernel MMD.) If you've heard of a "universal" kernel, those are characteristic, but there are a few kernels that are characteristic but not universal.
Update: here's where the "maximum" inHere's an explanation of the name comes from, which is also useful for understanding the MMD.
TheFor any kernel $k : \X \times \X \to \R$, there exists a feature map $\varphi: \X \to \h$ maps into$\varphi : \X \to \mathcal F$, where $\mathcal F$ is a reproducing kernelspecial Hilbert space called the reproducing kernel Hilbert space (RKHS) corresponding to $k$. These are spacesThis is a space of functions, and$f : \X \to \R$. These spaces satisfy a special key property (calledcondition, called the reproducing property): $\langle f, \varphi(x) \rangle_\h = f(x)$$\langle f, \varphi(x) \rangle_\F = f(x)$ for any $f \in \h$$f \in \F$.
In theThe simplest example, is the linear kernel $\X = \h = \R^d$$k(x, y) = x \cdot y$. This can be "implemented" with $\varphi(x) = x$, we view each$\h = \R^d$ and $f \in \h$ as$\varphi(x) = x$. But the function corresponding to someRKHS is instead the space of linear functions $w \in \R^d$$f_x(t) = x \cdot t$, byand $f(x) = w' x$$\varphi(x) = f_x$. Then theThe reproducing property is $\langle f, \varphi(x) \rangle_\h = \langle w, x \rangle_{\R^d}$ should make sense$\langle f_w, \varphi(x) \rangle_\h = \langle w, x \rangle_{\R^d}$.
Now, we can give an alternative characterization of the MMD: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rVert_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] \rangle_\h - \langle f, \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[\langle f, \varphi(X)\rangle_\h] - \E_{Y \sim Q}[\langle f, \varphi(Y) \rangle_\h] \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[f(X)] - \E_{Y \sim Q}[f(Y)] .\end{align} The second line is a general fact about norms in Hilbert spaces that follows immediately from Cauchy-Schwarz: $\sup_{f : \lVert f \rVert \le 1} \langle f, g \rangle_\h = \lVert g \rVert$ is achieved by $f = g / \lVert g \rVert$.
The fourth line depends on a technical condition known as Bochner integrability, but is true e.g. for bounded kernels or distributions with bounded support.
Then, at the end we use the reproducing property.
This last line is why it's called the "maximum mean discrepancy" – it's the maximum, over test functions $f$ in the unit ball of $\h$, of the mean difference between the two distributions. This is also a special case of an integral probability metric.
It might help to give slightly more of an overview of MMD.$\DeclareMathOperator{\E}{\mathbb E}\newcommand{\R}{\mathbb R}\newcommand{\X}{\mathcal X}\newcommand{\h}{\mathcal H}\DeclareMathOperator{\MMD}{MMD}$
In general, MMD is defined by the idea of representing distances between distributions as distances between mean embeddings of features. That is, say we have distributions $P$ and $Q$ over a set $\X$. The MMD is defined by a feature map $\varphi : \X \to \h$, where $\mathcal H$ is what's called a reproducing kernel Hilbert space. In general, the MMD is $$ \MMD(P, Q) = \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h .$$
As one example, we might have $\X = \h = \R^d$ and $\varphi(x) = x$. In that case: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ X ] - \E_{Y \sim Q}[ Y ] \rVert_{\R^d} \\&= \lVert \mu_P - \mu_Q \rVert_{\R^d} ,\end{align} so this MMD is just the distance between the means of the two distributions. Matching distributions like this will match their means, though they might differ in their variance or in other ways.
Your case is slightly different: we have $\mathcal X = \mathbb R^d$ and $\mathcal H = \mathbb R^p$, with $\varphi(x) = A' x$, where $A$ is a $d \times p$ matrix. So we have \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ A' X ] - \E_{Y \sim Q}[ A' Y ] \rVert_{\R^p} \\&= \lVert A' \E_{X \sim P}[ X ] - A' \E_{Y \sim Q}[ Y ] \rVert_{\R^p} \\&= \lVert A'( \mu_P - \mu_Q ) \rVert_{\R^p} .\end{align} This MMD is the difference between two different projections of the mean. If $p < d$ or the mapping $A'$ otherwise isn't invertible, then this MMD is weaker than the previous one: it doesn't distinguish between some distributions that the previous one does.
You can also construct stronger distances. For example, if $\X = \R$ and you use $\varphi(x) = (x, x^2)$, then the MMD becomes $\sqrt{(\E X - \E Y)^2 + (\E X^2 - \E Y^2)^2}$, and can distinguish not only distributions with different means but with different variances as well.
And you can get much stronger than that: if $\varphi$ maps to a general reproducing kernel Hilbert space, then you can apply the kernel trick to compute the MMD, and it turns out that many kernels, including the Gaussian kernel, lead to the MMD being zero if and only the distributions are identical.
Specifically, letting $k(x, y) = \langle \varphi(x), \varphi(y) \rangle_\h$, you get \begin{align} \MMD^2(P, Q) &= \lVert \E_{X \sim P} \varphi(X) - \E_{Y \sim Q} \varphi(Y) \rVert_\h^2 \\&= \langle \E_{X \sim P} \varphi(X), \E_{X' \sim P} \varphi(X') \rangle_\h + \langle \E_{Y \sim Q} \varphi(Y), \E_{Y' \sim Q} \varphi(Y') \rangle_\h - 2 \langle \E_{X \sim P} \varphi(X), \E_{Y \sim Q} \varphi(Y) \rangle_\h \\&= \E_{X, X' \sim P} k(X, X') + \E_{Y, Y' \sim Q} k(Y, Y') - 2 \E_{X \sim P, Y \sim Q} k(X, Y) \end{align} which you can straightforwardly estimate with samples.
Update: here's where the "maximum" in the name comes from.
The feature map $\varphi: \X \to \h$ maps into a reproducing kernel Hilbert space. These are spaces of functions, and satisfy a key property (called the reproducing property): $\langle f, \varphi(x) \rangle_\h = f(x)$ for any $f \in \h$.
In the simplest example, $\X = \h = \R^d$ with $\varphi(x) = x$, we view each $f \in \h$ as the function corresponding to some $w \in \R^d$, by $f(x) = w' x$. Then the reproducing property $\langle f, \varphi(x) \rangle_\h = \langle w, x \rangle_{\R^d}$ should make sense.
Now, we can give an alternative characterization of the MMD: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rVert_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] \rangle_\h - \langle f, \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[\langle f, \varphi(X)\rangle_\h] - \E_{Y \sim Q}[\langle f, \varphi(Y) \rangle_\h] \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[f(X)] - \E_{Y \sim Q}[f(Y)] .\end{align} The second line is a general fact about norms in Hilbert spaces: $\sup_{f : \lVert f \rVert \le 1} \langle f, g \rangle_\h = \lVert g \rVert$ is achieved by $f = g / \lVert g \rVert$. The fourth depends on a technical condition known as Bochner integrability but is true e.g. for bounded kernels or distributions with bounded support. Then at the end we use the reproducing property.
This last line is why it's called the "maximum mean discrepancy" – it's the maximum, over test functions $f$ in the unit ball of $\h$, of the mean difference between the two distributions.
It might help to give slightly more of an overview of MMD.$\DeclareMathOperator{\E}{\mathbb E}\newcommand{\R}{\mathbb R}\newcommand{\X}{\mathcal X}\newcommand{\h}{\mathcal H}\newcommand{\F}{\mathcal F}\DeclareMathOperator{\MMD}{MMD}$
In general, MMD is defined by the idea of representing distances between distributions as distances between mean embeddings of features. That is, say we have distributions $P$ and $Q$ over a set $\X$.
The MMD is defined based on a feature map $\varphi : \X \to \h$, where $\h$ is some Hilbert space; this corresponds to a kernel (as in SVMs, not KDE) by $k(x, y) = \langle \varphi(x), \varphi(y) \rangle_\h$. In general, the MMD is $$ \MMD(P, Q) = \big\lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \big\rVert_\h .$$
As one example, we might have $\X = \h = \R^d$ and $\varphi(x) = x$, corresponding to a linear kernel. In that case: \begin{align} \MMD(P, Q) &= \bigl\lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \bigr\rVert_\h \\&= \bigl\lVert \E_{X \sim P}[ X ] - \E_{Y \sim Q}[ Y ] \bigr\rVert_{\R^d} \\&= \bigl\lVert \mu_P - \mu_Q \bigr\rVert_{\R^d} ,\end{align} so this MMD is just the distance between the means of the two distributions. Matching distributions like this will match their means, though they might differ in their variance or in other ways.
Your case is slightly different: we have $\mathcal X = \mathbb R^d$ and $\mathcal H = \mathbb R^p$, with $\varphi(x) = A' x$, where $A$ is a $d \times p$ matrix. So we have \begin{align} \MMD(P, Q) &= \bigl\lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \bigr\rVert_\h \\&= \bigl\lVert \E_{X \sim P}[ A' X ] - \E_{Y \sim Q}[ A' Y ] \bigr\rVert_{\R^p} \\&= \bigl\lVert A' \E_{X \sim P}[ X ] - A' \E_{Y \sim Q}[ Y ] \bigr\rVert_{\R^p} \\&= \bigl\lVert A'( \mu_P - \mu_Q ) \bigr\rVert_{\R^p} .\end{align} This MMD is the difference between two different projections of the mean. If $p < d$ or the mapping $A'$ otherwise isn't invertible, then this MMD is weaker than the previous one: it doesn't distinguish between some distributions that the previous one does.
You can also construct stronger distances. For example, if $\X = \R$ and you use $\varphi(x) = (x, x^2)$ (giving a particular quadratic kernel), then the MMD becomes $\sqrt{(\E X - \E Y)^2 + (\E X^2 - \E Y^2)^2}$, and can distinguish not only distributions with different means but with different variances as well.
And you can get much stronger than that: for general choices of kernel, you can use the kernel trick to compute the MMD: \begin{align} \MMD^2(P, Q) &= \bigl\lVert \E_{X \sim P} \varphi(X) - \E_{Y \sim Q} \varphi(Y) \bigr\rVert_\h^2 \\&= \langle \E_{X \sim P} \varphi(X), \E_{X' \sim P} \varphi(X') \rangle_\h + \langle \E_{Y \sim Q} \varphi(Y), \E_{Y' \sim Q} \varphi(Y') \rangle_\h - 2 \langle \E_{X \sim P} \varphi(X), \E_{Y \sim Q} \varphi(Y) \rangle_\h \\&= \E_{X, X' \sim P} k(X, X') + \E_{Y, Y' \sim Q} k(Y, Y') - 2 \E_{X \sim P, Y \sim Q} k(X, Y) .\end{align} It's then straightforward to estimate this with samples, for any kernel function $k$ -- even ones where $\varphi$ is infinite-dimensional, like the Gaussian kernel (also called "squared exponential" or "exponentiated quadratic") $k(x, y) = \exp\left( -\frac{1}{2\sigma^2} \lVert x - y \rVert^2 \right)$.
If your choice of $k$ is "characteristic," then the MMD becomes a proper metric on distributions: it's zero if and only if the two distributions are the same. (This is unlike when you use, say, a linear kernel, where two distributions with the same mean have zero linear-kernel MMD.) If you've heard of a "universal" kernel, those are characteristic, but there are a few kernels that are characteristic but not universal.
Here's an explanation of the name, which is also useful for understanding the MMD.
For any kernel $k : \X \times \X \to \R$, there exists a feature map $\varphi : \X \to \mathcal F$, where $\mathcal F$ is a special Hilbert space called the reproducing kernel Hilbert space (RKHS) corresponding to $k$. This is a space of functions, $f : \X \to \R$. These spaces satisfy a special key condition, called the reproducing property: $\langle f, \varphi(x) \rangle_\F = f(x)$ for any $f \in \F$.
The simplest example is the linear kernel $k(x, y) = x \cdot y$. This can be "implemented" with $\h = \R^d$ and $\varphi(x) = x$. But the RKHS is instead the space of linear functions $f_x(t) = x \cdot t$, and $\varphi(x) = f_x$. The reproducing property is $\langle f_w, \varphi(x) \rangle_\h = \langle w, x \rangle_{\R^d}$.
Now, we can give an alternative characterization of the MMD: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rVert_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] \rangle_\h - \langle f, \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[\langle f, \varphi(X)\rangle_\h] - \E_{Y \sim Q}[\langle f, \varphi(Y) \rangle_\h] \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[f(X)] - \E_{Y \sim Q}[f(Y)] .\end{align} The second line is a general fact about norms in Hilbert spaces that follows immediately from Cauchy-Schwarz: $\sup_{f : \lVert f \rVert \le 1} \langle f, g \rangle_\h = \lVert g \rVert$ is achieved by $f = g / \lVert g \rVert$.
The fourth line depends on a technical condition known as Bochner integrability, but is true e.g. for bounded kernels or distributions with bounded support.
Then, at the end we use the reproducing property.
This last line is why it's called the "maximum mean discrepancy" – it's the maximum, over test functions $f$ in the unit ball of $\h$, of the mean difference between the two distributions. This is also a special case of an integral probability metric.
It might help to give slightly more of an overview of MMD.$\DeclareMathOperator{\E}{\mathbb E}\newcommand{\R}{\mathbb R}\newcommand{\X}{\mathcal X}\newcommand{\h}{\mathcal H}\DeclareMathOperator{\MMD}{MMD}$
In general, MMD is defined by the idea of representing distances between distributions as distances between mean embeddings of features. That is, say we have distributions $P$ and $Q$ over a set $\X$. The MMD is defined by a feature map $\varphi : \X \to \h$, where $\mathcal H$ is what's called a reproducing kernel Hilbert space. In general, the MMD is $$ \MMD(P, Q) = \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h .$$
As one example, we might have $\X = \h = \R^d$ and $\varphi(x) = x$. In that case: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ X ] - \E_{Y \sim Q}[ Y ] \rVert_{\R^d} \\&= \lVert \mu_P - \mu_Q \rVert_{\R^d} ,\end{align} so this MMD is just the distance between the means of the two distributions. Matching distributions like this will match their means, though they might differ in their variance or in other ways.
Your case is slightly different: we have $\mathcal X = \mathbb R^d$ and $\mathcal H = \mathbb R^p$, with $\varphi(x) = A' x$, where $A$ is a $d \times p$ matrix. So we have \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ A' X ] - \E_{Y \sim Q}[ A' Y ] \rVert_{\R^p} \\&= \lVert A' \E_{X \sim P}[ X ] - A' \E_{Y \sim Q}[ Y ] \rVert_{\R^p} \\&= \lVert A'( \mu_P - \mu_Q ) \rVert_{\R^p} .\end{align} This MMD is the difference between two different projections of the mean. If $p < d$ or the mapping $A'$ otherwise isn't invertible, then this MMD is weaker than the previous one: it doesn't distinguish between some distributions that the previous one does.
You can also construct stronger distances. For example, if $\X = \R$ and you use $\varphi(x) = (x, x^2)$, then the MMD becomes $\sqrt{(\E X - \E Y)^2 + (\E X^2 - \E Y^2)^2}$, and can distinguish not only distributions with different means but with different variances as well.
And you can get much stronger than that: if $\varphi$ maps to a general reproducing kernel Hilbert space, then you can apply the kernel trick to compute the MMD, and it turns out that many kernels, including the Gaussian kernel, lead to the MMD being zero if and only the distributions are identical.
Specifically, letting $k(x, y) = \langle \varphi(x), \varphi(y) \rangle_\h$, you get \begin{align} \MMD^2(P, Q) &= \lVert \E_{X \sim P} \varphi(X) - \E_{Y \sim Q} \varphi(Y) \rVert_\h^2 \\&= \langle \E_{X \sim P} \varphi(X), \E_{X' \sim P} \varphi(X') \rangle_\h + \langle \E_{Y \sim Q} \varphi(Y), \E_{Y' \sim Q} \varphi(Y') \rangle_\h - 2 \langle \E_{X \sim P} \varphi(X), \E_{Y \sim Q} \varphi(Y) \rangle_\h \\&= \E_{X, X' \sim P} k(X, X') + \E_{Y, Y' \sim Q} k(Y, Y') - 2 \E_{X \sim P, Y \sim Q} k(X, Y) \end{align} which you can straightforwardly estimate with samples.
Update: here's where the "maximum" in the name comes from.
The feature map $\varphi: \X \to \h$ maps into a reproducing kernel Hilbert space. These are spaces of functions, and satisfy a key property (called the reproducing property): $\langle f, \varphi(x) \rangle_\h = f(x)$ for any $f \in \h$.
In the simplest example, $\X = \h = \R^d$ with $\varphi(x) = x$, we view each $f \in \h$ as the function corresponding to some $w \in \R^d$, by $f(x) = w' x$. Then the reproducing property $\langle f, \varphi(x) \rangle_\h = \langle w, x \rangle_{\R^d}$ should make sense.
In more complex settings, like a Gaussian kernel, $f$ is a much more complicated function, but the reproducing property still holds.
Now, we can give an alternative characterization of the MMD: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rVert_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] \rangle_\h - \langle f, \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[\langle f, \varphi(X)\rangle_\h] - \E_{Y \sim Q}[\langle f, \varphi(Y) \rangle_\h] \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[f(X)] - \E_{Y \sim Q}[f(Y)] .\end{align} The second line is a general fact about norms in Hilbert spaces: $\sup_{f : \lVert f \rVert \le 1} \langle f, g \rangle_\h = \lVert g \rVert$ is achieved by $f = g / \lVert g \rVert$. The fourth depends on a technical condition known as Bochner integrability but is true e.g. for bounded kernels or distributions with bounded support. Then at the end we use the reproducing property.
This last line is why it's called the "maximum mean discrepancy" – it's the maximum, over test functions $f$ in the unit ball of $\h$, of the mean difference between the two distributions.
It might help to give slightly more of an overview of MMD.$\DeclareMathOperator{\E}{\mathbb E}\newcommand{\R}{\mathbb R}\newcommand{\X}{\mathcal X}\newcommand{\h}{\mathcal H}\DeclareMathOperator{\MMD}{MMD}$
In general, MMD is defined by the idea of representing distances between distributions as distances between mean embeddings of features. That is, say we have distributions $P$ and $Q$ over a set $\X$. The MMD is defined by a feature map $\varphi : \X \to \h$, where $\mathcal H$ is what's called a reproducing kernel Hilbert space. In general, the MMD is $$ \MMD(P, Q) = \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h .$$
As one example, we might have $\X = \h = \R^d$ and $\varphi(x) = x$. In that case: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ X ] - \E_{Y \sim Q}[ Y ] \rVert_{\R^d} \\&= \lVert \mu_P - \mu_Q \rVert_{\R^d} ,\end{align} so this MMD is just the distance between the means of the two distributions. Matching distributions like this will match their means, though they might differ in their variance or in other ways.
Your case is slightly different: we have $\mathcal X = \mathbb R^d$ and $\mathcal H = \mathbb R^p$, with $\varphi(x) = A' x$, where $A$ is a $d \times p$ matrix. So we have \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ A' X ] - \E_{Y \sim Q}[ A' Y ] \rVert_{\R^p} \\&= \lVert A' \E_{X \sim P}[ X ] - A' \E_{Y \sim Q}[ Y ] \rVert_{\R^p} \\&= \lVert A'( \mu_P - \mu_Q ) \rVert_{\R^p} .\end{align} This MMD is the difference between two different projections of the mean. If $p < d$ or the mapping $A'$ otherwise isn't invertible, then this MMD is weaker than the previous one: it doesn't distinguish between some distributions that the previous one does.
You can also construct stronger distances. For example, if $\X = \R$ and you use $\varphi(x) = (x, x^2)$, then the MMD becomes $\sqrt{(\E X - \E Y)^2 + (\E X^2 - \E Y^2)^2}$, and can distinguish not only distributions with different means but with different variances as well.
And you can get much stronger than that: if $\varphi$ maps to a general reproducing kernel Hilbert space, then you can apply the kernel trick to compute the MMD, and it turns out that many kernels, including the Gaussian kernel, lead to the MMD being zero if and only the distributions are identical.
Update: here's where the "maximum" in the name comes from.
The feature map $\varphi: \X \to \h$ maps into a reproducing kernel Hilbert space. These are spaces of functions, and satisfy a key property (called the reproducing property): $\langle f, \varphi(x) \rangle_\h = f(x)$ for any $f \in \h$.
In the simplest example, $\X = \h = \R^d$ with $\varphi(x) = x$, we view each $f \in \h$ as the function corresponding to some $w \in \R^d$, by $f(x) = w' x$. Then the reproducing property $\langle f, \varphi(x) \rangle_\h = \langle w, x \rangle_{\R^d}$ should make sense.
In more complex settings, like a Gaussian kernel, $f$ is a much more complicated function, but the reproducing property still holds.
Now, we can give an alternative characterization of the MMD: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rVert_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] \rangle_\h - \langle f, \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[\langle f, \varphi(X)\rangle_\h] - \E_{Y \sim Q}[\langle f, \varphi(Y) \rangle_\h] \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[f(X)] - \E_{Y \sim Q}[f(Y)] .\end{align} The second line is a general fact about norms in Hilbert spaces: $\sup_{f : \lVert f \rVert \le 1} \langle f, g \rangle_\h = \lVert g \rVert$ is achieved by $f = g / \lVert g \rVert$. The fourth depends on a technical condition known as Bochner integrability but is true e.g. for bounded kernels or distributions with bounded support. Then at the end we use the reproducing property.
This last line is why it's called the "maximum mean discrepancy" – it's the maximum, over test functions $f$ in the unit ball of $\h$, of the mean difference between the two distributions.
It might help to give slightly more of an overview of MMD.$\DeclareMathOperator{\E}{\mathbb E}\newcommand{\R}{\mathbb R}\newcommand{\X}{\mathcal X}\newcommand{\h}{\mathcal H}\DeclareMathOperator{\MMD}{MMD}$
In general, MMD is defined by the idea of representing distances between distributions as distances between mean embeddings of features. That is, say we have distributions $P$ and $Q$ over a set $\X$. The MMD is defined by a feature map $\varphi : \X \to \h$, where $\mathcal H$ is what's called a reproducing kernel Hilbert space. In general, the MMD is $$ \MMD(P, Q) = \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h .$$
As one example, we might have $\X = \h = \R^d$ and $\varphi(x) = x$. In that case: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ X ] - \E_{Y \sim Q}[ Y ] \rVert_{\R^d} \\&= \lVert \mu_P - \mu_Q \rVert_{\R^d} ,\end{align} so this MMD is just the distance between the means of the two distributions. Matching distributions like this will match their means, though they might differ in their variance or in other ways.
Your case is slightly different: we have $\mathcal X = \mathbb R^d$ and $\mathcal H = \mathbb R^p$, with $\varphi(x) = A' x$, where $A$ is a $d \times p$ matrix. So we have \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[ \varphi(X) ] - \E_{Y \sim Q}[ \varphi(Y) ] \rVert_\h \\&= \lVert \E_{X \sim P}[ A' X ] - \E_{Y \sim Q}[ A' Y ] \rVert_{\R^p} \\&= \lVert A' \E_{X \sim P}[ X ] - A' \E_{Y \sim Q}[ Y ] \rVert_{\R^p} \\&= \lVert A'( \mu_P - \mu_Q ) \rVert_{\R^p} .\end{align} This MMD is the difference between two different projections of the mean. If $p < d$ or the mapping $A'$ otherwise isn't invertible, then this MMD is weaker than the previous one: it doesn't distinguish between some distributions that the previous one does.
You can also construct stronger distances. For example, if $\X = \R$ and you use $\varphi(x) = (x, x^2)$, then the MMD becomes $\sqrt{(\E X - \E Y)^2 + (\E X^2 - \E Y^2)^2}$, and can distinguish not only distributions with different means but with different variances as well.
And you can get much stronger than that: if $\varphi$ maps to a general reproducing kernel Hilbert space, then you can apply the kernel trick to compute the MMD, and it turns out that many kernels, including the Gaussian kernel, lead to the MMD being zero if and only the distributions are identical.
Specifically, letting $k(x, y) = \langle \varphi(x), \varphi(y) \rangle_\h$, you get \begin{align} \MMD^2(P, Q) &= \lVert \E_{X \sim P} \varphi(X) - \E_{Y \sim Q} \varphi(Y) \rVert_\h^2 \\&= \langle \E_{X \sim P} \varphi(X), \E_{X' \sim P} \varphi(X') \rangle_\h + \langle \E_{Y \sim Q} \varphi(Y), \E_{Y' \sim Q} \varphi(Y') \rangle_\h - 2 \langle \E_{X \sim P} \varphi(X), \E_{Y \sim Q} \varphi(Y) \rangle_\h \\&= \E_{X, X' \sim P} k(X, X') + \E_{Y, Y' \sim Q} k(Y, Y') - 2 \E_{X \sim P, Y \sim Q} k(X, Y) \end{align} which you can straightforwardly estimate with samples.
Update: here's where the "maximum" in the name comes from.
The feature map $\varphi: \X \to \h$ maps into a reproducing kernel Hilbert space. These are spaces of functions, and satisfy a key property (called the reproducing property): $\langle f, \varphi(x) \rangle_\h = f(x)$ for any $f \in \h$.
In the simplest example, $\X = \h = \R^d$ with $\varphi(x) = x$, we view each $f \in \h$ as the function corresponding to some $w \in \R^d$, by $f(x) = w' x$. Then the reproducing property $\langle f, \varphi(x) \rangle_\h = \langle w, x \rangle_{\R^d}$ should make sense.
In more complex settings, like a Gaussian kernel, $f$ is a much more complicated function, but the reproducing property still holds.
Now, we can give an alternative characterization of the MMD: \begin{align} \MMD(P, Q) &= \lVert \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rVert_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] - \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim P}[\varphi(X)] \rangle_\h - \langle f, \E_{Y \sim Q}[\varphi(Y)] \rangle_\h \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[\langle f, \varphi(X)\rangle_\h] - \E_{Y \sim Q}[\langle f, \varphi(Y) \rangle_\h] \\&= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim P}[f(X)] - \E_{Y \sim Q}[f(Y)] .\end{align} The second line is a general fact about norms in Hilbert spaces: $\sup_{f : \lVert f \rVert \le 1} \langle f, g \rangle_\h = \lVert g \rVert$ is achieved by $f = g / \lVert g \rVert$. The fourth depends on a technical condition known as Bochner integrability but is true e.g. for bounded kernels or distributions with bounded support. Then at the end we use the reproducing property.
This last line is why it's called the "maximum mean discrepancy" – it's the maximum, over test functions $f$ in the unit ball of $\h$, of the mean difference between the two distributions.