Recently I have become aware of 'likelihood-free' methods being bandied about in literature. However I am not clear on what it means for an inference or optimization method to be likelihood-free.
In machine learning the goal is usually to maximise the likelihood of some parameters to fit a function e.g. the weights on a neural network.
So what exactly is the philosophy of a likelihood-free approach and why do adversarial networks such as GANs fall under this category?
There are many examples of methods not based on likelihoods in statistics (I don't know about machine learning). Some examples:
Fisher's pure significance tests. Based only on a sharply defined null hypothesis (such as no difference between milk first and milk last in the Lady Tasting Tea experiment. This assumption leads to a null hypothesis distribution, and then a p-value. No likelihood involved. This minimal inferential machinery cannot in itself give a basis for power analysis (no formally defined alternative) or confidence intervals (no formally defined parameter).
Associated to 1. is randomization tests Difference between Randomization test and Permutation test, which in its most basic form is a pure significance test.
Bootstrapping is done without the need for a likelihood function. But there are connections to likelihood ideas, for instance empirical likelihood.
Rank-based methods don't usually use likelihood.
Much of robust statistics.
Confidence intervals for the median (or other quantiles) can be based on order statistics. No likelihood is involved in the calculations. Confidence interval for the median, Best estimator for the variance of the empirical median
V Vapnik had the idea of transductive learning which seems to be related to https://en.wikipedia.org/wiki/Epilogism as discussed in the Black Swan Taleb and the Black Swan.
In the book Data Analysis and Approximate Models Laurie Davis builds a systematic theory of statistical models as approximations, confidence intervals got replaced by approximation intervals, and there are no parametric families of distributions, no $\text{N}(\mu, \sigma^2)$ only $\text{N}(9.37, 2.12^2)$ and so on. And no likelihoods.
At the moment you got a likelihood function, there is an immense machinery to build on. Bayesians cannot do without, and most others do use likelihood most of the time. But it is pointed out in a comment that even Bayesians try to do without, see Approximate_Bayesian_computation. There is even a new text on that topic.
But where do they come from? To get a likelihood function in the usual way, we need a lot of assumptions which can be difficult to justify.
It is interesting to ask if we can construct likelihood functions, in some way, from some of this likelihood-free methods. For instance, point 6. above, can we construct a likelihood function for the median from (a family of) confidence intervals calculated from order statistics? I should ask that as a separate question ...
Your last question about GAN's I must leave for others.
Specifically, [the recent] likelihood-free methods are a rewording of the ABC algorithms, where ABC stands for approximate Bayesian computation. This intends to cover inference methods that do not require the use of a closed-form likelihood function, but still intend to study a specific statistical model. They are free from the computational difficulty attached with the likelihood but not from the model that produces this likelihood. See for instance
On the machine learning side: In machine learning, you usually try to maximize $p(y|x)$, where $x$ is the target, and $y$ is the input (for example, x could be some random noise, and y would be an image). Now, how do we optimize this? A common way to do it, is to assume, that $p(y|x) = N(y|\mu(x), \sigma)$. If we assume this, it leads to the mean squared error. Note, we assumed as form for $p(y|x)$. However, if we dont assume any certain distribution, it is called likelihood-free learning.
Why do GANs fall under this? Well, the Loss function is a neural network, and this neural network is not fixed, but learned jointly. Therefore, we dont assume any form anymore (except, that $p(y|x)$ falls in the family of distributions, that can be represented by the discriminator, but for theory sake we say it is a universal function approximator anyway).
To add to the litany of answers, asymptotic statistics are in fact free of likelihoods.
A "likelihood" here refers to the probability model for the data. I may not care about that. But I may find some simple estimator, like the mean, that is an adequate summary of the data and I want to perform inference about the mean of the distribution (assuming it exists, which is often a reasonable assumption).
By the central limit theorem, the mean has an approximating normal distribution in large N when the variance also exists. I can create consistent tests (power goes to 1 as N goes to infinity when null is false) that are of the correct size. While I have a probability model (that is false) for the sampling distribution of the mean in finite sample sizes, I can obtain valid inference and unbiased estimation to augment my "useful summary of the data" (the mean).
It should be noted that tests based on the 95% CI for the median (i.e. option 6 in @kjetilbhalvorsen's answer) also rely on the central limit theorem to show that they are consistent. So it is not crazy to consider the simple T-test as a "non-parametric" or "non-likelihood based" test.
