2
$\begingroup$

Often times I'll write down an idea and have no idea if (1) It's a good idea, and (2) If it is a good idea, how much has it been applied / studied. I'm asking this question in hopes to get insight on these two questions. Perhaps by providing search terms or references (or by just pointing me to something better!).

Say I have a training data $\{x_i,y_i\}_{i=1,..n}$ where $x_i \in R^d$ and $y_i$ are scalars. I want to do a regression to predict $y$ given the vector $x$, however it's also important that I also give a measure of confidence in my predictions. Ideally, I'd like a whole posterior distribution $p(y|x)$. I've tried quantile regression and nearest neighbor techniques, but I wanted something a bit more parametric.

I'd like to model my outcome using some parametric model $p(y;\theta)$, where $p$ is some known PDF with parameters $\theta$. Maximum likelihood can then be applied to give the best estimate of $\theta$ given my training data by minimizing the negative log likelihood $L(\theta) = \sum_i -\log p(y_i;\theta)$

However, what if I want my model to have some kind of heteroscedasticity, where $\theta = \theta(x)$ - meaning I want my parameters to vary with my input $x$ giving a "local" conditional distribution.

What are some ways to do this? My attempt was pick a basis $\{\phi_j(x)\}$ (perhaps a linear or quadratic basis) and model my parameters as $\theta_i(x) = \sum_j c^i_j \phi_j(x)$ where $i$ ranges over the number of parameters in $p$. The coefficients in this model are then fit by optimizing $L(\theta(x))$ using standard numerical optimization techniques in MATLAB, scipy or R.

Does this type of conditional maximum likelihood estimation have a name? I've googled variants and I'm having trouble finding something similar to this method. Most references I find, $\theta$ is constant and isn't allowed to vary with $x$. Any references/search terms you might have are appreciated.

$\endgroup$
3
+50
$\begingroup$

Your question seems to have a mix of concepts, so I'm a little unclear on what specifically your question is. I hope my answer can get close to the mark.

Jim is correct that if your approach is to:

  1. come up with a probability model for the response data, $y$, given some parameters $\theta$ and the predictors $x$ ( $Pr(y|\theta, x)$ aka the likelihood) and

  2. find the maximum of the likelihood function for $\theta$ given the data,

then this technique is still called maximum likelihood, no matter what your probability model is.

Alternatively, you could:

  1. come up with a probability model for the data given some parameters $\theta$, $Pr(y|\theta, x)$, and
  2. come up with a reasonable prior probability for $\theta$, $Pr(\theta)$, and
  3. solve for the probability of the parameters given the data and your prior, $Pr(\theta|y, x)$ (aka the posterior), perhaps using MCMC methods

then the technique you would be using would be called Bayesian inference, no matter what you probability model is.

If you are asking specifically about what to call the methods wherein you fit a regression model that includes heteroskedascity that is perhaps related to your $x$ values, well, there are different names for this depending on your underlying probability model and your domain specific jargon. For example, one probability model that has built in heteroskedascity is the Poisson model. The Poisson distribution has only one parameter, $\lambda$ that describes both the mean and the variance. Fitting a regression model where $y$ can be described by a Poisson distribution (integer counts) is called Poisson regression, and the heteroskedascity comes "baked in". Poisson regression is in the class of models described as generalized linear models. You could also have a probability model that is more familiar to you if you're coming from the world of least squares, where $x$ describes the mean of $y$ with Normally distributed errors, but you relax the homoskedastic assumption on using generalized least squares, where once again we are in the class of models described as generalized linear models.

Basically, if you can write down the probability model for your data, there's a way to fit it (or attempt to fit it). Changing assumptions about the error structure may put you in some other class of models, and so you might have to figure out how to describe it in terms that are familiar to your audience. But in general if you are maximizing the likelihood for that probability model, it's still called maximum likelihood, and if you are coming up with prior and solving for the posterior it's called Bayesian inference. People tend to align more strongly with one camp or another, so, depending on your audience, you may want to avoid mixing terms such as maximizing the likelihood and posterior distribution. Just a heads up, conditional likelihood is something different from joint or marginal likelihood as well, but that is only a term you have used and it wasn't clear that it was pertinent to your question, so I haven't addressed it.

$\endgroup$
  • $\begingroup$ Thanks for the comments, I guess I'm doing is maximum likelihood with a more parametrized log-likelihood. I fail to see how I can use MCMC (at least in my problem). If I solve for the posterior, I get p(theta | x,y) ~ p(y | theta,x ) p(theta), where x,y are fixed. If p is fixed (say, gamma or normal or Possion), and depends on theta & x, then I still want to find a theta that maximizes my probability given the data - except now the prior on theta has an effect in the objective. So from this perspective it's like maximum likelihood with a regularization term. $\endgroup$ – MarkV Jul 20 '15 at 20:20
  • $\begingroup$ No, Bayesian inference is NOT like maximum likelihood with a regularization term. Mechanically the two approaches may share some similarities, but they are quite different in theory. But there are some problems that you cannot solve using maximum likelihood (perhaps because you have to work with the marginal rather than conditional likelihood). If you think that that data are fixed and the parameters are random, then you are a Bayesian and you should use MCMC. If you think that the data are random and the parameters are fixed then you are frequentist and should use maximum likelihood. $\endgroup$ – Dalton Hance Jul 21 '15 at 0:03
  • $\begingroup$ The issue with your question is that you have asked in such general terms that it is difficult to come up with answer. What is your specific probability model? There is an incredible wealth of models and specific model structures that all have different names depending on what your probability model is and what your variance structure and sampling/experimental design structure look like. $\endgroup$ – Dalton Hance Jul 21 '15 at 0:08
  • $\begingroup$ I apologize for being vague. Honestly when I wrote it I thought the approach would be well known, but maybe I overestimated how clear the idea was. I did stumble upon Mixed Density Networks which uses a mixture of gaussians with weights, means and variances that depend upon the input x. I wondered if there's a more general family of methods like this, but I guess maximum likelihood covers it. $\endgroup$ – MarkV Jul 22 '15 at 0:07
  • $\begingroup$ And (going back two comments) I don't think it's fair to say that they are not like each other. Obviously the theory is different. But if I want a MAP estimate of a parameter, I'm pretty sure the Bayesian approach looks just like ML with regularization. Doesn't it say that right here or am I misunderstanding? en.wikipedia.org/wiki/Maximum_a_posteriori_estimation $\endgroup$ – MarkV Jul 22 '15 at 0:11
1
$\begingroup$

I'd say it's still called maximum likelihood estimation. You just have a slightly more complex likelihood function to maximize.

For example in a regression model instead of the parameter $\sigma^2$ representing the variance of an observation you might want that variance to be proportional to one of the predictor variables: $x_2\times\sigma^2$. Just incorporate that into the likelihood function.

$\endgroup$
  • $\begingroup$ Thanks for the response. I'll leave the bounty open for a few days in case someone can provide at least some references of this technique being applied. It's a simple enough idea I guess, but still I wanted to see how others may have used it. $\endgroup$ – MarkV Jul 16 '15 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.