Modelling parameters in maximum likelihood

Question

Often times I'll write down an idea and have no idea if (1) It's a good idea, and (2) If it is a good idea, how much has it been applied / studied. I'm asking this question in hopes to get insight on these two questions. Perhaps by providing search terms or references (or by just pointing me to something better!).

Say I have a training data $\{x_i,y_i\}_{i=1,..n}$ where $x_i \in R^d$ and $y_i$ are scalars. I want to do a regression to predict $y$ given the vector $x$, however it's also important that I also give a measure of confidence in my predictions. Ideally, I'd like a whole posterior distribution $p(y|x)$. I've tried quantile regression and nearest neighbor techniques, but I wanted something a bit more parametric.

I'd like to model my outcome using some parametric model $p(y;\theta)$, where $p$ is some known PDF with parameters $\theta$. Maximum likelihood can then be applied to give the best estimate of $\theta$ given my training data by minimizing the negative log likelihood $L(\theta) = \sum_i -\log p(y_i;\theta)$

However, what if I want my model to have some kind of heteroscedasticity, where $\theta = \theta(x)$ - meaning I want my parameters to vary with my input $x$ giving a "local" conditional distribution.

What are some ways to do this? My attempt was pick a basis $\{\phi_j(x)\}$ (perhaps a linear or quadratic basis) and model my parameters as $\theta_i(x) = \sum_j c^i_j \phi_j(x)$ where $i$ ranges over the number of parameters in $p$. The coefficients in this model are then fit by optimizing $L(\theta(x))$ using standard numerical optimization techniques in MATLAB, scipy or R.

Does this type of conditional maximum likelihood estimation have a name? I've googled variants and I'm having trouble finding something similar to this method. Most references I find, $\theta$ is constant and isn't allowed to vary with $x$. Any references/search terms you might have are appreciated.

Answer 1

Your question seems to have a mix of concepts, so I'm a little unclear on what specifically your question is. I hope my answer can get close to the mark.

Jim is correct that if your approach is to:

1. come up with a probability model for the response data, $y$, given some parameters $\theta$ and the predictors $x$ ( $Pr(y|\theta, x)$ aka the likelihood) and

2. find the maximum of the likelihood function for $\theta$ given the data,

then this technique is still called maximum likelihood, no matter what your probability model is.

Alternatively, you could:

1. come up with a probability model for the data given some parameters $\theta$, $Pr(y|\theta, x)$, and
2. come up with a reasonable prior probability for $\theta$, $Pr(\theta)$, and
3. solve for the probability of the parameters given the data and your prior, $Pr(\theta|y, x)$ (aka the posterior), perhaps using MCMC methods

then the technique you would be using would be called Bayesian inference, no matter what you probability model is.

If you are asking specifically about what to call the methods wherein you fit a regression model that includes heteroskedascity that is perhaps related to your $x$ values, well, there are different names for this depending on your underlying probability model and your domain specific jargon. For example, one probability model that has built in heteroskedascity is the Poisson model. The Poisson distribution has only one parameter, $\lambda$ that describes both the mean and the variance. Fitting a regression model where $y$ can be described by a Poisson distribution (integer counts) is called Poisson regression, and the heteroskedascity comes "baked in". Poisson regression is in the class of models described as generalized linear models. You could also have a probability model that is more familiar to you if you're coming from the world of least squares, where $x$ describes the mean of $y$ with Normally distributed errors, but you relax the homoskedastic assumption on using generalized least squares, where once again we are in the class of models described as generalized linear models.

Basically, if you can write down the probability model for your data, there's a way to fit it (or attempt to fit it). Changing assumptions about the error structure may put you in some other class of models, and so you might have to figure out how to describe it in terms that are familiar to your audience. But in general if you are maximizing the likelihood for that probability model, it's still called maximum likelihood, and if you are coming up with prior and solving for the posterior it's called Bayesian inference. People tend to align more strongly with one camp or another, so, depending on your audience, you may want to avoid mixing terms such as maximizing the likelihood and posterior distribution. Just a heads up, conditional likelihood is something different from joint or marginal likelihood as well, but that is only a term you have used and it wasn't clear that it was pertinent to your question, so I haven't addressed it.

Answer 2

I'd say it's still called maximum likelihood estimation. You just have a slightly more complex likelihood function to maximize.

For example in a regression model instead of the parameter $\sigma^2$ representing the variance of an observation you might want that variance to be proportional to one of the predictor variables: $x_2\times\sigma^2$. Just incorporate that into the likelihood function.