Likelihood analysis for exponential distribution

Question

Assume a collection of independent exponential random variables $y_{1}, \ldots, y_{n}$ with means $\mu_{1}, \ldots, \mu_{n}$; where $\mu_{i} = \beta_{0}+\beta_{1}x_{i}$.

How can I find the profile likelihood of $\beta_{1}$?

From what I gather, the profile likelihood is a technique for eliminating so-called "nuisance parameters" from a given inference. This is typically achieved by maximising the likelihood with respect to the nuisance parameters whilst holding the parameters of interest fixed.

For the two-parameter model described above, because we are only interested in $\beta_{1}$, we can consider $\beta_{0}$ as a nuisance parameter.

ATTEMPT:

Firstly, given that $\mu_{1}, \ldots, \mu_{n}$ are the means for $y_{1}, \ldots, y_{n}$ exponential variables, and, in general notation, the mean of the exponential distribution is given by $\lambda^{-1}$, then each random variable must follow an $\text{Exp}((\lambda^{-1})^{-1}) \implies \text{Exp}(\mu^{-1})$; in other words:

For $i=1, \ldots, n$:

$$ \mu_{i}^{-1}\text{exp}\left\{-\mu_{i}^{-1}y_{i}\right\} $$

Now, to determine the form of the likelihood, we form $p(x_{1}) \times p(x_{2}) \times \ldots \times p(x_{n})$, hence,

$$ L(\mu) = \prod_{i=1}^{n} \left[\mu_{i}^{-1}\text{exp}\left\{-\mu_{i}^{-1}y_{i}\right\}\right] $$

Simplifying we get,

$$ = \left[\prod_{i=1}^{n}\mu_{i}^{-1}\right]\text{exp}\left\{-\sum_{i=1}^{n}\frac{y_{i}}{\mu_{i}}\right\} $$

Because $\text{log}\left(\prod_{i=1}^{n}\mu_{i}^{-1}\right) = \sum_{i=1}^{n}\text{log}\left(\mu_{i}^{-1}\right)$, we get:

$$ \text{log}(L(\theta)) = \sum_{i=1}^{n}\text{log}\left(\mu_{i}^{-1}\right)-\sum_{i=1}^{n}\frac{y_{i}}{\mu_{i}} $$

Now, subbing in for $\mu_{i} = \beta_{0}+\beta_{1}x_{i}$, gives:

$$ = \sum_{i=1}^{n}\text{log}(\beta_{0}+\beta_{1}x_{i})^{-1}-\sum_{i=1}^{n}\frac{y_{i}}{\beta_{0}+\beta_{1}x_{i}} $$

Now that we have the log-likelihood, can we just find the MLE for $\beta_{0}$, substitute the acquired MLE for $\beta_{0}$ back into the log-likelihood and simplify? Will this achieve the profile likelihood for $\beta_{1}$?

In other words, where $\hat{\beta_{0}}$ is the MLE for $\beta_{0}$, will the profile likelihood be given by the following?

$$ L_{p}(\beta_{1}) = \sum_{i=1}^{n}\text{log}(\hat{\beta}_{0}+\beta_{1}x_{i})^{-1}-\sum_{i=1}^{n}\frac{y_{i}}{\hat{\beta}_{0}+\beta_{1}x_{i}} $$

Note the "hat" notation for $\beta_{0}$in the above.

Answer 1

Inserting the 1st order condition for the nuisance parameter into the the likelihood is just part of the joint estimation procedure. It does not give you a "Profile" (or "concentrated") likelihood. Profile likelihoods can be obtained only when the estimator of the nuisance parameter does not contain the parameter of interest, or if one accepts to use an estimate for the nuisance parameter obtained through other means. Otherwise, one will have to estimate jointly, as is your case, since the 1st-order condition for $\beta_1$ contains $\beta_0$. By the way, in this case I don't see any reason not to estimate them jointly, but let's say you would want that for some reason.

You could obtain an estimate for $\beta_0$ through the Method of Moments, $\tilde \beta_0$, since you have

$$E(y_i) = \beta_0 + \beta_1 x_i,\;\; i=1,...n$$ $$\operatorname{Var}(y_i) = \Big(\beta_0 + \beta_1 x_i\Big)^2,\;\; i=1,...n$$

Note that, under the Law of Large Numbers,

$$\bar y = \frac 1n \sum_{i=1}^ny_i \approx \frac 1n \sum_{i=1}^nE(y_i) = \beta_0 + \beta_1\bar x\Rightarrow \beta_1 = \frac {\bar y -\beta_0}{\bar x},\; \qquad [1]$$

where the bar indicates the sample mean.

Define now the random variables $$Z_i = \Big(Y_i - E(Y_i)\Big)^2,\;\; i=1,...n$$ which under LLN gives that

$$\frac 1n \sum_{i=1}^nz_i \approx \frac 1n \sum_{i=1}^nE(z_i) = \frac 1n\sum_{i=1}^n\operatorname{Var}(y_i) = \frac 1n\sum_{i=1}^n\Big(\beta_0 + \beta_1 x_i\Big)^2$$ and so

$$\frac 1n \sum_{i=1}^n\Big(y_i - (\beta_0 + \beta_1 x_i)\Big)^2 \approx \frac 1n\sum_{i=1}^n\Big(\beta_0 + \beta_1 x_i\Big)^2$$ $$\Rightarrow \sum_{i=1}^ny_i^2 =2 \sum_{i=1}^n\Big(\beta_0 + \beta_1 x_i\Big)y_i, \qquad [2]$$ Inserting eq.$[1]$ into eq. $[2]$ we can obtain a MoM estimate for $\beta_0$, $\tilde \beta_0$, insert this into the likelihood, and then calculate the MLE for $\beta_1$ conditional on the "outside" estimate for $\beta_0$.