6
$\begingroup$

Suppose we observe values from the density function:

$$f_Y(y|\theta)=\theta(1-\theta)^{y-1} \quad \quad \quad y=1,2,...,$$

where $\theta=\beta x/(1+ \beta x)$ and the parameter $\beta >0$ is unknown. Given $(x_i,Y_i), i=1,...,n$, show that the asymptotic variance of $\hat{\beta}$ is:

$$\mathbb{V}(\hat{\beta})=\frac{\beta^2}{\sum^n_{i=1}(1+\beta x_i)^{-1}}.$$


My working: We know that the MLE is:

$$\hat{\beta}=\frac{n}{\sum^n_{i=1}x_iY_i(1+\beta x_i)^{-1}}.$$

So, I found $\hat{\beta}$ with the following working:

$$f_y(y|\beta,\mathbf{x})=\frac{\beta x}{1+\beta x}(\frac{1}{1+\beta x})^{y-1} \implies l(\beta)=n[\log(\beta x)-\log(1+\beta x)]-(\sum (y_i) -n)\log(1+\beta x)$$

$$\implies \frac{\partial l(\beta)}{\partial \beta}=\frac{n}{\beta}-\frac{nx}{1+\beta x}-\frac{x(\sum y_i -n)}{1+\beta x}=0 \implies \hat{\beta}=\frac{n}{x(\sum y_i-n)}.$$

From here, I am not getting the same asymptotic variance as stated in the problem. I expect that my calculation of $\hat{\beta}$ is wrong. Where am I messing up?


Updated working: Since we have that $\sqrt{n}(\hat{\beta}-\beta)\to N(0,\frac{1}{i(\beta)})$, the asymptotic variance is:

$$i(\beta)^{-1}= \Bigg( -nE[\frac{\partial^2 l(\beta|y)}{\partial \beta^2}] \Bigg)^{-1}.$$

$$\frac{\partial^2 l(\beta|y)}{\partial \beta^2}=-\frac{1}{\beta^2}+\frac{yx^2}{(1+\beta x)^2} \implies E[\frac{\partial^2 l(\beta|y)}{\partial \beta^2}]=-\frac{1}{\beta^2}+\frac{x^2}{(1+\beta x)^2}\cdot E(Y),$$

where $\mathbb{E}(Y)=\frac{1+\beta x}{\beta x}$, since $f(y)\sim \text{Geometric}(\theta)$. Thus,

$$\mathbb{V}(\hat{\beta})=\frac{\beta^2}{(1+\beta x)^{-1}}.$$

$\endgroup$
6
$\begingroup$

Your present working appears to be taking all the $x_i$ values to be the same, which is not sufficient generality to properly describe your problem. In your initial setup for the problem there is no stipulation that these values must be the same. Taking the values $\mathbf{x} = (x_1,...,x_n)$ to be fixed explanatory variables, your response variables are independent (but not identically distributed) geometric random variables:

$$y_i | \mathbf{x} \sim \text{Geom}(\theta_i) \quad \quad \quad \theta_i \equiv \frac{\beta x_i}{1 + \beta x_i}.$$

As you can see, this generality makes the problem much more complicated than how you are treating it. If the $x_i$ values are different to each other then the response variables $y_i$ are not identically distributed, which will give you a more complicated likelihood function than what you are using. As shown below, this leads to an MLE that is defined (in general) only by an implicit function.


Finding the MLE: To get you back on track, I will show you how to derive the implicit equation for the MLE. For the observed data $\mathbf{y} = (y_1,...,y_n)$ we have the log-likelihood function:

$$\begin{aligned} \ell_{\mathbf{y}}(\beta) &= \sum_{i=1}^n \ln \text{Geom} \bigg( y_i \Bigg| \frac{\beta x_i}{1 + \beta x_i} \bigg) \\[6pt] &= \sum_{i=1}^n \Big[ \ln(\beta) + \ln(x_i) - y_i \ln(1+\beta x_i) \Big] \\[6pt] &= n \ln(\beta) + \sum_{i=1}^n \ln(x_i) - \sum_{i=1}^n y_i \ln(1+\beta x_i). \\[6pt] \end{aligned}$$

So you have the score function and Hessian function:

$$\begin{aligned} s_\mathbf{y}(\beta) \equiv \frac{\partial \ell_{\mathbf{y}}}{\partial \beta} (\beta) &= \frac{n}{\beta} - \sum_{i=1}^n \frac{x_i y_i}{1+\beta x_i}, \\[6pt] H_\mathbf{y}(\beta) \equiv \frac{\partial^2 \ell_{\mathbf{y}}}{\partial \beta^2} (\beta) &= - \frac{n}{\beta^2} + \sum_{i=1}^n \frac{x_i^2 y_i}{(1+\beta x_i)^2}. \\[6pt] \end{aligned}$$

At any critical point $\hat{\beta}$ (satisfying the critical point equation $s_\mathbf{y}(\hat{\beta}) =0$) you have:

$$\begin{aligned} H_\mathbf{y}(\hat{\beta}) &= - \frac{n}{\hat{\beta}^2} + \sum_{i=1}^n \frac{x_i^2 y_i}{(1+\hat{\beta} x_i)^2} \\[6pt] &= - \Bigg[ \frac{1}{n} \bigg( \sum_{i=1}^n \frac{x_i y_i}{1+\hat{\beta} x_i} \bigg)^2 - \sum_{i=1}^n \frac{x_i^2 y_i}{(1+\hat{\beta} x_i)^2} \Bigg] \\[6pt] &= - \Bigg[ \frac{1}{n} \bigg( \sum_{i=1}^n \sum_{j=1}^n \frac{x_i y_i}{1+\hat{\beta} x_i} \frac{x_j y_j}{1+\hat{\beta} x_j} \bigg) - \sum_{i=1}^n \frac{x_i^2 y_i}{(1+\hat{\beta} x_i)^2} \Bigg] \\[6pt] &= - \sum_{i=1}^n \frac{x_i y_i}{1+\hat{\beta} x_i} \bigg( \frac{1}{n} \sum_{j=1}^n \frac{x_j y_j}{1+\hat{\beta} x_j} - \frac{x_i}{1+\hat{\beta} x_i} \bigg). \\[6pt] \end{aligned}$$

The maximising point is defined implicitly by the critical point equation:

$$\frac{n}{\hat{\beta}} = \sum_{i=1}^n \frac{x_i y_i}{1+\hat{\beta} x_i}.$$

In the special case where $x \equiv x_1 = \cdots = x_n$ you have $\hat{\beta} = 1/(x (\bar{y}_n-1))$, but this explicit result does not hold in general.


Finding the asymptotic distribution of the MLE: If you want to find the asymptotic variance of the MLE, there are a few ways to do it. The complicated way is to differentiate the implicit function multiple times to get a Taylor approximation to the MLE, and then use this to get an asymptotic result for the variance of the MLE. That is quite a bit of work, and I will leave you to do it if you want to, but the above should get you started by giving you the correct equation for the MLE. You can read more about dealing with implicitly defined random variables using Taylor series and the delta method in Benichou and Gail (1989).

The simpler way to get the MLE is to rely on asymptotic theory for MLEs. Under some regularity conditions, you have the asymptotic distribution:

$$\sqrt{n}(\hat{\beta} - \beta)\overset{\rightarrow}{\sim} \text{N} \bigg( 0, \frac{1}{\mathcal{I}(\beta)} \bigg),$$

where $\mathcal{I}$ is the expected Fisher information for a single observation. Since $\mathbb{E}(\theta_i Y_i) = 1$ and $\mathbb{E}(\theta_i^2 Y_i^2) = 2-\theta_i$, in this case we have the Fisher information function for $n$ observations as being:

$$\begin{aligned} n\mathcal{I}(\beta) &= \mathbb{E}(s_\mathbf{Y}(\beta)^2) \\[6pt] &= \mathbb{E} \Bigg( \bigg( \frac{n}{\beta} - \sum_{i=1}^n \frac{x_i Y_i}{1+\beta x_i} \bigg)^2 \Bigg) \\[6pt] &= \frac{n^2}{\beta^2} \cdot \mathbb{E} \Bigg( \bigg( 1 - \frac{1}{n} \sum_{i=1}^n \theta_i Y_i \bigg)^2 \Bigg) \\[6pt] &= \frac{n^2}{\beta^2} \cdot \mathbb{E} \Bigg( 1 - \frac{2}{n} \sum_{i=1}^n \theta_i Y_i + \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n \theta_i \theta_j Y_i Y_j \Bigg) \\[6pt] &= \frac{n^2}{\beta^2} \cdot \mathbb{E} \Bigg( 1 - \frac{2}{n} \sum_{i=1}^n \theta_i Y_i + \frac{1}{n^2} \sum_{i=1}^n \theta_i^2 Y_i^2 + \frac{1}{n^2} \sum_{i \neq j} \theta_i \theta_j Y_i Y_j \Bigg) \\[6pt] &= \frac{n^2}{\beta^2} \cdot \Bigg( 1 - 2 + \frac{1}{n^2} \sum_{i=1}^n (2-\theta_i) + \frac{n-1}{n} \Bigg) \\[6pt] &= \frac{n^2}{\beta^2} \cdot \Bigg( 1 - 2 + \frac{2}{n} - \frac{1}{n^2} \sum_{i=1}^n \theta_i + 1 - \frac{1}{n} \Bigg) \\[6pt] &= \frac{n^2}{\beta^2} \cdot \Bigg( \frac{1}{n} - \frac{1}{n^2} \sum_{i=1}^n \theta_i \Bigg) \\[6pt] &= \frac{n(1 - \bar{\theta}_n)}{\beta^2}. \\[6pt] \end{aligned}$$

Here is an alternative derivation using the regularity conditions:

$$\begin{aligned} n\mathcal{I}(\beta) &= -\mathbb{E}(H_\mathbf{Y}(\beta)) \\[6pt] &= \mathbb{E} \Bigg( \frac{n}{\beta^2} - \sum_{i=1}^n \frac{x_i^2 Y_i}{(1+\beta x_i)^2} \Bigg) \\[6pt] &= \frac{n}{\beta^2} \cdot \mathbb{E} \Bigg( 1 - \frac{1}{n} \sum_{i=1}^n \theta_i^2 Y_i \Bigg) \\[6pt] &= \frac{n}{\beta^2} \cdot \Bigg( 1 - \frac{1}{n} \sum_{i=1}^n \theta_i \Bigg) \\[6pt] &= \frac{n(1-\bar{\theta}_n)}{\beta^2}. \\[6pt] \end{aligned}$$

So in this case you have the asymptotic variance:

$$\frac{1}{\mathcal{I}(\beta)} = \frac{\beta^2}{(1 - \bar{\theta}_\infty)}$$

where $\bar\theta_\infty = \lim_{n\rightarrow\infty} \bar\theta_n$. Based on this, for large but finite sample sizes, a large-sample variance for your estimator $\hat\beta$ is given by

$$\mathbb{V}(\hat{\beta}) \approx \frac{\hat\beta^2}{n(1 - \bar{\theta}_n)}$$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I really like your solution, where you explain the error in my thought and also share the correct work. When I work through it again, I'm getting the second term (the sum term) in the Hessian function is being added rather than subtracted. Is there a reason it remains as subtraction? $\endgroup$ – Ron Snow Apr 19 at 18:27
  • $\begingroup$ I have in my notes that the asymptotic variance is $\frac{1}{i(\beta)}$; however, I am not obtaining the correct answer. I put updated my post to include my new work. $\endgroup$ – Ron Snow Apr 19 at 19:04
  • 1
    $\begingroup$ I have updated the answer to correct the Hessian and add the derivation of the asymptotic variance. $\endgroup$ – Ben Apr 19 at 22:26
  • $\begingroup$ Thank you for adding the derivation of the asymptotic variance. I don't know why my derivation using $i(\beta)=-E[\frac{\partial}{\partial \beta}logf(y|\beta)]$ isn't returning the same answer. $\endgroup$ – Ron Snow Apr 19 at 23:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.