5
$\begingroup$

For any particular nonlinear regression: $$Y_i = f(\mathbb{x_i},\theta) + \epsilon_i, i=1,...,n$$ I currently have standard errors for each of the $\theta_j$ obtained via the Gauss-Newton algorithm

Is there a systematic relationship between these standard errors and the sample size such that I would be able to predict the standard error if I had a different sample size?

Secondly, forgetting nonlinear regression, can this be done easily for linear regression? Is it just proportional to $\frac{1}{\sqrt{n}}$?

$\endgroup$

2 Answers 2

5
$\begingroup$

(A somewhat handwavy answer which hopefully gives some sense of the circumstances under which the answer could be a qualified yes)

Let's start with linear regression;

The standard errors come from the variance-covariance matrix, $\text{Var}(\hat \beta) = \sigma^2 (X'X)^{-1}$.

In the large and small samples, the population quantity $\sigma^2$ is assumed to be the same, so what's left? The design matrix, $X$. If you imagine there's a distribution in $x$-space, and the rows are random samples from it, then there's a sense in which the large and small samples must both estimate the same population quantities. Specifically, consider $(X'X)/n$ where the $X$ is made up of randomly chosen rows, $x_i$; then let $C_i = x_i x_i'$ be considered random variables. Under this scheme, the $C_i$ are i.i.d. and $(X'X)/n = \bar c$ will converge to the mean of the distribution of $C$ under the usual conditions for means to converge. Note that $C$ is a random matrix and so its mean will be a matrix. [In practice you can stack it up into a vector, $\text{vec}(C)$, or eliminating the redundant half, $\text{vech}(C)$, which among other things means we could write its variance-covariance as a matrix.]

Then $\sigma^2 {(X'X)}^{-1} = \sigma^2/n\cdot {(X'X/n)}^{-1}$, where the term in parentheses on the right converges to a constant as $n \rightarrow\infty$.

If, however, the samples don't cover the same parts of $x$-space, or not in the same proportions, that sense would clearly be lost.

In the case of nonlinear regression, the variance-estimates are based around a linear approximation at the optimum, so most of the notions should carry over. All being well convergence-wise (if the nls calculations don't converge this may not apply), and assuming the small sample and the large sample are dealing with the same region of x-space in the same proportions, the small sample and the large sample should both approximate the same variance-covariance matrix.

In that sense, under random sampling and assuming they're dealing with the same distribution over $x$-space, a small sample will be like a pilot study for the remainder of the larger sample.

However, those are pretty stringent assumptions and in practice I'd imagine they'd be frequently not satisfied. In an experiment, for example, you could make sure to cover the $x$-space in such a fashion over observations so that it behaved like a random sample from the same distribution. In a situation where you have to take the $x_i$ as they come, you may not have it, and generally won't.

$\endgroup$
0
0
$\begingroup$

I ran into this same dilemma.

Although there may be an analytic approach for this, my sample was small enough so I could ran simulated regressions:

  1. Sample your data with replacement
  2. Run your regression
  3. Estimate your coefficients and standard errors
  4. Repeat N times until you have an stable distribution

At the end of the process, the mean of your estimated coefficients should approximate the coefficients of your original regression (i.e. without sampling) and you'll have a distribution for the standard errors that should give you a decent idea of how you could expect them to change as you increase your sample.

This, off course, assumes your new data will come from the same Generation Process, and that for any given observation, probability of being part of any of the two samples (original and new) is the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.