15
$\begingroup$

Suppose I have a random sample $\lbrace X_i, Y_i\rbrace_{i=1}^n$. Assume this sample is such that the Gauss-Markov assumptions are satisfied such that I can construct an OLS estimator where

$$\hat{\beta}_1^{OLS} = \frac{\text{Cov}(X,Y)}{\text{Var(X)}}$$ $$\hat{\beta}_0^{OLS} = \bar{Y} - \bar{X} \hat{\beta}_1^{OLS}$$

Now suppose I take my data set and double it, meaning there is an exact copy for each of the $n$ $(X_i,Y_i)$ pairs.

My Question

How does this affect my ability to use OLS? Is it still consistent and identified?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ This is a linear transformation of the data, why don't you try seeing how this affects the estimator directly? $\endgroup$
    – JohnK
    Commented Jun 2, 2016 at 17:33
  • 4
    $\begingroup$ @JohnK It's difficult to conceive of this as a "linear transformation" when it actually introduces additional data! $\endgroup$
    – whuber ♦
    Commented Jun 2, 2016 at 17:35
  • $\begingroup$ @whuber Linear in the sense that $x_i \mapsto 2x_i$. $\endgroup$
    – JohnK
    Commented Jun 2, 2016 at 17:38
  • 11
    $\begingroup$ @Johnk The OP isn't doubling the values in the data set but stacking two copies of the same data set. $\endgroup$
    – Glen_b
    Commented Jun 2, 2016 at 17:39
  • 1
    $\begingroup$ $x_i \mapsto (x_i,x_i)$ is still linear ... $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jan 5, 2019 at 10:46

2 Answers 2

Reset to default
26
$\begingroup$

Do you have a good reason to do the doubling (or duplication?) It doesn't make much statistical sense, but still it is interesting to see what happens algebraically. In matrix form your linear model is $$ \DeclareMathOperator{\V}{\mathbb{V}} Y = X \beta + E, $$ the least square estimator is $\hat{\beta}_{\text{ols}} = (X^T X)^{-1} X^T Y $ and the variance matrix is $ \V \hat{\beta}_{\text{ols}}= \sigma^2 (X^t X)^{-1} $. "Doubling the data" means that $Y$ is replaced by $\begin{pmatrix} Y \\ Y \end{pmatrix}$ and $X$ is replaced by $\begin{pmatrix} X \\ X \end{pmatrix}$. The ordinary least squares estimator then becomes $$ \left(\begin{pmatrix}X \\ X \end{pmatrix}^T \begin{pmatrix} X \\ X \end{pmatrix} \right )^{-1} \begin{pmatrix} X \\ X \end{pmatrix}^T \begin{pmatrix} Y \\ Y \end{pmatrix} \, =\, \\ (X^T X + X^T X)^{-1} (X^T Y + X^T Y ) \, = \, (2 X^T X)^{-1} 2 X^T Y = \\ \hat{\beta}_{\text{ols}} $$ so the calculated estimator doesn't change at all. But the calculated variance matrix becomes wrong: Using the same kind of algebra as above, we get the variance matrix $\frac{\sigma^2}{2}(X^T X)^{-1}$, half of the correct value. A consequence is that confidence intervals will shrink with a factor of $\frac{1}{\sqrt{2}}$.

The reason is that we have calculated as if we still have iid data, which is untrue: the pair of doubled values obviously have a correlation equal to $1.0$. If we take this into account and use weighted least squares correctly, we will find the correct variance matrix.

From this, more consequences of the doubling will be easy to find as an exercise, for instance, the value of R-squared will not change.

Improve this answer
$\endgroup$
8
  • 7
    $\begingroup$ And in fact, you can show that this is equivalent to a weighted least squares with weight 2 on each observation. $\endgroup$
    – Andrew M
    Commented Jun 2, 2016 at 19:21
  • 3
    $\begingroup$ Surprisingly, there are areas of statistics in which there is (arguably) sometimes a reason to do this. It goes by the name "data cloning". $\endgroup$
    – Flounderer
    Commented Jun 2, 2016 at 23:17
  • $\begingroup$ Interesting! datacloning.org/dc.html $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 3, 2016 at 7:30
  • $\begingroup$ In Python, the $\sigma^2$ is estimated using $\hat{\sigma}^2$ which is computed using $\frac{1}{N - p - 1}$ factor rather than $\frac{1}{N}$, this actually makes the factor of $\frac{1}{\sqrt{2}}$ slightly bigger. $\endgroup$
    – 24n8
    Commented May 6, 2021 at 19:29
  • 1
    $\begingroup$ @kjetilbhalvorsen I think I was confused by the terms $rank$. In this case, it means the column-rank. If $X$ is full column rank, then $X^TX$ is invertible. And I see your point, adding new rows does not change the column rank. $\endgroup$
    – Gavin
    Commented Mar 16, 2022 at 23:31
0
$\begingroup$

I am not yet familiar enough with the theory to give you a very mathematical answer, but intuitively, OLS only cares about proportions in which different cases are present. This makes sense when you recall that OLS chooses the coefficients that minimize the mean of the squared residuals, and the mean reflects purely the proportions of its inputs (in the sense that the mean of (1, 3, 3) is the same as the mean of a dataset with a million 1s and two million 3s). So, doubling the dataset will get you the identical model.

Here's an R example, where we generate a random regression problem and notice that the coefficients are unchanged when doubling the data:

nc = sample(1:10, 1, replace = T)
n = sample(11:500, 1, replace = T)
x = as.matrix(replicate(nc, rnorm(n)))
coef = rnorm(nc)
sd.resid = runif(1, 0, 5)

y = x %*% matrix(coef) + rnorm(n, sd = sd.resid)

print(cbind(
  coef(lm(y ~ x)),
  coef(lm(c(y, y) ~ rbind(x, x)))))

One run gives me:

                   [,1]        [,2]
(Intercept) -0.10002238 -0.10002238
x1          -2.14801619 -2.14801619
x2           0.23120764  0.23120764
x3           0.05360792  0.05360792
x4           1.91972198  1.91972198
x5          -1.09887264 -1.09887264
x6           0.04248358  0.04248358
Improve this answer
$\endgroup$
12
  • 6
    $\begingroup$ As statisticians, we do far more than just estimate parameters. At the least, we evaluate how uncertain they are. Doubling the data will (typically and approximately) halve the variances, making the results look much more solid than they ought to. This error has huge ramifications for interval estimation, such as confidence and prediction intervals. Another way to characterize the mistake is to recognize that each doubled pair of responses is perfectly correlated, so applying OLS to them as if they were independent uses the wrong error structure. $\endgroup$
    – whuber ♦
    Commented Jun 2, 2016 at 17:32
  • 1
    $\begingroup$ I agree, but to me, that is a matter of inferences about the model rather the model itself. Doing something like computing a confidence interval for a parameter is not part of OLS per se, even if the parameter comes from OLS. $\endgroup$
    – Kodiologist
    Commented Jun 2, 2016 at 17:41
  • 1
    $\begingroup$ Yes, that's why I said inferences about the model. I would agree that OLS per se is a kind of algorithmic curve-fitting. $\endgroup$
    – Kodiologist
    Commented Jun 2, 2016 at 17:45
  • 1
    $\begingroup$ In that case, what does one call the procedure of computing the coefficients of a linear model such that the sum of the squared deviations is minimized? $\endgroup$
    – Kodiologist
    Commented Jun 2, 2016 at 17:54
  • 1
    $\begingroup$ @Iamanon You calculate the variance as if the errors in $(Y,Y)^\prime$ were iid--but they are not: they come in perfectly correlated pairs. $\endgroup$
    – whuber ♦
    Commented Aug 9, 2020 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.