10
$\begingroup$

Suppose I have a random sample $\lbrace X_i, Y_i\rbrace_{i=1}^n$. Assume this sample is such that the Gauss-Markov assumptions are satisfied such that I can construct an OLS estimator where

$$\hat{\beta}_1^{OLS} = \frac{\text{Cov}(X,Y)}{\text{Var(X)}}$$ $$\hat{\beta}_0^{OLS} = \bar{Y} - \bar{X} \hat{\beta}_1^{OLS}$$

Now suppose I take my data set and double it, meaning there is an exact copy for each of the $n$ $(X_i,Y_i)$ pairs.

My Question

How does this affect my ability to use OLS? Is it still consistent and identified?

$\endgroup$
5
  • 1
    $\begingroup$ This is a linear transformation of the data, why don't you try seeing how this affects the estimator directly? $\endgroup$
    – JohnK
    Jun 2, 2016 at 17:33
  • 3
    $\begingroup$ @JohnK It's difficult to conceive of this as a "linear transformation" when it actually introduces additional data! $\endgroup$
    – whuber
    Jun 2, 2016 at 17:35
  • $\begingroup$ @whuber Linear in the sense that $x_i \mapsto 2x_i$. $\endgroup$
    – JohnK
    Jun 2, 2016 at 17:38
  • 9
    $\begingroup$ @Johnk The OP isn't doubling the values in the data set but stacking two copies of the same data set. $\endgroup$
    – Glen_b
    Jun 2, 2016 at 17:39
  • $\begingroup$ $x_i \mapsto (x_i,x_i)$ is still linear ... $\endgroup$ Jan 5, 2019 at 10:46

2 Answers 2

20
$\begingroup$

Do you have a good reason to do the doubling (or duplication?) It doesn't make much statistical sense, but still it is interesting to see what happens algebraically. In matrix form your linear model is $$ \DeclareMathOperator{\V}{\mathbb{V}} Y = X \beta + E, $$ the least square estimator is $\hat{\beta}_{\text{ols}} = (X^T X)^{-1} X^T Y $ and the variance matrix is $ \V \hat{\beta}_{\text{ols}}= \sigma^2 (X^t X)^{-1} $. "Doubling the data" means that $Y$ is replaced by $\begin{pmatrix} Y \\ Y \end{pmatrix}$ and $X$ is replaced by $\begin{pmatrix} X \\ X \end{pmatrix}$. The ordinary least squares estimator then becomes $$ \left(\begin{pmatrix}X \\ X \end{pmatrix}^T \begin{pmatrix} X \\ X \end{pmatrix} \right )^{-1} \begin{pmatrix} X \\ X \end{pmatrix}^T \begin{pmatrix} Y \\ Y \end{pmatrix} = \\ (x^T X + X^T X)^{-1} (X^T Y + X^T Y ) = (2 X^T X)^{-1} 2 X^T Y = \\ \hat{\beta}_{\text{ols}} $$ so the calculated estimator doesn't change at all. But the calculated variance matrix becomes wrong: Using the same kind of algebra as above, we get the variance matrix $\frac{\sigma^2}{2}(X^T X)^{-1}$, half of the correct value. A consequence is that confidence intervals will shrink with a factor of $\frac{1}{\sqrt{2}}$.

The reason is that we have calculated as if we still have iid data, which is untrue: the pair of doubled values obviously have a correlation equal to $1.0$. If we take this into account and use weighted least squares correctly, we will find the correct variance matrix.

From this, more consequences of the doubling will be easy to find as an exercise, for instance, the value of R-squared will not change.

$\endgroup$
8
  • 6
    $\begingroup$ And in fact, you can show that this is equivalent to a weighted least squares with weight 2 on each observation. $\endgroup$
    – Andrew M
    Jun 2, 2016 at 19:21
  • 2
    $\begingroup$ Surprisingly, there are areas of statistics in which there is (arguably) sometimes a reason to do this. It goes by the name "data cloning". $\endgroup$
    – Flounderer
    Jun 2, 2016 at 23:17
  • $\begingroup$ Interesting! datacloning.org/dc.html $\endgroup$ Jun 3, 2016 at 7:30
  • $\begingroup$ In Python, the $\sigma^2$ is estimated using $\hat{\sigma}^2$ which is computed using $\frac{1}{N - p - 1}$ factor rather than $\frac{1}{N}$, this actually makes the factor of $\frac{1}{\sqrt{2}}$ slightly bigger. $\endgroup$
    – 24n8
    May 6, 2021 at 19:29
  • $\begingroup$ It may be an obvious question. But after duplicating the data, say just add duplicate one row of $X$, then the new data matrix $X$ is not full rank, thus not invertible, and neither is $X^TX$. But how do we still have $\hat{\beta} = (X^TX)^{-1}X^TY$? That will imply the solution is unique(but it is not because $X$ is not full rank) $\endgroup$
    – Gavin
    Mar 16 at 22:26
0
$\begingroup$

I am not yet familiar enough with the theory to give you a very mathematical answer, but intuitively, OLS only cares about proportions in which different cases are present. This makes sense when you recall that OLS chooses the coefficients that minimize the mean of the squared residuals, and the mean reflects purely the proportions of its inputs (in the sense that the mean of (1, 3, 3) is the same as the mean of a dataset with a million 1s and two million 3s). So, doubling the dataset will get you the identical model.


Here's an R example, where we generate a random regression problem and notice that the coefficients are unchanged when doubling the data:

nc = sample(1:10, 1, replace = T)
n = sample(11:500, 1, replace = T)
x = as.matrix(replicate(nc, rnorm(n)))
coef = rnorm(nc)
sd.resid = runif(1, 0, 5)

y = x %*% matrix(coef) + rnorm(n, sd = sd.resid)

print(cbind(
  coef(lm(y ~ x)),
  coef(lm(c(y, y) ~ rbind(x, x)))))

One run gives me:

                   [,1]        [,2]
(Intercept) -0.10002238 -0.10002238
x1          -2.14801619 -2.14801619
x2           0.23120764  0.23120764
x3           0.05360792  0.05360792
x4           1.91972198  1.91972198
x5          -1.09887264 -1.09887264
x6           0.04248358  0.04248358
$\endgroup$
11
  • 5
    $\begingroup$ As statisticians, we do far more than just estimate parameters. At the least, we evaluate how uncertain they are. Doubling the data will (typically and approximately) halve the variances, making the results look much more solid than they ought to. This error has huge ramifications for interval estimation, such as confidence and prediction intervals. Another way to characterize the mistake is to recognize that each doubled pair of responses is perfectly correlated, so applying OLS to them as if they were independent uses the wrong error structure. $\endgroup$
    – whuber
    Jun 2, 2016 at 17:32
  • 1
    $\begingroup$ I agree, but to me, that is a matter of inferences about the model rather the model itself. Doing something like computing a confidence interval for a parameter is not part of OLS per se, even if the parameter comes from OLS. $\endgroup$ Jun 2, 2016 at 17:41
  • $\begingroup$ Inference is impossible without information about the potential inaccuracy. Otherwise, you have no valid response to the question "how sure are you of your answer?" That is why (at a minimum) computing a variance-covariance matrix of the residuals is an essential part of OLS. Anything less is just algorithmic curve fitting. $\endgroup$
    – whuber
    Jun 2, 2016 at 17:42
  • 1
    $\begingroup$ Yes, that's why I said inferences about the model. I would agree that OLS per se is a kind of algorithmic curve-fitting. $\endgroup$ Jun 2, 2016 at 17:45
  • 1
    $\begingroup$ In that case, what does one call the procedure of computing the coefficients of a linear model such that the sum of the squared deviations is minimized? $\endgroup$ Jun 2, 2016 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.