5
$\begingroup$

Suppose I have a random sample $\lbrace X_i, Y_i\rbrace_{i=1}^n$. Assume this sample is such that the Gauss-Markov assumptions are satisfied such that I can construct an OLS estimator where

$$\hat{\beta}_1^{OLS} = \frac{\text{Cov}(X,Y)}{\text{Var(X)}}$$ $$\hat{\beta}_0^{OLS} = \bar{Y} - \bar{X} \hat{\beta}_1^{OLS}$$

Now suppose I take my data set and double it, meaning there is an exact copy for each of the $n$ $(X_i,Y_i)$ pairs.

My Question

How does this affect my ability to use OLS? Is it still consistent and identified?

$\endgroup$
  • 1
    $\begingroup$ This is a linear transformation of the data, why don't you try seeing how this affects the estimator directly? $\endgroup$ – JohnK Jun 2 '16 at 17:33
  • 2
    $\begingroup$ @JohnK It's difficult to conceive of this as a "linear transformation" when it actually introduces additional data! $\endgroup$ – whuber Jun 2 '16 at 17:35
  • $\begingroup$ @whuber Linear in the sense that $x_i \mapsto 2x_i$. $\endgroup$ – JohnK Jun 2 '16 at 17:38
  • 8
    $\begingroup$ @Johnk The OP isn't doubling the values in the data set but stacking two copies of the same data set. $\endgroup$ – Glen_b -Reinstate Monica Jun 2 '16 at 17:39
  • $\begingroup$ $x_i \mapsto (x_i,x_i)$ is still linear ... $\endgroup$ – kjetil b halvorsen Jan 5 at 10:46
13
$\begingroup$

Do you have a good reason to do the doubling (or duplication?) It doesn't make much statistical sense, but still it is interesting to see what happens algebraically. In matrix form your linear model is $$ \DeclareMathOperator{\V}{\mathbb{V}} Y = X \beta + E, $$ the least square estimator is $\hat{\beta}_{\text{ols}} = (X^T X)^{-1} X^T Y $ and the variance matrix is $ \V \hat{\beta}_{\text{ols}}= \sigma^2 (X^t X)^{-1} $. "Doubling the data" means that $Y$ is replaced by $\begin{pmatrix} Y \\ Y \end{pmatrix}$ and $X$ is replaced by $\begin{pmatrix} X \\ X \end{pmatrix}$. The ordinary least squares estimator then becomes $$ \left(\begin{pmatrix}X \\ X \end{pmatrix}^T \begin{pmatrix} X \\ X \end{pmatrix} \right )^{-1} \begin{pmatrix} X \\ X \end{pmatrix}^T \begin{pmatrix} Y \\ Y \end{pmatrix} = \\ (x^T X + X^T X)^{-1} (X^T Y + X^T Y ) = (2 X^T X)^{-1} 2 X^T Y = \\ \hat{\beta}_{\text{ols}} $$ so the calculated estimator doesn't change at all. But the calculated variance matrix becomes wrong: Using the same kind of algebra as above, we get the variance matrix $\frac{\sigma^2}{2}(X^T X)^{-1}$, half of the correct value. A consequence is that confidence intervals will shrink with a factor of $\frac{1}{\sqrt{2}}$.

The reason is that we have calculated as if we still have iid data, which is untrue: the pair of doubled values obviously have a correlation equal to $1.0$. If we take this into account and use weighted least squares correctly, we will find the correct variance matrix.

From this, more consequences of the doubling will be easy to find as an exercise, for instance, the value of R-squared will not change.

$\endgroup$
  • 6
    $\begingroup$ And in fact, you can show that this is equivalent to a weighted least squares with weight 2 on each observation. $\endgroup$ – Andrew M Jun 2 '16 at 19:21
  • 2
    $\begingroup$ Surprisingly, there are areas of statistics in which there is (arguably) sometimes a reason to do this. It goes by the name "data cloning". $\endgroup$ – Flounderer Jun 2 '16 at 23:17
  • $\begingroup$ Interesting! datacloning.org/dc.html $\endgroup$ – kjetil b halvorsen Jun 3 '16 at 7:30
1
$\begingroup$

I am not yet familiar enough with the theory to give you a very mathematical answer, but intuitively, OLS only cares about proportions in which different cases are present. This makes sense when you recall that OLS chooses the coefficients that minimize the mean of the squared residuals, and the mean reflects purely the proportions of its inputs (in the sense that the mean of (1, 3, 3) is the same as the mean of a dataset with a million 1s and two million 3s). So, doubling the dataset will get you the identical model.


Here's an R example, where we generate a random regression problem and notice that the coefficients are unchanged when doubling the data:

nc = sample(1:10, 1, replace = T)
n = sample(11:500, 1, replace = T)
x = as.matrix(replicate(nc, rnorm(n)))
coef = rnorm(nc)
sd.resid = runif(1, 0, 5)

y = x %*% matrix(coef) + rnorm(n, sd = sd.resid)

print(cbind(
  coef(lm(y ~ x)),
  coef(lm(c(y, y) ~ rbind(x, x)))))

One run gives me:

                   [,1]        [,2]
(Intercept) -0.10002238 -0.10002238
x1          -2.14801619 -2.14801619
x2           0.23120764  0.23120764
x3           0.05360792  0.05360792
x4           1.91972198  1.91972198
x5          -1.09887264 -1.09887264
x6           0.04248358  0.04248358
$\endgroup$
  • 4
    $\begingroup$ As statisticians, we do far more than just estimate parameters. At the least, we evaluate how uncertain they are. Doubling the data will (typically and approximately) halve the variances, making the results look much more solid than they ought to. This error has huge ramifications for interval estimation, such as confidence and prediction intervals. Another way to characterize the mistake is to recognize that each doubled pair of responses is perfectly correlated, so applying OLS to them as if they were independent uses the wrong error structure. $\endgroup$ – whuber Jun 2 '16 at 17:32
  • 1
    $\begingroup$ I agree, but to me, that is a matter of inferences about the model rather the model itself. Doing something like computing a confidence interval for a parameter is not part of OLS per se, even if the parameter comes from OLS. $\endgroup$ – Kodiologist Jun 2 '16 at 17:41
  • $\begingroup$ Inference is impossible without information about the potential inaccuracy. Otherwise, you have no valid response to the question "how sure are you of your answer?" That is why (at a minimum) computing a variance-covariance matrix of the residuals is an essential part of OLS. Anything less is just algorithmic curve fitting. $\endgroup$ – whuber Jun 2 '16 at 17:42
  • 1
    $\begingroup$ Yes, that's why I said inferences about the model. I would agree that OLS per se is a kind of algorithmic curve-fitting. $\endgroup$ – Kodiologist Jun 2 '16 at 17:45
  • 1
    $\begingroup$ In that case, what does one call the procedure of computing the coefficients of a linear model such that the sum of the squared deviations is minimized? $\endgroup$ – Kodiologist Jun 2 '16 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.