First of all an example dataset, both a picture and the R code to generate it. Here, X1 represents the ID of the individual / observation, X2 some factor1 and X3 some factor2.

enter image description here

df <- data.frame(matrix(nrow = 10, ncol = 4))
df[1:10,1] <- c(rep(1,2),2:9)
df[1:10,2] <- c("A","B", rep("A",8))
df[,2] <- as.factor(df[,2])
df[,3] <- c("XYZ", "XYZ", rep("ABC",8))
df[,3] <- as.factor(df[,3])
df[,4] <- c(5,5,runif(8))

If I would construct a linear model from this, my linear model in R (using lm()) will think I have 10 observations, while in fact I have 9. As you can see, the first two lines belong to observation/individual 1, but because this individual has two values for factor X2, it is shown twice in my dataset.

In my reallife dataset I have a lot of these cases. My dataset has 4.500 (4.5K) observations, and has a length of 60K+ because an individual can have multiple values for the same factor (for every factor in my dataset).

How can I let my linear model in R know that the first two lines are from the same observation? Is it through weights, or should I not use OLS to begin with?

EDIT In my real situation I have historical sales data (time series) for many products. To make predictions about these sales, I employed some time-series methods (e.g. exponential smoothing, ARIMA). Furthermore, I need to make predictions about sales for other products for which I do not have historical sales data. These are new products and I'd like to make an estimation about their sales, based on the products that do have sales data. In addition to having sales data, for each product I have product information at a more detailed level: all factors (e.g., color of the product, design, ...). The problem I have is that the weekly sales are on a less detailed product level (e.g., sales of a t-shirt in multiple colors, designs, ...).

My idea was to make a linear model using OLS, for which I use the average weekly sales as response (X4 here) in function of product information (X2 and X3). Then I would like to use this model and make predictions for new products for which I have such production information like X2 and X3. The problem (if it is a problem) is that I now have observations/products in my dataset more than once, due to the product information being on a more detailed level (e.g., the t-shirt in colors blue and red).


closed as unclear what you're asking by StatsStudent, mkt, Michael Chernick, whuber Feb 7 at 22:18

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ 1) Are you trying to include multiple observations from the same person in the same model (e.g. are trying to model X2)? Or 2) are you trying to remove points that should not be in the model? $\endgroup$ – mkt Feb 7 at 16:36
  • $\begingroup$ Sorry, I did not include my response variable in the example. I editted it. X4 is the variable I would like to model in function of X2 and X3. Note that the response is always the same for an observation that is in the dataset multiple times (i.e.: line 1 and 2 in my example). $\endgroup$ – Amonet Feb 7 at 16:48
  • 2
    $\begingroup$ Unless you are doing something in the field of quantum physics, how does a single observation have two different values for a variable at the same time? Surely you must be missing another variable that tracks time or some other dimensions that tracks observation order or something. I think you'll need to explain your data in a real-world context for us to fully understand based on the number of modifications and edits you've made. Can you tell us what your data represents in a real-world context? $\endgroup$ – StatsStudent Feb 7 at 16:57
  • 1
    $\begingroup$ Updated the question and deleted some of the comments. $\endgroup$ – Amonet Feb 7 at 17:20
  • 2
    $\begingroup$ Can you rewrite the problem in terms of the actual business problem you are trying to solve. Leave statistician s to tell you how to create right representation. $\endgroup$ – seanv507 Feb 7 at 20:19

You should not be using OLS if you plan to do any statistical inference. OLS assumes independent observations, which you obviously don't have if the same person appears in your dataset twice. You should look into random effects or mixed effects models. Other models might be appropriate too, depending on the distribution of your response variable.

Update based on comment

If all you plan to do is make predictions, you could simply fit your model with some randomly selected training data, and then validate your model on a separate validation dataset. The model that has the smallest mean squared predicted error (or whatever error criteria you want to use) should be used. I don't think you'd need to concern yourself with weighting -- at least not to arrive at proper variance estimates/standard errors (that's not to say modifying weights may help you obtain better predictions).

  • $\begingroup$ I plan to make predictions, so I guess I should be using for example lmer from the lme4 pakcage in R? $\endgroup$ – Amonet Feb 7 at 16:49
  • $\begingroup$ @Amonet, I updated my answer based on your comment. $\endgroup$ – StatsStudent Feb 7 at 16:53
  • 1
    $\begingroup$ @Amonet Statsstudent's edit is right. Prediction won't care about correlated errors, but inference does. $\endgroup$ – Demetri Pananos Feb 7 at 16:53
  • $\begingroup$ Let's say I use lm(x4 ~ x2 + x3, data = data), for now without training or test set. Then my predictions would be valid, despite that observations are in multiple times in the dataset? As you're speaking about weights, won't my model give more weight to observation 1 (line 1 and 2), because it's in twice? Whereas all other observations in this example are in just once? $\endgroup$ – Amonet Feb 7 at 16:58
  • 2
    $\begingroup$ I suppose this depends on what you mean by "valid." Yes, point estimates will be accurate. Variance estimates will be not. Each of the two observations will be given equal weight. $\endgroup$ – StatsStudent Feb 7 at 17:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.