First of all an example dataset, both a picture and the R code to generate it. Here,
X1 represents the ID of the individual / observation,
X2 some factor1 and
X3 some factor2.
df <- data.frame(matrix(nrow = 10, ncol = 4)) df[1:10,1] <- c(rep(1,2),2:9) df[1:10,2] <- c("A","B", rep("A",8)) df[,2] <- as.factor(df[,2]) df[,3] <- c("XYZ", "XYZ", rep("ABC",8)) df[,3] <- as.factor(df[,3]) df[,4] <- c(5,5,runif(8))
If I would construct a linear model from this, my linear model in R (using
lm()) will think I have 10 observations, while in fact I have 9. As you can see, the first two lines belong to observation/individual 1, but because this individual has two values for
factor X2, it is shown twice in my dataset.
In my reallife dataset I have a lot of these cases. My dataset has 4.500 (4.5K) observations, and has a length of 60K+ because an individual can have multiple values for the same factor (for every factor in my dataset).
How can I let my linear model in R know that the first two lines are from the same observation? Is it through weights, or should I not use OLS to begin with?
EDIT In my real situation I have historical sales data (time series) for many products. To make predictions about these sales, I employed some time-series methods (e.g. exponential smoothing, ARIMA). Furthermore, I need to make predictions about sales for other products for which I do not have historical sales data. These are new products and I'd like to make an estimation about their sales, based on the products that do have sales data. In addition to having sales data, for each product I have product information at a more detailed level: all factors (e.g., color of the product, design, ...). The problem I have is that the weekly sales are on a less detailed product level (e.g., sales of a t-shirt in multiple colors, designs, ...).
My idea was to make a linear model using OLS, for which I use the average weekly sales as response (
X4 here) in function of product information (
X3). Then I would like to use this model and make predictions for new products for which I have such production information like
X3. The problem (if it is a problem) is that I now have observations/products in my dataset more than once, due to the product information being on a more detailed level (e.g., the t-shirt in colors blue and red).