# Approaches to modeling data like this in R

A couple years ago I performed a linear regression on data that looked like this:

   company year     y       x1      x2      x3      x4
1        A 2012  1.83  34811.8 14755.5   278.2     0.0
2        B 2012  3.87  10435.5  9692.6   522.2   317.9
3        C 2012 19.76 199670.6 23428.7 10675.5  2815.8
4        D 2012  1.22   3204.4  2087.5  2282.8  2804.1
5        E 2012  0.00      5.2    53.5     0.2   193.8
6        F 2012  0.81 161936.0 25777.9  2364.8   540.6
7        G 2012  1.22   1479.3    28.6     0.4     3.9
8        H 2012  2.24   9716.3   888.2  2073.9  1059.1
9        I 2012 25.25 331396.9 15162.0 87062.1 32724.7
10       J 2012  0.20   9812.0 10363.4    49.9 36664.9
11       K 2012  1.02  62715.3  5746.5  1007.7   866.3
12       L 2012  3.87 121397.5  5842.2  1481.6   621.0
13       M 2012 12.22 243189.5 50370.8 16747.1 23025.8
14       N 2012 18.33 147305.6 87916.3 15098.3 16449.7
15       O 2012  0.61  20699.1  8345.6     0.0    26.4
16       P 2012  2.44  30735.1  1840.6  4900.1     0.0


Each row is a different company and the main objective was to interpret the coefficients. Its been 3 years since that regression and I want to look at it again with data from each year so the dataset would look like this:

   company year     y       x1      x2       x3      x4
1        A 2012  1.83  34811.8 14755.5    278.2     0.0
2        B 2012  3.87  10435.5  9692.6    522.2   317.9
3        C 2012 19.76 199670.6 23428.7  10675.5  2815.8
4        D 2012  1.22   3204.4  2087.5   2282.8  2804.1
5        E 2012  0.00      5.2    53.5      0.2   193.8
6        F 2012  0.81 161936.0 25777.9   2364.8   540.6
7        G 2012  1.22   1479.3    28.6      0.4     3.9
8        H 2012  2.24   9716.3   888.2   2073.9  1059.1
9        I 2012 25.25 331396.9 15162.0  87062.1 32724.7
10       J 2012  0.20   9812.0 10363.4     49.9 36664.9
11       K 2012  1.02  62715.3  5746.5   1007.7   866.3
12       L 2012  3.87 121397.5  5842.2   1481.6   621.0
13       M 2012 12.22 243189.5 50370.8  16747.1 23025.8
14       N 2012 18.33 147305.6 87916.3  15098.3 16449.7
15       O 2012  0.61  20699.1  8345.6      0.0    26.4
16       P 2012  2.44  30735.1  1840.6   4900.1     0.0
17       A 2013  0.20   4832.1 10691.6      0.6     0.0
18       B 2013  3.02  12575.8  1270.3    106.6   368.0
19       C 2013 16.00 184628.5 38269.7   5343.1  4645.6
20       D 2013  1.76   4684.6  1445.2   2150.1  1727.0
21       E 2013  1.27      4.3    22.9     38.3   314.6
22       F 2013  0.39 141808.6 26368.8    673.6  2259.2
23       G 2013  0.59    986.3    38.6      7.0     5.8
24       H 2013  2.83  20111.4  3518.3    549.5    59.6
25       I 2013 21.17 303925.9 20248.0 107366.7 19979.1
26       J 2013  1.37   7792.8 16000.7     33.5 39541.7
27       K 2013  1.66 141071.9 11136.1    162.2     0.0
28       L 2013  3.80 130359.7  8882.5     40.5   520.8
29       M 2013 10.63 280250.3 39029.7  16208.6 29284.3
30       N 2013 19.41 145278.1 55141.6  14115.5  1783.4
31       O 2013  0.98   1517.6  3610.4      0.0   547.3
32       P 2013  3.32 101484.2  1140.5   5489.9     0.0
33       A 2014  0.10      0.0  9520.7      0.9     0.0
34       B 2014  4.02  14886.8  2331.5      0.0   631.8
35       C 2014 14.22 143760.9 50222.1   6118.1  4342.1
36       D 2014  0.88    936.1  1802.7   1273.6  4394.3
37       E 2014  0.78    231.5    15.8     64.1   291.9
38       F 2014  0.78 244303.2 29148.3   3161.4  4908.1
39       G 2014  0.78   1032.6    30.3      1.3     7.8
40       H 2014  2.55  26322.6 11726.1   2859.2     0.0
41       I 2014 21.96 614241.5  9138.2  94273.7 17702.0
42       J 2014  1.27   8946.5 13853.7    693.9 19672.0
43       K 2014  1.18 164269.7  7088.1     29.7   825.0
44       L 2014  2.35 107152.3  3275.2     94.7   490.9
45       M 2014  8.73 284267.4 51896.4  12838.1 28019.5
46       N 2014 20.69  84554.6 32341.0  11408.2   624.9
47       O 2014  1.08      0.0  7663.2      0.0     0.0
48       P 2014  3.63 109392.9  5229.2   4691.0    11.1


When I think about this dataset I don't immediately think its a time series but I also don't think I should be ignoring year all together and regressing it like so in R:

lm(y ~ x1 + x2 + x3 + x4)


So I'm wondering how I should model this dataset. Should I just include dummy variables for year or are there better approaches here?

• I'd consider treating it as panel data – Glen_b -Reinstate Monica Jun 16 '15 at 2:46
• I would suggest that you shouldn't worry about modeling it in R and instead just worry about modeling it. – shadowtalker Jun 16 '15 at 2:55
• @ssdecontrol thanks for the advice. I hear what your are saying but I think connecting theory to practice i.e code is how I learn best. – moku Jun 16 '15 at 14:45
• @moku if your goal is to learn R, you should practice R. If your goal is to learn modeling, you should practice modeling. It is obviously necessary to limit the models we used to those that we can implement on a computer. But there are a lot of software packages available, at the end of the day a software package or programming language is just a tool. Choose the right tools for the job, and not the right job for the tool. That said, R is a great tool and I don't want to discourage you from learning it – shadowtalker Jun 16 '15 at 14:54
• That was a long comment, but I guess my main point is: don't confuse the solution with the tool used to obtain the solution – shadowtalker Jun 16 '15 at 14:58

See the plm package, it has many examples of applications. Panel data models seems a good start for your data. See some initial code:

library(plm)

mp <- plm(y ~ x1 + x2 + x3 + x4,model = "pooling",
data = df, index = c("company","year"))
summary(mp) #no dummies, like lm()

m1 <- plm(y ~ x1 + x2 + x3 + x4, model = "within",
data = df, index = c("company","year"))
summary(m1) #fe
fixef(m1)  # individual

m2 <- plm(y ~ x1 + x2 + x3 + x4,model = "within",effect = "twoways",
data = df, index = c("company","year"))
summary(m2) #fe
fixef(m2)  # individual
fixef(m2,effect="time")  # time

pFtest(m1, mp) #individuals yes
pFtest(m2, m1) #time no

• Thanks for this it will certainly help me get started. I was thinking more about my time dimension as it relates to my response variable and I wonder how time effects the response consider that the response is a zero-sum metric e.g. one company must lose for another company to gain. Does the panel approach still work in this case? – moku Jun 16 '15 at 14:43
• There are some variations of the basic models, but I don't think that your situation is considered. It seems you will probably need to make some transformations in key variables. – Robert Jun 16 '15 at 21:20

I would generally include those, it's hard to say why you want to do it in your case because we don't know any specifics about the context. One example that that comes to mind:

If you are trying to inspect the effect of $x_1$ on $y$ you have to take into account that you may not be including some "unobserved" effects that cause massive disturbances in your model, these might cause the estimation of the effect of $x_1$ to be wrong. Let's say that:

• $x_1$ is years of education of the subject
• $y$ how happy a person
• Every two years there are elections causing massive happiness spikes in the general population

In this case, you would want to include a dummy $ELECTION$ to catch the spike effect that would otherwise overshadow your analysis of the pure effect of $x_1$ (essentially controlling for it). Problematically, you don't know the value of $ELECTION$ because it's unobserved. Including year might allow you to catch the spike anyway even though you didn't even know $ELECTION$ was a problem in the first place :).

So if you say you don't think it's a time series, the question really is whether there might have been some things that could have disturbed the y-value anyway throughout time.