1
$\begingroup$

So I am working on this regression and I am doing it for multiple levels of Females. I have done one regression for Females 18-34 and got this output

F18<-read.csv("C:/Users/marissa.ferguson/Desktop/Unrated/F18-34.csv", header = T, sep = ",", na.strings = "?")
female<-na.omit(F18)
set.seed(1000)
train.size<-0.8
train.index<- sample.int(length(subfemale$DemoMedianRtg), round(length(subfemale$DemoMedianRtg)*train.size))
train.sample<-subfemale[train.index,]
test.sample<-subfemale[-train.index,]

> Overall<-lm(DemoMedianRtg~ DP+Subscribers+Tier+SubRange+Male.+Female.+Avg.Age+NewTier+NewSubs+Avg.Income, data=train.sample)
> summary(Overall)

Call:
lm(formula = DemoMedianRtg ~ DP + Subscribers + Tier + SubRange + 
Male. + Female. + Avg.Age + NewTier + NewSubs + Avg.Income, 
data = train.sample)

Residuals:
  Min        1Q    Median        3Q       Max 
-0.038269 -0.014386  0.003568  0.012190  0.029538 

Coefficients: (8 not defined because of singularities)
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)      2.258e+00  5.495e-01   4.108 0.000375 ***
DPEarly Fringe   8.178e-03  1.129e-02   0.724 0.475665    
DPEarly Morning  1.012e-02  1.739e-02   0.582 0.566011    
DPLate Fringe    7.304e-02  1.643e-02   4.445 0.000157 ***
DPOvernight      3.327e-02  1.968e-02   1.691 0.103281    
DPPrimeTime      1.999e-02  1.188e-02   1.682 0.105005    
DPWeekend        3.971e-02  1.054e-02   3.769 0.000895 ***
Subscribers     -2.142e-05  5.280e-06  -4.057 0.000428 ***
TierTier2       -9.673e-01  2.090e-01  -4.627 9.79e-05 ***
TierTier3       -1.341e+00  3.010e-01  -4.457 0.000152 ***
SubRange40-50K          NA         NA      NA       NA    
SubRange60-70K          NA         NA      NA       NA    
SubRange80-90K  -3.722e-01  9.075e-02  -4.101 0.000382 ***
SubRange90-100K -1.958e-01  3.001e-02  -6.524 7.82e-07 ***
Male.            9.721e-02  7.822e-02   1.243 0.225490    
Female.                 NA         NA      NA       NA    
Avg.Age                 NA         NA      NA       NA    
NewTierTier2            NA         NA      NA       NA    
NewTierTier3            NA         NA      NA       NA    
NewSubs                 NA         NA      NA       NA    
Avg.Income              NA         NA      NA       NA

As you can see Female. is one level. When I do it for another level of females I get this output

F2554<-read.csv("C:/Users/marissa.ferguson/Desktop/Unrated/F25-54.csv", header = T, sep = ",", na.strings = "?")
 female<-na.omit(F2554)
set.seed(1000)
train.size<-0.8
train.index<- sample.int(length(subfemale$DemoMedianRtg), round(length(subfemale$DemoMedianRtg)*train.size))
train.sample<-subfemale[train.index,]
test.sample<-subfemale[-train.index,]

Overall<-lm(DemoMedianRtg~ Qtr+DP+Subscribers+Tier+SubRange+Male.+Female.+Avg.Age+NewTier+NewSubs+Avg.Income, data=train.sample)

Coefficients: (39 not defined because of singularities)
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)      -2.022e+00  7.314e-01  -2.765 0.005911 ** 
Qtr2015 Q4        3.252e-02  1.602e-02   2.030 0.042898 *  
Qtr2016 Q1        1.529e-02  1.721e-02   0.888 0.374835    
Qtr2016 Q2        1.265e-02  2.170e-02   0.583 0.560224    
DPEarly Fringe    1.724e-02  1.842e-02   0.936 0.349910    
DPEarly Morning  -2.050e-03  3.006e-02  -0.068 0.945656    
DPLate Fringe     4.026e-02  2.070e-02   1.946 0.052270 .  
DPOvernight      -2.201e-02  2.603e-02  -0.846 0.398193    
DPPrimeTime       8.661e-02  1.909e-02   4.538 7.14e-06 ***
DPWeekend         4.809e-02  2.040e-02   2.358 0.018760 *  
Subscribers       1.638e-05  6.596e-06   2.483 0.013360 *  
TierTier2        -1.314e-01  1.155e-01  -1.138 0.255800    
TierTier3         6.884e-01  3.691e-01   1.865 0.062803 .  
SubRange110-120K -3.832e-01  1.126e-01  -3.402 0.000722 ***
SubRange30-40K    8.673e-01  2.440e-01   3.555 0.000414 ***
SubRange40-50K    1.229e-01  8.418e-02   1.460 0.144792    
SubRange50-60K           NA         NA      NA       NA    
SubRange60-70K    7.460e-01  2.604e-01   2.865 0.004347 ** 
SubRange70-80K   -9.732e-02  1.917e-01  -0.508 0.611830    
SubRange80-90K    2.315e-01  1.211e-01   1.912 0.056398 .  
SubRange90-100K   8.464e-02  6.387e-02   1.325 0.185729    
Male.0.24        -2.775e-01  6.075e-02  -4.567 6.25e-06 ***
Male.0.25        -5.249e-02  4.436e-02  -1.183 0.237277    
Male.0.29        -7.252e-02  6.412e-02  -1.131 0.258537    
Male.0.31         1.320e-01  5.029e-02   2.624 0.008956 ** 
Male.0.33         7.011e-02  1.235e-01   0.568 0.570611    
Male.0.34        -1.765e-01  6.584e-02  -2.682 0.007572 ** 
Male.0.35         1.708e-02  7.568e-02   0.226 0.821518    
Male.0.36        -9.714e-02  9.060e-02  -1.072 0.284170    
Male.0.37        -2.328e-01  8.924e-02  -2.609 0.009369 ** 
Male.0.39                NA         NA      NA       NA    
Male.0.4         -9.214e-02  8.566e-02  -1.076 0.282642    
Male.0.41        -1.629e-01  6.081e-02  -2.679 0.007631 ** 
Male.0.42        -2.339e-01  5.697e-02  -4.106 4.71e-05 ***
Male.0.44        -2.548e-01  6.245e-02  -4.079 5.26e-05 ***
Male.0.45        -9.951e-02  8.220e-02  -1.211 0.226632    
Male.0.46        -7.510e-02  1.079e-01  -0.696 0.486864    
Male.0.47        -4.412e-02  5.739e-02  -0.769 0.442357    
Male.0.48        -2.330e-01  6.124e-02  -3.805 0.000159 ***
Male.0.5         -2.787e-01  7.532e-02  -3.700 0.000240 ***
Male.0.51        -2.241e-01  6.402e-02  -3.501 0.000506 ***
Male.0.52        -1.690e-01  5.705e-02  -2.962 0.003203 ** 
Male.0.53        -1.254e-01  8.424e-02  -1.489 0.137125    
Male.0.55        -2.398e-01  5.648e-02  -4.246 2.60e-05 ***
Male.0.56        -3.289e-01  9.453e-02  -3.479 0.000548 ***
Male.0.57        -1.017e-01  8.750e-02  -1.162 0.245633    
Male.0.58        -3.471e-01  8.574e-02  -4.048 6.00e-05 ***
Male.0.59        -3.855e-01  1.067e-01  -3.612 0.000335 ***
Male.0.6          6.813e-01  1.378e-01   4.943 1.06e-06 ***
Male.0.62        -2.782e-01  7.264e-02  -3.830 0.000144 ***
Male.0.63         4.929e-01  1.042e-01   4.730 2.94e-06 ***
Male.0.64                NA         NA      NA       NA    
Male.0.65         4.818e-01  8.043e-02   5.990 4.04e-09 ***
Male.0.66        -4.060e-01  1.459e-01  -2.783 0.005591 ** 
Male.0.71        -2.394e-03  1.239e-01  -0.019 0.984588    
Male.0.76        -1.261e-01  9.176e-02  -1.374 0.169912    
Male.n/a         -1.728e-01  6.243e-02  -2.768 0.005846 ** 
Female.0.29              NA         NA      NA       NA    
Female.0.34              NA         NA      NA       NA    
Female.0.35              NA         NA      NA       NA    
Female.0.36              NA         NA      NA       NA    
Female.0.37              NA         NA      NA       NA    
Female.0.38              NA         NA      NA       NA    
Female.0.4               NA         NA      NA       NA    
Female.0.41              NA         NA      NA       NA    
Female.0.42              NA         NA      NA       NA    
Female.0.43              NA         NA      NA       NA    
Female.0.44              NA         NA      NA       NA    
Female.0.45              NA         NA      NA       NA    
Female.0.47              NA         NA      NA       NA    
Female.0.48              NA         NA      NA       NA    
Female.0.49              NA         NA      NA       NA    
Female.0.5               NA         NA      NA       NA    
Female.0.52              NA         NA      NA       NA    
Female.0.53              NA         NA      NA       NA    
Female.0.54              NA         NA      NA       NA    
Female.0.55              NA         NA      NA       NA    
Female.0.56              NA         NA      NA       NA    
Female.0.58              NA         NA      NA       NA    
Female.0.59              NA         NA      NA       NA    
Female.0.6               NA         NA      NA       NA    
Female.0.61              NA         NA      NA       NA    
Female.0.63              NA         NA      NA       NA    
Female.0.65              NA         NA      NA       NA    
Female.0.66              NA         NA      NA       NA    
Female.0.67              NA         NA      NA       NA    
Female.0.69              NA         NA      NA       NA    
Female.0.71              NA         NA      NA       NA    
Female.0.75              NA         NA      NA       NA    
Female.0.76              NA         NA      NA       NA    
Female.0.77              NA         NA      NA       NA    
Female.n/a               NA         NA      NA       NA    
Avg.Age          -3.740e-03  1.244e-03  -3.006 0.002785 ** 
NewTierTier2      2.158e-01  1.048e-01   2.059 0.040012 *  
NewTierTier3      4.357e-01  1.294e-01   3.367 0.000819 ***
NewTierTier4             NA         NA      NA       NA    
NewSubs           1.056e-05  2.873e-06   3.677 0.000262 ***
Avg.Income       -3.272e-06  1.437e-06  -2.277 0.023193 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

I am confused on why Female. and Male. are both different when I have copied and pasted the code and only changed the csv I am using for them. Any help would be appreciated. Is there a way to keep it at one variable? Or are the multiple levels necessary?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Your lm() commands both call train.sample, which is not defined anywhere in your code. Please post a reproducible example to get better help. $\endgroup$ – mkt - Reinstate Monica Aug 10 '16 at 18:21
  • 3
    $\begingroup$ Check your data. read.csv reads in character columns as factors by default. Factors create different levels in lm. $\endgroup$ – phiver Aug 10 '16 at 18:25
  • $\begingroup$ @phiver the only thing is in my data set they are decimal points, not characters $\endgroup$ – Marissa Aug 10 '16 at 23:54
1
$\begingroup$

The output is giving you a very strong hint:

Coefficients: (8 not defined because of singularities)

Without a reproducible example it is impossible to be sure, but the problems you have are very likely caused by including variables that are derived from others. For example, you can't have a binary variable male which is 1 if the subject is male and 0 if female, and another variable female which is 1 if the subject female and 0 if the subject is male because these are perfectly collinear - you only need one of them. Similarly, you can't derive the mean of several variables, and then include those variables and the mean variable.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Does it matter that Female. and Male. are percentages? They are still highly correlated and I take one out of the regression, but even when i do that they are still not factors in the first $\endgroup$ – Marissa Aug 10 '16 at 22:57
  • $\begingroup$ If male is 24% then female is 76% right ? If you know the male % then you know the female %, so it's exactly the same problem. I don't know what you mean about factors - if you want R to treat a variable as a factor, then you use as.factor(variable) $\endgroup$ – Robert Long Aug 11 '16 at 7:49
  • $\begingroup$ I have gotten rid of one of the variables but it keeps making the variables the same, itll have male like the first output where male is one variable in the regression, but then it will have male with a number by it(factoring, im guessing) for another data set. I want them to both be one variable in the regression. $\endgroup$ – Marissa Aug 11 '16 at 15:42
  • $\begingroup$ @Marissa that is because the variables are different in the other dataset. Please post the result of str(dataset1) and str(dataset2) into the question $\endgroup$ – Robert Long Aug 11 '16 at 16:05
0
$\begingroup$

As @phiver said, check if your variable is a factor. When a variable is a factor, the lm, glm and gnm functions consider it as a categorical variable.

Here is an example

age <- 1:10
age.factor <- as.factor(1:10)
y <- age + rnorm(10)

lm(y~age) # age as one variable, so age * b

lm(y~age.factor) # age as a categorical variable
# so age1 is a coefficient, age2 is a coefficient, ...
# or 
lm(y~0+age.factor) # to remove the intercept

is.factor(age) # FALSE
is.factor(age.factor) #TRUE
|cite|improve this answer
$\endgroup$
  • $\begingroup$ It isn't a factor for the first part, but is for the second part. I want them to be the same, so how can I do that? $\endgroup$ – Marissa Aug 10 '16 at 22:03
  • $\begingroup$ From what I understand from your code excerpt, try train.sample$Female. <- as.factor(train.sample$Female.) to have categorical variable. Or, to get rid of categorical variable, try something like as.numeric(levels(aFactorVariable)) then assign it to your sample. $\endgroup$ – Étienne Vanasse Aug 10 '16 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.