# Why is R Changing my Categorical Variable into one Variable for the Regression

So I am working on this regression and I am doing it for multiple levels of Females. I have done one regression for Females 18-34 and got this output

F18<-read.csv("C:/Users/marissa.ferguson/Desktop/Unrated/F18-34.csv", header = T, sep = ",", na.strings = "?")
female<-na.omit(F18)
set.seed(1000)
train.size<-0.8
train.index<- sample.int(length(subfemale$DemoMedianRtg), round(length(subfemale$DemoMedianRtg)*train.size))
train.sample<-subfemale[train.index,]
test.sample<-subfemale[-train.index,]

> Overall<-lm(DemoMedianRtg~ DP+Subscribers+Tier+SubRange+Male.+Female.+Avg.Age+NewTier+NewSubs+Avg.Income, data=train.sample)
> summary(Overall)

Call:
lm(formula = DemoMedianRtg ~ DP + Subscribers + Tier + SubRange +
Male. + Female. + Avg.Age + NewTier + NewSubs + Avg.Income,
data = train.sample)

Residuals:
Min        1Q    Median        3Q       Max
-0.038269 -0.014386  0.003568  0.012190  0.029538

Coefficients: (8 not defined because of singularities)
Estimate Std. Error t value Pr(>|t|)
(Intercept)      2.258e+00  5.495e-01   4.108 0.000375 ***
DPEarly Fringe   8.178e-03  1.129e-02   0.724 0.475665
DPEarly Morning  1.012e-02  1.739e-02   0.582 0.566011
DPLate Fringe    7.304e-02  1.643e-02   4.445 0.000157 ***
DPOvernight      3.327e-02  1.968e-02   1.691 0.103281
DPPrimeTime      1.999e-02  1.188e-02   1.682 0.105005
DPWeekend        3.971e-02  1.054e-02   3.769 0.000895 ***
Subscribers     -2.142e-05  5.280e-06  -4.057 0.000428 ***
TierTier2       -9.673e-01  2.090e-01  -4.627 9.79e-05 ***
TierTier3       -1.341e+00  3.010e-01  -4.457 0.000152 ***
SubRange40-50K          NA         NA      NA       NA
SubRange60-70K          NA         NA      NA       NA
SubRange80-90K  -3.722e-01  9.075e-02  -4.101 0.000382 ***
SubRange90-100K -1.958e-01  3.001e-02  -6.524 7.82e-07 ***
Male.            9.721e-02  7.822e-02   1.243 0.225490
Female.                 NA         NA      NA       NA
Avg.Age                 NA         NA      NA       NA
NewTierTier2            NA         NA      NA       NA
NewTierTier3            NA         NA      NA       NA
NewSubs                 NA         NA      NA       NA
Avg.Income              NA         NA      NA       NA


As you can see Female. is one level. When I do it for another level of females I get this output

F2554<-read.csv("C:/Users/marissa.ferguson/Desktop/Unrated/F25-54.csv", header = T, sep = ",", na.strings = "?")
female<-na.omit(F2554)
set.seed(1000)
train.size<-0.8
train.index<- sample.int(length(subfemale$DemoMedianRtg), round(length(subfemale$DemoMedianRtg)*train.size))
train.sample<-subfemale[train.index,]
test.sample<-subfemale[-train.index,]

Overall<-lm(DemoMedianRtg~ Qtr+DP+Subscribers+Tier+SubRange+Male.+Female.+Avg.Age+NewTier+NewSubs+Avg.Income, data=train.sample)

Coefficients: (39 not defined because of singularities)
Estimate Std. Error t value Pr(>|t|)
(Intercept)      -2.022e+00  7.314e-01  -2.765 0.005911 **
Qtr2015 Q4        3.252e-02  1.602e-02   2.030 0.042898 *
Qtr2016 Q1        1.529e-02  1.721e-02   0.888 0.374835
Qtr2016 Q2        1.265e-02  2.170e-02   0.583 0.560224
DPEarly Fringe    1.724e-02  1.842e-02   0.936 0.349910
DPEarly Morning  -2.050e-03  3.006e-02  -0.068 0.945656
DPLate Fringe     4.026e-02  2.070e-02   1.946 0.052270 .
DPOvernight      -2.201e-02  2.603e-02  -0.846 0.398193
DPPrimeTime       8.661e-02  1.909e-02   4.538 7.14e-06 ***
DPWeekend         4.809e-02  2.040e-02   2.358 0.018760 *
Subscribers       1.638e-05  6.596e-06   2.483 0.013360 *
TierTier2        -1.314e-01  1.155e-01  -1.138 0.255800
TierTier3         6.884e-01  3.691e-01   1.865 0.062803 .
SubRange110-120K -3.832e-01  1.126e-01  -3.402 0.000722 ***
SubRange30-40K    8.673e-01  2.440e-01   3.555 0.000414 ***
SubRange40-50K    1.229e-01  8.418e-02   1.460 0.144792
SubRange50-60K           NA         NA      NA       NA
SubRange60-70K    7.460e-01  2.604e-01   2.865 0.004347 **
SubRange70-80K   -9.732e-02  1.917e-01  -0.508 0.611830
SubRange80-90K    2.315e-01  1.211e-01   1.912 0.056398 .
SubRange90-100K   8.464e-02  6.387e-02   1.325 0.185729
Male.0.24        -2.775e-01  6.075e-02  -4.567 6.25e-06 ***
Male.0.25        -5.249e-02  4.436e-02  -1.183 0.237277
Male.0.29        -7.252e-02  6.412e-02  -1.131 0.258537
Male.0.31         1.320e-01  5.029e-02   2.624 0.008956 **
Male.0.33         7.011e-02  1.235e-01   0.568 0.570611
Male.0.34        -1.765e-01  6.584e-02  -2.682 0.007572 **
Male.0.35         1.708e-02  7.568e-02   0.226 0.821518
Male.0.36        -9.714e-02  9.060e-02  -1.072 0.284170
Male.0.37        -2.328e-01  8.924e-02  -2.609 0.009369 **
Male.0.39                NA         NA      NA       NA
Male.0.4         -9.214e-02  8.566e-02  -1.076 0.282642
Male.0.41        -1.629e-01  6.081e-02  -2.679 0.007631 **
Male.0.42        -2.339e-01  5.697e-02  -4.106 4.71e-05 ***
Male.0.44        -2.548e-01  6.245e-02  -4.079 5.26e-05 ***
Male.0.45        -9.951e-02  8.220e-02  -1.211 0.226632
Male.0.46        -7.510e-02  1.079e-01  -0.696 0.486864
Male.0.47        -4.412e-02  5.739e-02  -0.769 0.442357
Male.0.48        -2.330e-01  6.124e-02  -3.805 0.000159 ***
Male.0.5         -2.787e-01  7.532e-02  -3.700 0.000240 ***
Male.0.51        -2.241e-01  6.402e-02  -3.501 0.000506 ***
Male.0.52        -1.690e-01  5.705e-02  -2.962 0.003203 **
Male.0.53        -1.254e-01  8.424e-02  -1.489 0.137125
Male.0.55        -2.398e-01  5.648e-02  -4.246 2.60e-05 ***
Male.0.56        -3.289e-01  9.453e-02  -3.479 0.000548 ***
Male.0.57        -1.017e-01  8.750e-02  -1.162 0.245633
Male.0.58        -3.471e-01  8.574e-02  -4.048 6.00e-05 ***
Male.0.59        -3.855e-01  1.067e-01  -3.612 0.000335 ***
Male.0.6          6.813e-01  1.378e-01   4.943 1.06e-06 ***
Male.0.62        -2.782e-01  7.264e-02  -3.830 0.000144 ***
Male.0.63         4.929e-01  1.042e-01   4.730 2.94e-06 ***
Male.0.64                NA         NA      NA       NA
Male.0.65         4.818e-01  8.043e-02   5.990 4.04e-09 ***
Male.0.66        -4.060e-01  1.459e-01  -2.783 0.005591 **
Male.0.71        -2.394e-03  1.239e-01  -0.019 0.984588
Male.0.76        -1.261e-01  9.176e-02  -1.374 0.169912
Male.n/a         -1.728e-01  6.243e-02  -2.768 0.005846 **
Female.0.29              NA         NA      NA       NA
Female.0.34              NA         NA      NA       NA
Female.0.35              NA         NA      NA       NA
Female.0.36              NA         NA      NA       NA
Female.0.37              NA         NA      NA       NA
Female.0.38              NA         NA      NA       NA
Female.0.4               NA         NA      NA       NA
Female.0.41              NA         NA      NA       NA
Female.0.42              NA         NA      NA       NA
Female.0.43              NA         NA      NA       NA
Female.0.44              NA         NA      NA       NA
Female.0.45              NA         NA      NA       NA
Female.0.47              NA         NA      NA       NA
Female.0.48              NA         NA      NA       NA
Female.0.49              NA         NA      NA       NA
Female.0.5               NA         NA      NA       NA
Female.0.52              NA         NA      NA       NA
Female.0.53              NA         NA      NA       NA
Female.0.54              NA         NA      NA       NA
Female.0.55              NA         NA      NA       NA
Female.0.56              NA         NA      NA       NA
Female.0.58              NA         NA      NA       NA
Female.0.59              NA         NA      NA       NA
Female.0.6               NA         NA      NA       NA
Female.0.61              NA         NA      NA       NA
Female.0.63              NA         NA      NA       NA
Female.0.65              NA         NA      NA       NA
Female.0.66              NA         NA      NA       NA
Female.0.67              NA         NA      NA       NA
Female.0.69              NA         NA      NA       NA
Female.0.71              NA         NA      NA       NA
Female.0.75              NA         NA      NA       NA
Female.0.76              NA         NA      NA       NA
Female.0.77              NA         NA      NA       NA
Female.n/a               NA         NA      NA       NA
Avg.Age          -3.740e-03  1.244e-03  -3.006 0.002785 **
NewTierTier2      2.158e-01  1.048e-01   2.059 0.040012 *
NewTierTier3      4.357e-01  1.294e-01   3.367 0.000819 ***
NewTierTier4             NA         NA      NA       NA
NewSubs           1.056e-05  2.873e-06   3.677 0.000262 ***
Avg.Income       -3.272e-06  1.437e-06  -2.277 0.023193 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


I am confused on why Female. and Male. are both different when I have copied and pasted the code and only changed the csv I am using for them. Any help would be appreciated. Is there a way to keep it at one variable? Or are the multiple levels necessary?

• Your lm() commands both call train.sample, which is not defined anywhere in your code. Please post a reproducible example to get better help.
– mkt
Aug 10, 2016 at 18:21
• Check your data. read.csv reads in character columns as factors by default. Factors create different levels in lm. Aug 10, 2016 at 18:25
• @phiver the only thing is in my data set they are decimal points, not characters Aug 10, 2016 at 23:54

As @phiver said, check if your variable is a factor. When a variable is a factor, the lm, glm and gnm functions consider it as a categorical variable.

Here is an example

age <- 1:10
age.factor <- as.factor(1:10)
y <- age + rnorm(10)

lm(y~age) # age as one variable, so age * b

lm(y~age.factor) # age as a categorical variable
# so age1 is a coefficient, age2 is a coefficient, ...
# or
lm(y~0+age.factor) # to remove the intercept

is.factor(age) # FALSE
is.factor(age.factor) #TRUE

• It isn't a factor for the first part, but is for the second part. I want them to be the same, so how can I do that? Aug 10, 2016 at 22:03
• From what I understand from your code excerpt, try train.sample$Female. <- as.factor(train.sample$Female.) to have categorical variable. Or, to get rid of categorical variable, try something like as.numeric(levels(aFactorVariable)) then assign it to your sample. Aug 10, 2016 at 23:06

The output is giving you a very strong hint:

Coefficients: (8 not defined because of singularities)

Without a reproducible example it is impossible to be sure, but the problems you have are very likely caused by including variables that are derived from others. For example, you can't have a binary variable male which is 1 if the subject is male and 0 if female, and another variable female which is 1 if the subject female and 0 if the subject is male because these are perfectly collinear - you only need one of them. Similarly, you can't derive the mean of several variables, and then include those variables and the mean variable.

• Does it matter that Female. and Male. are percentages? They are still highly correlated and I take one out of the regression, but even when i do that they are still not factors in the first Aug 10, 2016 at 22:57
• If male is 24% then female is 76% right ? If you know the male % then you know the female %, so it's exactly the same problem. I don't know what you mean about factors - if you want R to treat a variable as a factor, then you use as.factor(variable) Aug 11, 2016 at 7:49
• I have gotten rid of one of the variables but it keeps making the variables the same, itll have male like the first output where male is one variable in the regression, but then it will have male with a number by it(factoring, im guessing) for another data set. I want them to both be one variable in the regression. Aug 11, 2016 at 15:42
• @Marissa that is because the variables are different in the other dataset. Please post the result of str(dataset1) and str(dataset2) into the question Aug 11, 2016 at 16:05