How to interpret regression function with categorical variable?

Question

I am trying to figure out how to interpret a regression function with no intercept and one categorical variable performed on a survey data. Each participant marks which actions, from a list of 25, they perceived as crimes. The survey data collects the age, sex, the year in college and income level of the participant.

$$crime = 0.38x_{age} - 10.3x_{female}I_{sex} - 8.01x_{male}I_{male} + 0.18x_{college} + 0.29x_{income}$$

Are the following interpretations of regression coefficients correct?

• $\beta_{age}$ interpretation: This coefficient estimates 0.38 increase in crime score for each additional year of age of the participant, holding other variables constant.
• $\beta_{female}$ interpretation: This coefficient estimates 10.3 decrease in crime score if the survey participant is female, holding other variables constant.

---

Here is the R code that generated my model,

reg_model <- lm(crimes ~ 0 + age + sex + college 
                + income , data = crime_data)

From the R formula documentation it says "It can also used to remove the intercept term: when fitting a linear model y ~ x - 1 specifies a line through the origin. A model with no intercept can be also specified as y ~ x + 0 or y ~ 0 + x."

If I try to build the model with an intercept using the following R code,

reg_model <- lm(crimes ~ age + sex + college + 
                income , data = crime_data)

I get the following model, $$crime = - 10.3 + 0.38x_{age} + 2.29x_{male}I_{male} + 0.18x_{college} + 0.29x_{income}$$

I thought it would feel nicer if I can distinctly say how much male and female affects the crime score. If I take out the 0 from the formula, I only have one sex in my model. I was not sure how to explain the effect of the missing sex on the crime score.

---

Here is the question I am trying to solve, Link to csv dataset: https://pastebin.com/eJJqUfmr

Source: Freund, R.J. Wilson, W.J., and D. L. Mohr (2010). Statistical Methods, 3rd Edition, Academic Press. ISBN-13: 978-0123749703; Chapter 8 : Multiple Linear Regression; Exercise problem 11

Answer 1

Note that your initial model uses level means coding.

The answers to your explicit questions are:

1. Yes, if age goes up by $1$ (year, I assume), the crime score is predicted to increase by $0.38$, holding all other covariates equal.
2. No (not exactly), being female does not decrease the crime score by 10.3 if the survey participant is female holding other variables constant. Instead, if all other variables are exactly equal to $0$ (e.g., the person is in the process of being born), the predicted crime score for a female is $-10.3$. Whether that's particularly meaningful is a different issue—it's just part of the linear model.

---

A concern on this thread (e.g., in the comments) is whether the two listed models are the same. The two models are indeed identical. Because there is a categorical variable here, suppressing the intercept just changes the meanings of the intercept and the difference between the levels to the predicted means for the two individual levels when all other variables are $0$. It may help to read my answer here: How can logistic regression have a factorial predictor and no intercept? Note that the coefficients for age, college, and income are identical between the two models. The difference is the last two estimated coefficients. In the first, female = -10.3 & male = -8.01; in the second, the intercept is -10.3, & the difference between male and female is 2.29. These yield the same predicted values for all combinations of predictor values.

Answer 2

What precisely is implied by omitting the intercept?

@gung and @kjetil have explained in their answers how for this kind of model, with a factor variable included as a predictor, when asked to omit the intercept R, meaning lm(), changes its parameterisation. The model fitted is really another version of the same model. How well this is documented and how widely it is understood are separate issues.

Here I first give the data in a form suitable for reading into Stata, not because I assume that other readers use Stata, but because I guess a copy here might be as useful as the version on pastebin, which in any case may not remain accessible indefinitely. They are close to a form that could be read into many different programs with minor editing.

clear
input byte(age sex college income crimes)
19 0 2 56 13
19 1 2 59 16
20 0 2 55 13
21 0 2 60 13
20 0 2 52 14
24 0 3 54 14
25 0 3 55 13
25 0 3 59 16
27 1 4 56 16
28 1 4 52 14
38 0 4 59 20
29 1 4 63 25
30 1 4 55 19
21 1 3 29  8
21 1 2 35 11
20 0 2 33 10
19 0 2 27  6
21 0 3 24  7
21 1 2 53 15
16 1 2 63 23
18 1 2 72 25
18 1 2 75 22
18 0 2 61 16
19 1 2 65 19
19 1 2 70 19
20 1 2 78 18
19 0 2 76 16
18 0 2 53 12
31 0 4 59 23
32 1 4 62 25
32 1 4 55 22
31 0 4 57 25
30 1 4 46 17
29 0 4 35 14
29 0 4 32 12
28 0 4 30 10
27 0 4 29  8
26 0 4 28  7
25 0 4 25  5
24 0 3 33  9
23 0 3 26  7
23 1 3 28  9
22 0 3 38 10
22 0 3 24  4
22 0 3 28  6
end

Now to the heart of the matter. I note that here there is one indicator for sex which from the magnitude of the coefficient just below I infer to be the same as male which the OP doen't give.

Unsurprisingly, Stata can reproduce a plain regression with intercept.

. regress crimes income age sex college

      Source |       SS           df       MS      Number of obs   =        45
-------------+----------------------------------   F(4, 40)        =     52.26
       Model |   1319.7758         4  329.943951   Prob > F        =    0.0000
    Residual |  252.535307        40  6.31338267   R-squared       =    0.8394
-------------+----------------------------------   Adj R-squared   =    0.8233
       Total |  1572.31111        44  35.7343434   Root MSE        =    2.5126

------------------------------------------------------------------------------
      crimes |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      income |   .2933889   .0302029     9.71   0.000     .2323465    .3544313
         age |   .3777469   .1974061     1.91   0.063    -.0212256    .7767194
         sex |   2.293912   .8495281     2.70   0.010     .5769517    4.010872
     college |   .1793265    1.18586     0.15   0.881    -2.217386    2.576039
       _cons |  -10.30475   2.277652    -4.52   0.000    -14.90805   -5.701441
------------------------------------------------------------------------------

So, this is a just a model with systematic part

$\text{crimes} = \beta_0 + \beta_1 \text{income} + \beta_2 \text{age} + \beta_3 \text{sex} + \beta_4 \text{college}$

Here $\text{sex}$ is 0 and 1 and need not be declared as a factor variable in Stata (although that would do no harm). I note that $\text{college}$ is treated as measured, not as a categorical variable, as in the question.

The quotation given by the OP

It can also used to remove the intercept term: when fitting a linear model y ~ x - 1 specifies a line through the origin. A model with no intercept can be also specified as y ~ x + 0 or y ~ 0 + x.

led me -- in comments now deleted -- to suppose that R would fit the following model, with systematic part

$\text{crimes} = \beta_1 \text{income} + \beta_2 \text{age} + \beta_3 \text{sex} + \beta_4 \text{college}$

As said, that is not what R does. But it is an interpretation of what omitting the intercept means. It's not, in my view, a defensible model for these data. There are no grounds for postulating a model fit that goes through the origin, which here as often is not only way outside the data but also impossible as a data point (no student with age 0, for a start).

That's nevertheless easy to do in Stata. Here for what they are worth (nothing) are the results:

. regress crimes income age sex college, noconstant

      Source |       SS           df       MS      Number of obs   =        45
-------------+----------------------------------   F(4, 41)        =    280.95
       Model |  10464.2351         4  2616.05876   Prob > F        =    0.0000
    Residual |  381.764941        41  9.31134003   R-squared       =    0.9648
-------------+----------------------------------   Adj R-squared   =    0.9614
       Total |       10846        45  241.022222   Root MSE        =    3.0514

------------------------------------------------------------------------------
      crimes |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      income |   .2312656   .0326698     7.08   0.000     .1652876    .2972437
         age |   .0497223   .2229822     0.22   0.825    -.4005994     .500044
         sex |   2.534587   1.029674     2.46   0.018      .455119    4.614056
     college |   .4111346   1.438808     0.29   0.777    -2.494596    3.316865
------------------------------------------------------------------------------

To spell it out, this is a quite different model with one fewer parameter and different coefficient estimates. As before sex is 0 and 1 and need not be declared as a factor variable in Stata (although that would do no harm).

I confirm that the same numerical results are obtained in Stata in either case if sex is declared as a factor variable.

By the way, nothing of interest really follows from the difference in $R^2$ here. The first $R^2$ measures how much better the model is than predicting the mean of crimes; the second measures how much better the model is than predicting zero crimes, which I can't see as a benchmark of statistical or social science or policy interest.

That this is possible in Stata is not a bug or bogus. The main reason for the option noconstant in regress in Stata is the same main reason as explained in the quotation above from R documentation. Occasionally there are good grounds for forcing a model fit through the origin. The most common example in my experience is fitting $y = bx$ rather than $y = a + bx$ (after all many elementary laws of physics are of the first kind).

Discussing what Stata does is here just a way of checking results and of underlining that different interpretations of omitting the intercept exist. What other software would or could do by way of omitting the intercept is equally germane, in my view.

Dataset given in the example: How to analyse? Does it even make sense?

The OP gave their dataset. I now make some partial remarks on the different but interesting question of how to analyse their data. It is evident that income is the most important linear predictor: see the scatter plot below.

The response is bounded [0, 25].
The upper limit for response of 25 is attained in the data, so a linear functional form appears less suitable than say fractional logit.

I scale to [0, 1], fit a logit model and plot a model summary in terms of the original scale.

. gen crimes2 = crimes/25

. fracreg logit crimes2 income age i.sex college

Iteration 0:   log pseudolikelihood = -29.235193  
Iteration 1:   log pseudolikelihood = -26.141125  
Iteration 2:   log pseudolikelihood =  -26.12196  
Iteration 3:   log pseudolikelihood = -26.121956  
Iteration 4:   log pseudolikelihood = -26.121956  

Fractional logistic regression                  Number of obs     =         45
                                                Wald chi2(4)      =     146.02
                                                Prob > chi2       =     0.0000
Log pseudolikelihood = -26.121956               Pseudo R2         =     0.1490

------------------------------------------------------------------------------
             |               Robust
     crimes2 |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      income |   .0528822   .0065389     8.09   0.000     .0400662    .0656982
         age |   .0786006   .0469061     1.68   0.094    -.0133338    .1705349
       1.sex |    .456912   .1565497     2.92   0.004     .1500802    .7637437
     college |   .0104632   .2507818     0.04   0.967    -.4810602    .5019865
       _cons |   -4.28356   .5641885    -7.59   0.000    -5.389349   -3.177771
------------------------------------------------------------------------------

Although in principle quite different, the indications about predictors seem similar. income is the biggest deal followed by sex.

Yet further analysis might start by noting the correlation between age and college, which takes the values 2, 3, 4. Closer scrutiny reveals a puzzling pattern, here shown by a plot with stem-and-leaf flavour:

On average I would expect people to be about 1 or 2 years older for each year in college, taking note of part-timers, etc. Here the medians are 19, 23, 29 years!

I suspect that the data are invented on this and other grounds: is it plausible that 45 students know their parental income as implied?

Answer 3

This answer is mostly an extended comment to @Nick Cox, focusing on the evident differences in treatment of linear models without intercept an R and Stata (based on the outputs from the crimes data used here).

R: When there is a factor variable (sex in the example) in the model formula, the effect of omitting the intercept is a reparametrization of sex: With the intercept, sex is represented only with one of its dummies (1 df), while with intercept omitted, sex keeps its two dummies (2 df.) The sum of the two dummies is the constant vector with only ones, since exactly one of the dummies is 1, the other is zero. That is exactly equal to how the intercept is coded, as a vector of only ones. So, if the two dummies and the intercept were kept, there would be an exact colinearity. Algebraically (linear algebra from nursing school, as I saw someone else refer to it,) this means that the column (or rank) space of the two model matrices are identical, so the models are identical.

Let us look at the R output:

summary(mod_noincpt)

Call:
lm(formula = crimes ~ 0 + age + factor(sex) 
     + college + income, data = mycrime)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.7871 -1.1422 -0.4302  1.1349  6.1541 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
age            0.3777     0.1974   1.914  0.06285 .  
factor(sex)0 -10.3048     2.2776  -4.524 5.32e-05 ***
factor(sex)1  -8.0108     2.4803  -3.230  0.00248 ** 
college        0.1793     1.1859   0.151  0.88056    
income         0.2934     0.0302   9.714 4.43e-12 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.513 on 40 degrees of freedom
Multiple R-squared:  0.9767,    Adjusted R-squared:  0.9738 
F-statistic: 335.6 on 5 and 40 DF,  p-value: < 2.2e-16

summary(mod_incpt)

Call:
lm(formula = crimes ~ age + factor(sex) + 
      college + income, data = mycrime)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.7871 -1.1422 -0.4302  1.1349  6.1541 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -10.3048     2.2776  -4.524 5.32e-05 ***
age            0.3777     0.1974   1.914   0.0629 .  
factor(sex)1   2.2939     0.8495   2.700   0.0101 *  
college        0.1793     1.1859   0.151   0.8806    
income         0.2934     0.0302   9.714 4.43e-12 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.513 on 40 degrees of freedom
Multiple R-squared:  0.8394,    Adjusted R-squared:  0.8233 
F-statistic: 52.26 on 4 and 40 DF,  p-value: 2.322e-15

The coefficient estimates and standard errors are identical, only labelled differently, the two corresponding lines are:

factor(sex)0 -10.3048     2.2776  -4.524 5.32e-05 ***
(Intercept)  -10.3048     2.2776  -4.524 5.32e-05 ***

The other lines are also labelled identically. There is one other difference, the value of R-squared. Repeating the relevant parts (in same order, without intercept first):

Multiple R-squared:  0.9767,    Adjusted R-squared:  0.9738 
Multiple R-squared:  0.8394,    Adjusted R-squared:  0.8233 

This could be considered an error in the R implementation, the reason is probably that is it difficult to verify algorithmically if a model without an explicit intercept contains a constant vector in the model space (column space of model matrix), so it is not attempted. But note that the F-tests are identical, with identical dfs.

Stata. I will refer to output in Nick's answer. The important points to consider is: For both the with-intercept and without-intercept models, sex are represented with only 1 df. That means that sex is wrongly represented in the without-intercept model, and the estimated model is completely bogus, nonsensical. Indeed this model treats half of the sex factor as error (that is, one of two dfs). Should be reported as an error to Stata. The reason behind leaving out one dummy for factors is that the left-out dummy is implicitly represented via the intercept: the left-out dummy is the intercept (vector of ones) minus sum of kept dummies. Without an intercept in the model (implicit or explicit) this logic simply fails. Stata output, in this case, is bogus.

Then the F-tests: They are different, just compare the df for residual, which are different. The F-test given for the without-intercept model is meaningless, since the error sum of squares used includes a part of the sex factor, as explained.

Answer 4

is crime a continuous variable ? (i guess yes but want to be sure).

Yes your interpretation is correct and you can do the same for your dummary variable, without the growth aspect.

Being a female diminish more the crime than being a man if you want even to make a comparison that is more appropriate when you have categorical.

Answer 5

The category variable coefficients indicate how much your prediction is incremented if a given individual belongs to the corresponding category.

What strikes me is that you have two different coefficients for a binary variable. Normal practice is to take one of the groups as reference, with the coefficient indicating the variation that changing to the other group produces.

NOTE: As it stands right now, on the after-edit added second model, being male increases "crime"'s prediction by 2.29 units (all other variables staying equal)