3
$\begingroup$

I'm trying to analyze two negatively-correlated variables, A and B (where A is the independent variable) while somehow taking into account a categorical variable C, with the intention of highlighting data that deviates above expected values.

For example, in the following subset of my data:

#, A, B, C
1, 14, 55, "X"
2, 12, 75, "X"
3, 10, 65, "X"
4, 14, 40, "Y"
5, 12, 30, "Y"
6, 10, 35, "Y"

Average:
A, B
14, 55
12, 60
10, 65    

I'd like to be able to highlight data point 2 because it deviates above the average value, but I'd also like to highlight data point 4, because although it deviates below the average value, it deviates above the expected value within its category.

I know how to do a simple linear regression on A and B, but I don't know how to account for the categorical variable.

$\endgroup$
3
$\begingroup$

Based on your example provided, the qualitative variable "C" only has two levels, or possible values, we can incorporate it into a regression model by creating an indicator or dummy variable that takes on two possible numerical values.

Having that, $x_{1} = A, x_{2} = B$, and $x_{3} = C$, for linear regression, $$y = \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \beta_{3}x_{3} + \epsilon_{i}$$ $$\therefore y = \beta_{0} + \beta_{1}A + \beta_{2}B + \beta_{3}C + \epsilon_{i}$$

From the variable "C", we can create a new variable that takes the form,

$$\ C = \left\{ \begin{array}{l l} 1 & \quad \text{if $ith$ is X}\\ 0 & \quad \text{if $ith$ is Y} \end{array} \right.\\$$

and use this variable as a predictor in the regression equation. This results in the model

$$\ y_{i} = \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \beta_{3}x_{3} + \epsilon_{i} = \left\{ \begin{array}{l l} \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \beta_{3}x_{3} + \epsilon_{i} & \quad \text{if $ith$ is X}\\ \beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \epsilon_{i} & \quad \text{if $ith$ is Y} \end{array} \right.\\$$

Now $\beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2}$ can be interpreted as Y, while $\beta_{0} + \beta_{1}x_{1} + \beta_{2}x_{2} + \beta_{3}x_{3}$ is interpreted as X.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Using R you can just fit a linear model, such as fit <- lm(A ~ B + C, data = your_data_name). This will produce two intercepts, one for each level ("X" and "Y") of the factor "C".

From there you can perform diagnostics by simply invoking plot(fit), or with specific commands, such as dfbeta(fit).

EDIT: Following is an example of OLS regression calculated both with linear algebra and with the R built-in functions to kind of avoid leaving any stoned unturned...

Scenario: We want to predict the money spent in self-indulgence in one year (don't ask...) in $US based on two parameters: demographics (categorical or factor variable with levels "ch" for child; "f" for female; and "m" for male) and height (in cm). I know... silly... But I'll stay out of controversy... Here are the vectors to code the y variable (yearly expenditures) and to code for the demographics dummy variable:

y <- c(50, 60, 800, 1000, 1800, 2000)
ch <- c(1, 1, 0, 0, 0, 0)  # Selecting the children
f <- c(0, 0, 1, 1, 0, 0)   # Selecting the women
m <- c(0, 0, 0, 0, 1, 1)   # Selecting the men

Let's construct the design matrix A:

A <- cbind(ch, f, m)

It looks like this:

     ch f m
[1,]  1 0 0
[2,]  1 0 0
[3,]  0 1 0
[4,]  0 1 0
[5,]  0 0 1
[6,]  0 0 1

And let's calculate the OLS model, remembering the formula:

$\hat{\beta}=(A^{T}A)^{-1}A^{T}y$

solve(t(A) %*% A) %*% t(A) %*% y
   [,1]
ch   55
f   900
m  1900

Exactly the average expenditure for each demographic group! Makes sense...

Using the built-in formula in R:

lm(y ~ A)

(Intercept)          Ach           Af           Am  
       1900        -1845        -1000           NA  

We get the same results with the (Intercept) corresponding to males (that's why "Am" is NA); children being 1900 - 1845 = 55; and women, 1900 - 1000= 900.

Now let's add the variable height in cm:

H <- c(90, 80, 145, 150, 175, 180) #children, women, men

And let's add this vector to the design matrix:

A <- cbind(A, H)

And run the regression - we want to predict money spent based on gender and height:

round(solve(t(A) %*% A) %*% t(A) %*% y,3)

        [,1]
ch -1021.667
f   -968.333
m   -348.333
H     12.667

Now the intercepts make no intuitive sense, because we want a slope to fit the entire data cloud: H = 12.7. The built-in function results in:

Coefficients:
(Intercept)          Ach           Af           Am           AH  
    -348.33      -673.33      -620.00           NA        12.67  

Corresponding to -348.33 - 673.33 = -1021.66 for children and -348.33 - 620.00 = -968.33 for females.

Let's see if it sort of works:

round(predict(fit, A = as.data.frame(H)),0)
 1      2    3      4     5      6 
 118   -8    868    932   1868   1932 

compared to y <- c(50, 60, 800, 1000, 1800, 2000)... Not that bad...

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Your answer is fine, provided OP cares to use R. The question isn't software-specific so I think your answer could be even more useful if it weren't software-specific. $\endgroup$ – Patrick Coulombe Mar 31 '15 at 18:47
  • 1
    $\begingroup$ OK. I tried to make it more general, although using R to parallel the linear algebra design and hat matrices... Hope is better... $\endgroup$ – Antoni Parellada Apr 2 '15 at 0:59
0
$\begingroup$

You can:

  • make different regressions, for each value of "C"
  • OR change "C" into dummy variables and perform regression on all variables.
| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

To use them in a linear regression, you need to select a base category and create a variable for all other categories. So, in your example, you could create a dummy variable for 'B', which is equal to 1 when the category is 'B' and 0 otherwise. The coefficient on that variable will be the difference between when the category is 'A' and 'B'. You might also wish to create interactions with the other variables to see if there is a slope effect as well as a level effect of the different categories.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

A standard method is to make a variable (called dummy variable) and code it 1 for either "Y" or "X" than regress on that variable. For example D = 1 if C = "Y" and 0 otherwise.

So you will estimate $A =\beta_0 + \beta_1B + \beta_2D + \epsilon$

In case you use R: D <- as.numeric(C == "Y") lm(A ~ B + D)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.