Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite
2

What's it called when I'm trying to remove the effect of a variable on another variable? Am I in the right ballpark using the terms "detrending" or "normalizing" or "controlling for Y on X"? What's the general process for doing this?

The problem I'm working with currently is as follows:

I work at a mine that loads trains that are 160 wagons in length. After the trains are filled, they leave to get unloaded and then later return to the mine. Sometimes the wagons end up in different positions in the train. Our aim is to maximise tonnes in every wagon.

We've noticed two things

  1. some wagons IDs load consistently low in tonnes
  2. wagons in poisitions near the start or end of the train load consistently low. i.e. tonnes go down as distance from middle position goes up

I've got a linear model fitted in R like

    > lm(df$tonnes ~ df$dist_fr_middle)
    Coefficients:
        (Intercept)    df$dist_fr_middle  
        113.92001     -0.03915

How do I go about creating a new column in my dataframe for distance-from-middle-adjusted tonnes?

Thanks for your help!

share|improve this question
In R df is used for the density function of the F distribution. Assigning new values to built in names is possible, but it may lead to unpredictable results. – GaBorgulya Apr 28 '11 at 10:58
I do not understand what such adjusted data will be used for, it will not provide the solution that maximizes tonnes in every wagon? Residuals, as GaBorgulya suggested, shows just the deviations from the average. Well it is not what you are actually looking for, isn;t it? I think, you have to play with some mean value. For example $\max(tonnes)$ will be just intercept term of your regression 113.92, but it is hardly reachable. You may try to fill the gap between mean(df$tonnes) and this value by some technically reasonable proportion. – Dmitrij Celov Apr 28 '11 at 11:43

1 Answer

up vote 1 down vote accepted

The simple solution:

df$locationAdjustedTonnes = resid(lm(df$tonnes ~ df$dist_fr_middle))

A general way is tweaking this (The example below gives the same result as the one above):

df$locationAdjustedTonnes = df$tonnes - predict(lm(df$tonnes ~ df$dist_fr_middle))
share|improve this answer
Hi GaBorgulya, it seems that the way you've written those, df$locationAdjustedTonnes returns the difference between actual values and the model. Was that intentional? If so, no worries, I've changed it to df$locationAdjustedTonnes = df$tonnes - resid(lm(df$tonnes ~ df$dist_fr_middle)). – TMOD Apr 28 '11 at 22:09
Apologies for not knowing what I'm doing, but I was hoping that a model wouldn't fit all values of df$tonnes exactly onto the regression line. I thought it would instead adjust all values of df$tonnes by adding the mean residual for a particular dist_fr_middle to each value at that dist_from_middle – TMOD Apr 28 '11 at 22:16
You wanted "distance-from-middle-adjusted tonnes" and (I believe) that means the observed tonnes minus the expected tonnes considering distance-from-middle. – GaBorgulya Apr 28 '11 at 22:22
Sorry, you're right, I made a typo. Please disregard my previous comment. qplot(dist, locationAdjustedTonnes, data = df) does show the data with the linear effect of dist_fr_middle removed. – TMOD May 14 '11 at 8:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.