11
$\begingroup$

I am trying to standardize dataset columns for linear regression.

One of the columns have standard deviation = 0. 

def standardize(X):
    return (X - mean(X)) / std(X)

So this code doesn't work.

Are there any tricks to solve this problem? I have tried two things

  1. Throw aways the column with standard deviation 0 because it's a useless parameter.
  2. Add some very small noise like $10^{-10}$ to one of the elements of the column so that the standardize function works.

Thank you!

Improve this question
$\endgroup$
4
  • $\begingroup$ You are doing right. As the second option it will be better to add random normally distributed noise with small standard deviation to every value in the column. But since the first option is more simple, it is preferred. $\endgroup$
    – O_Devinyak
    Aug 29, 2012 at 7:29
  • 17
    $\begingroup$ If the standard deviation is zero, then the column is populated by one value. So if your goal is to prepare the data for regression, you can throw the column out, since it will contribute nothing to the regression. Adding small noise will only give you more problems. $\endgroup$
    – mpiktas
    Aug 29, 2012 at 7:48
  • 4
    $\begingroup$ (With the caveat that @mpiktas implicitly assumes the regression will contain a constant term, which is usually the case.) Note that adding small noise will cause this column and the constant to be extremely parallel, potentially creating all kinds of havoc in the numerical solutions due to high multicollinearity. But why standardize the columns in the first place? This will happen automatically, using almost any good solution method. $\endgroup$
    – whuber ♦
    Aug 29, 2012 at 12:20
  • 5
    $\begingroup$ @Pratik, I don't think there's a way to transform a variable with zero variance to one with mean 0 and variance 1. If you do option (2) above, you are effectively just replacing your variable with standardized random numbers. $\endgroup$
    – Macro
    Aug 29, 2012 at 13:02

3 Answers 3

Reset to default
7
$\begingroup$

The situation you describe will arise as a result of one of these two scenarios:

  1. The column you're referring to is the column of 1's which is added to your matrix of covariates so that your linear regression has an intercept term.
  2. The column is a different column than the previously-mentioned column of ones, giving you two columns of constants [****].

For Scenario 1: skip that column, standardize all the other columns, and then run the regression as you normally would.

For Scenario 2, however, you'll have to get rid of that additional constant column entirely. In fact, regardless of the question of Standardization, you'll never be able to run the regression with two constant columns since then you would have perfect collinearity. The result is that even if you try running the regression, the computer program will spit out an error message and quit halfway through [Note: this is because an OLS regression requires the matrix X'X to be non-singular for things to work out correctly].

Anyway, good luck with your, um, regressing!

[****] Just to clarify: What I mean by "two columns of constants" is that you have one column in which every element is '1' and a second column in which every element is some constant 'k'...

Improve this answer
$\endgroup$
2
$\begingroup$

The right way would be to delete the feature column from the data. But as a temporary hack -

You could just replace the 0 std to 1 for that feature. This would basically mean that the scaled value would be zero for all the data points for that feature. This makes sense as this implies that the feature values do not deviate even a bit form the mean(as the values is constant, the constant is the mean.)

FYI- This is what sklearn does! https://github.com/scikit-learn/scikit-learn/blob/7389dbac82d362f296dc2746f10e43ffa1615660/sklearn/preprocessing/data.py#L70

Improve this answer
$\endgroup$
2
  • $\begingroup$ The standard deviation being 0 means the feature is constant and hence not useful for prediction. Changing the standard deviation for a constant makes no sense. $\endgroup$
    – Michael R. Chernick
    Feb 11, 2019 at 5:17
  • $\begingroup$ Yes it won't make any sense from the modelling perspective, I just suggested it as a hack to not go through the trouble of deleting the column every time a constant feature come in a data set. $\endgroup$
    – kabrapankaj32
    Feb 11, 2019 at 22:32
0
$\begingroup$

The feature that has zero variance is useless, remove it.

Consider this, if this was the only feature, you wouldn't learn anything about the response to this feature from the data. In multivariate case, it takes linear algebra to come to the same conclusion, but the idea's the same.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.