2
$\begingroup$

I'm trying to run a linear regression in python to determine house prices given many features. Some of these are numeric and some are non-numeric. I'm attempting to do one hot encoding for the non-numeric columns and attach the new, numeric, columns to the old dataframe and drop the non-numeric columns. This is done on both the training data and test data.

I then took the intersection of the two columns features (since I had some encodings that were only located in the testing data). Afterwards, it goes into a linear regression. The code is the following:

non_numeric = list(set(list(train)) - set(list(train._get_numeric_data())))
train = pandas.concat([train, pandas.get_dummies(train[non_numeric])], axis=1)
train.drop(non_numeric, axis=1, inplace=True)

train = train._get_numeric_data()
train.fillna(0, inplace = True)

non_numeric = list(set(list(test)) - set(list(test._get_numeric_data())))
test = pandas.concat([test, pandas.get_dummies(test[non_numeric])], axis=1)
test.drop(non_numeric, axis=1, inplace=True)

test = test._get_numeric_data()
test.fillna(0, inplace = True)

feature_columns = list(set(train) & set(test))
#feature_columns.remove('SalePrice')
X = train[feature_columns]
y = train['SalePrice']

lm = LinearRegression()
lm.fit(X, y)

import numpy
predictions = numpy.absolute(lm.predict(test).round(decimals = 2))

The issue that I'm having is that I get these absurdly high Sale Prices as output, somewhere in the hundreds of millions of dollars (sometimes even in the trillions). Before I tried one hot encoding I got reasonable numbers in the hundreds of thousands of dollars. I'm having trouble figuring out what changed. I posted this on stackoverflow and got a response suggesting that it might be a collinearity issue, but I tried setting fit_intercept parameter of LinearRegression to False as well as setting drop_first parameter of get_dummies to True.

Also, if there is a better way to do this I'd be eager to hear about it.

$\endgroup$

2 Answers 2

1
$\begingroup$

There is at least one point that seems very suspicious.

Consider the lines

train = pandas.concat([train, pandas.get_dummies(train[non_numeric])], axis=1)

and

test = pandas.concat([test, pandas.get_dummies(test[non_numeric])], axis=1)

Specifically, the parts

pandas.get_dummies(train[non_numeric])

and

pandas.get_dummies(test[non_numeric])

Note that this depends on the values of the matrices. There is no reason implying that the generated columns must be the same, and so it's hard to guess the effect on the prediction of the test data.

In general, when performing get_dummies, it is better to do it before train/test splits (including cross-validation). This is an unsupervised transformation anyway, so it is not "peeking".

$\endgroup$
3
  • $\begingroup$ That worked, thanks. Do you have any idea why feature_columns = list(set(train) & set(test)) didn't take care of this problem? $\endgroup$
    – David
    Jun 26, 2017 at 2:28
  • $\begingroup$ @David IIUC, it is because the line you mentioned is determined by the headers of the DataFrame, whereas the code that follows after it, manipulates things based on the content of the DataFrame. Hence, even if the headers are the same for the train and test DataFrames, the resulting columns may not be identical. $\endgroup$
    – Ami Tavory
    Jun 26, 2017 at 3:58
  • $\begingroup$ Ok, thanks. I thought that the headers would be consistent with the data in them since they contain the header label and the categorical label, but I guess not. $\endgroup$
    – David
    Jun 27, 2017 at 2:15
0
$\begingroup$

I think your linear regression has a collinearity issue. When you call pd.get_dummies, you get a dummy column for every level of the categorical variable

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'a': [1, 2, 1], 'b': [1, 3, 1], 'c': ['a', 'b', 'a']})

In [3]: pd.get_dummies(df)
Out[13]:
   a  b  c_a  c_b
0  1  1    1    0
1  2  3    0    1
2  1  1    1    0

If you then plug this directly into a linear regression, the dummy columns for each categorical variable will be co-linear. This means your regression fitting algorithm will not converge, and your coefficients will not be unstable, and border on meaningless.

I would recommend using patsy to construct design matrices for regression in python. It will take care of co-linearity issues for you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.