1
$\begingroup$

I have 30 features in my self-collected dataset where I want to build a regression model. When I look at my data, most of the attributes (95% of the data points) are skewed on a very small range. Out of 30 features, only 2 features has sort-of normal distribution and other features are ranged within a majority range. Here is a graph of the distribution of some of the attributes:

Distribution

My question is: is it safe/normal to work with such data? Or should I treat the 5% of the data as outliers and remove them, so that the rest of the 95% will have a normal-like distribution?

Intuitively, since majority of the attributes have very similar values, I am afraid that they won't have a distinguishing effect in the prediction. But I might be wrong, so I would like to ask here.

Improve this question
$\endgroup$
6
  • $\begingroup$ A sample of the attributes: i.imgur.com/ZRKOX7c.png $\endgroup$
    – Weka
    Jun 12, 2013 at 3:46
  • 5
    $\begingroup$ Hi @Weka and welcome to the site. First, I would discourage you from excluding 5% of the data just that the rest looks like a normal distribution. If these 5% data were correctly measured, you should include them in the analysis. Maybe a transformation would be applicable in your case, but remember: it is a common misunderstanding that the original variables in a linear regression must be normally distributed. The residuals, though, should be approximately normal. $\endgroup$
    – COOLSerdash
    Jun 12, 2013 at 8:47
  • $\begingroup$ Actually my main problem is: I want to use optimization family with this data and I believe I have to transform this data into discrete values. However, when I do that via Weka->Filter->Discretize, almost all of the values fall into the same bucket. This also applies to the value I want to predict. So basically, I end up with a dataset where 95% of the labels are assigned to 1 single majority class. Do you think this is a problem, or is it not important? $\endgroup$
    – Weka
    Jun 12, 2013 at 9:10
  • $\begingroup$ I don't know what you mean by "optimization family"? Why do you have to discretizie the data for that? Do you want to do a classification? In your question, you state that you want to build a regression model. In any case: please edit your question and include more information about what exactly you want to do and with what software etc. Thanks. $\endgroup$
    – COOLSerdash
    Jun 12, 2013 at 9:15
  • 1
    $\begingroup$ There is a procedure called support vector regression (SVR) that shares many similarities with SVM but is for a continuous dependent variable. To run it in Weka, go to "Classify"->"Choose"->"Functions"->"SMOreg". $\endgroup$
    – COOLSerdash
    Jun 12, 2013 at 10:15

3 Answers 3

Reset to default
1
$\begingroup$

First thing I'd suggest is that it looks like the huge spikes are zero. What do the graphs look like if you exclude zero? There is a class of problem called Zero Inflation Models where you have a lot of zero values and need to distinguish between zeros during times when non-zeros are possible, and zeros that are during times where non-zeros aren't really possible.

For example, if you're looking at the network traffic for desktop computers in a business, you might find that people turn off their machines (or the machines go to sleep) outside of normal business hours, so you can't get non-zero usage for those machines during those hours. During the day, you might see zero usage, but it's more random and depends on what the user is doing.

Second, are you concerned about extreme values (for say bandwidth provisioning)? There is an entire field of statistics called Extreme Value Theory, from which you may find the Weibull distribution useful.

Improve this answer
$\endgroup$
1
$\begingroup$

I would recommend keeping all data. As @COOLSerdash notes, the inputs to a regression don't need to be normally distributed, the residuals do (in order to do certain statistical tests - if you don't care about inference, you can disregard normality).

The long tails may be just the input data values that are most important to your dependent variable! Look at some transformations, e.g., logs or Box-Cox, and use cross-validation to choose the best predictive model.

Improve this answer
$\endgroup$
0
$\begingroup$

I would suggest exploring some common data transformations such as square-root and log transforms. Examine the normality of each and remember which ones you have used when interpreting your coefficients. In terms of outliers, there are quite a few approaches to how to handle them. One thing you can always try when there are a few outliers is run the regression with and without them...and compare the results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.