# How to deal with a data-set having all the samples with every feature-point almost similar?

I have a very large training set where each of the feature's datapoints are very similar (float values in the range 95.0 to 97.0) ?

For example, following is some part of the training data :

95.08273,94.13686,95.843,95.83886,95.38811,1 94.37234,93.47385,94.54948,94.67984,93.80062,1 94.02294,94.96799,95.075,95.41348,94.93842,1 95.1664,94.84861,94.82346,95.61005,96.62745,0 95.23271,94.87994,95.42258,95.48337,96.3997,0 93.77203,94.3065,94.33946,93.70812,93.42625,0 94.79427,94.70049,94.40502,94.61435,94.92593,1

where the last column is the class-label - only 0 and 1.

How should I handle this kind of data if the training set is very very large (> 100k) ?

Please suggest an appropriate classifier (using scikit-learn) as well.

• Can you give us more details? What problems are you encountering? Sep 10, 2016 at 14:19
• I have updated the details, can you look at the data and suggest an approach. I am predicting the class label : 0 and 1. @mdewey Sep 11, 2016 at 9:10
• How should I deal with this problem ? @whuber Sep 14, 2016 at 12:38

Great question -- hopefully an easy solution (unless you have more complicating details to provide).

All you need to do is transform each column of the dataset with the formula...

Zi = (xi - mu) / sigma

where... xi is any individual datapoint, mu is the mean for the set, and sigma is either sample or population standard deviation (as appropriate), and Zi is the standardized point value

What does this do?
- Allows you to view the data in a unit-removed form where the center/mean/average/balance point of the data is 0
- Allows you to easily understand the distance from center, because the z-value denotes how many standard deviations the value is from the center
- If you have multiple features/variables/columns of corresponding data, removes any confusion from unusual scales or units when looking at data (especially as data features compare to each other). For a set like this, standardizing will essentially expand that tiny range to make it much easier to differentiate values.

Based on the samples you provided, I don't observe anything obviously complicating. In fact, you seem to have quite precise details. Perhaps you want to look into the sampling method and ask yourself if this many significant digits is reasonably accurate -- if not, there is an argument for rounding according to the precision of the data collection method.

A couple of points:

(1) Please keep in mind that the "range*" of a dataset (the difference between the max and min values) is not a good way to assess similarity. More relevant would be the distribution of values and the quantity of distinct values in your dataset. It is perfectly fine that numbers are close in numerical value, but you still have an interesting dataset with sufficient variability and the potential for some nicely statistically significant results in, say, a regression. If these were integer values, then we might have a problem -- and maybe even need to consider looking at the data as categorical or binary.

(2) A large dataset is very rarely a downside. In most applications, the more the better!

*Quotes to imply that I'm using the term in the strict statistical sense, rather than the colloquial, or common, usage.

• I have a doubt regarding your formula, is the mean as well standard-deviation taken of the single column containing a particular xi's ? @Omeed Sep 13, 2016 at 2:42