# How to deal with a data-set having all the samples with every feature-point almost similar?

I have a very large training set where each of the feature's datapoints are very similar (float values in the range 95.0 to 97.0) ?

For example, following is some part of the training data :

95.08273,94.13686,95.843,95.83886,95.38811,1 94.37234,93.47385,94.54948,94.67984,93.80062,1 94.02294,94.96799,95.075,95.41348,94.93842,1 95.1664,94.84861,94.82346,95.61005,96.62745,0 95.23271,94.87994,95.42258,95.48337,96.3997,0 93.77203,94.3065,94.33946,93.70812,93.42625,0 94.79427,94.70049,94.40502,94.61435,94.92593,1

where the last column is the class-label - only 0 and 1.

How should I handle this kind of data if the training set is very very large (> 100k) ?

Please suggest an appropriate classifier (using scikit-learn) as well.

• Can you give us more details? What problems are you encountering? – mdewey Sep 10 '16 at 14:19
• I have updated the details, can you look at the data and suggest an approach. I am predicting the class label : 0 and 1. @mdewey – Jarvis Sep 11 '16 at 9:10
• How should I deal with this problem ? @whuber – Jarvis Sep 14 '16 at 12:38

Great question -- hopefully an easy solution (unless you have more complicating details to provide).

All you need to do is transform each column of the dataset with the formula...

Zi = (xi - mu) / sigma

where... xi is any individual datapoint, mu is the mean for the set, and sigma is either sample or population standard deviation (as appropriate), and Zi is the standardized point value

What does this do?
- Allows you to view the data in a unit-removed form where the center/mean/average/balance point of the data is 0
- Allows you to easily understand the distance from center, because the z-value denotes how many standard deviations the value is from the center
- If you have multiple features/variables/columns of corresponding data, removes any confusion from unusual scales or units when looking at data (especially as data features compare to each other). For a set like this, standardizing will essentially expand that tiny range to make it much easier to differentiate values.

Based on the samples you provided, I don't observe anything obviously complicating. In fact, you seem to have quite precise details. Perhaps you want to look into the sampling method and ask yourself if this many significant digits is reasonably accurate -- if not, there is an argument for rounding according to the precision of the data collection method.

A couple of points:

(1) Please keep in mind that the "range*" of a dataset (the difference between the max and min values) is not a good way to assess similarity. More relevant would be the distribution of values and the quantity of distinct values in your dataset. It is perfectly fine that numbers are close in numerical value, but you still have an interesting dataset with sufficient variability and the potential for some nicely statistically significant results in, say, a regression. If these were integer values, then we might have a problem -- and maybe even need to consider looking at the data as categorical or binary.

(2) A large dataset is very rarely a downside. In most applications, the more the better!

*Quotes to imply that I'm using the term in the strict statistical sense, rather than the colloquial, or common, usage.

Best wishes with your analysis!

• I followed your suggestion, but the accuracy is still not much, what should I exactly do for the kind of data above ? Any suggestions on which classifier I should use ? @Omeed – Jarvis Sep 11 '16 at 3:11
• Whats the data? What are you predicting? Why not just loop through like in this code: stats.stackexchange.com/questions/234321/… – Omeed Sep 11 '16 at 4:25
• The data I posted above, where I am predicting whether the class label is 0 or 1, the data I posted in my question is a part of the training set data, where the last column is the class label - 0 or 1. @Omeed – Jarvis Sep 11 '16 at 6:31
• I have a doubt regarding your formula, is the mean as well standard-deviation taken of the single column containing a particular xi's ? @Omeed – Jarvis Sep 13 '16 at 2:42