0
$\begingroup$

I have a very large training set where each of the feature's datapoints are very similar (float values in the range 95.0 to 97.0) ?

For example, following is some part of the training data :

95.08273,94.13686,95.843,95.83886,95.38811,1 94.37234,93.47385,94.54948,94.67984,93.80062,1 94.02294,94.96799,95.075,95.41348,94.93842,1 95.1664,94.84861,94.82346,95.61005,96.62745,0 95.23271,94.87994,95.42258,95.48337,96.3997,0 93.77203,94.3065,94.33946,93.70812,93.42625,0 94.79427,94.70049,94.40502,94.61435,94.92593,1

where the last column is the class-label - only 0 and 1.

How should I handle this kind of data if the training set is very very large (> 100k) ?

Please suggest an appropriate classifier (using scikit-learn) as well.

|cite|improve this question
$\endgroup$
  • $\begingroup$ Can you give us more details? What problems are you encountering? $\endgroup$ – mdewey Sep 10 '16 at 14:19
  • $\begingroup$ I have updated the details, can you look at the data and suggest an approach. I am predicting the class label : 0 and 1. @mdewey $\endgroup$ – Jarvis Sep 11 '16 at 9:10
  • $\begingroup$ How should I deal with this problem ? @whuber $\endgroup$ – Jarvis Sep 14 '16 at 12:38
4
$\begingroup$

Great question -- hopefully an easy solution (unless you have more complicating details to provide).

Standardize your data!

All you need to do is transform each column of the dataset with the formula...

Zi = (xi - mu) / sigma

where... xi is any individual datapoint, mu is the mean for the set, and sigma is either sample or population standard deviation (as appropriate), and Zi is the standardized point value

What does this do?
- Allows you to view the data in a unit-removed form where the center/mean/average/balance point of the data is 0
- Allows you to easily understand the distance from center, because the z-value denotes how many standard deviations the value is from the center
- If you have multiple features/variables/columns of corresponding data, removes any confusion from unusual scales or units when looking at data (especially as data features compare to each other). For a set like this, standardizing will essentially expand that tiny range to make it much easier to differentiate values.

Based on the samples you provided, I don't observe anything obviously complicating. In fact, you seem to have quite precise details. Perhaps you want to look into the sampling method and ask yourself if this many significant digits is reasonably accurate -- if not, there is an argument for rounding according to the precision of the data collection method.

A couple of points:

(1) Please keep in mind that the "range*" of a dataset (the difference between the max and min values) is not a good way to assess similarity. More relevant would be the distribution of values and the quantity of distinct values in your dataset. It is perfectly fine that numbers are close in numerical value, but you still have an interesting dataset with sufficient variability and the potential for some nicely statistically significant results in, say, a regression. If these were integer values, then we might have a problem -- and maybe even need to consider looking at the data as categorical or binary.

(2) A large dataset is very rarely a downside. In most applications, the more the better!

*Quotes to imply that I'm using the term in the strict statistical sense, rather than the colloquial, or common, usage.

Best wishes with your analysis!

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I followed your suggestion, but the accuracy is still not much, what should I exactly do for the kind of data above ? Any suggestions on which classifier I should use ? @Omeed $\endgroup$ – Jarvis Sep 11 '16 at 3:11
  • $\begingroup$ Whats the data? What are you predicting? Why not just loop through like in this code: stats.stackexchange.com/questions/234321/… $\endgroup$ – Omeed Sep 11 '16 at 4:25
  • $\begingroup$ The data I posted above, where I am predicting whether the class label is 0 or 1, the data I posted in my question is a part of the training set data, where the last column is the class label - 0 or 1. @Omeed $\endgroup$ – Jarvis Sep 11 '16 at 6:31
  • $\begingroup$ I have a doubt regarding your formula, is the mean as well standard-deviation taken of the single column containing a particular xi's ? @Omeed $\endgroup$ – Jarvis Sep 13 '16 at 2:42
  • $\begingroup$ Yes each standardizing operation is for each column separately (it's own mean and SD). Read more about standardizing and z-scores for more info -- it's a pretty basic concept that you should know before doing more complicated modeling. So far as advice about predicting, without explaning what column values represent (e.g., income, height, etc.) impossible to help you pick a classifier and "solve" the prediction problem. Start with KNN, logistic regrssion, and SVM... could go as fancy as neural nets if needed $\endgroup$ – Omeed Sep 13 '16 at 14:41
1
$\begingroup$

Try other feature scaling methods: https://en.wikipedia.org/wiki/Feature_scaling

Do some feature engineering: for example, differences and ratios of variables could be better predictors.

|cite|improve this answer
$\endgroup$
0
$\begingroup$

Use a StandardScaler() if you can use scikit-learn. It will shape your data into a form most optimal for the classifiers.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.