I have learned that you are supposed to scale features as a preprocessing step for most ML algorithms (so that all features are of the same magnitude and no direction is preferred, for example in Gradient Descent).

But what is the best approach if one feature varies between orders of magnitude, i.e. meaningful values for some feature $f$ are $f\in\{0, 1, 2, 3, 4, 100, 101, 102, 100000, 100001, 100002\}.$

If I scale, I lose all meaning of difference between for example 0,1,2,3 and 4. If I take the logarithm (as to alleviate the differenc between 100000 and the rest), I have the same problem.

  • $\begingroup$ Do I derive correctly that this feature is thought to be continuous? (If not, you could probably just use it as categorial feature instead.) If yes: you might still be able to derive some further information from it, e.g. if you have feature values only from certain ranges (0-4, 100-200, 1000000-200000, etc). $\endgroup$ – geekoverdose Jun 30 '16 at 14:19
  • $\begingroup$ In my case, it just happens to have values like above (think of the length of some metal component for machines, some are really tiny, some are extraordinary big). But their "impact" is not directly linearly related to their size: A tiny intricate metal part might be pretty expensive in comparison to something big, but in the same order of magnitude the effect would be linear. For example: Tiny machine part = 10\$. Slightly bigger machine part (still tiny) = 15 \$. Large machine part = 10\$, very large machine part = 15 \$. $\endgroup$ – Mercury Bench Jun 30 '16 at 14:24
  • $\begingroup$ So it is linearly proportional in some lengthscale and then "resets" up until the next order of magnitude. $\endgroup$ – Mercury Bench Jun 30 '16 at 14:27
  • $\begingroup$ OK, this sounds like there is additional information comprised in that variable. If e.g. splitting this variable into multiple variables (which you could scale properly) would be an option for you I'll try to formulate a short answer for this. $\endgroup$ – geekoverdose Jun 30 '16 at 14:33

From your description, I'd probably add an additional feature preprocessing step for that variable. The core reason for this is that you essentially 2 pieces of information encoded in this variable:

  1. which range the original value belongs to (could be numeric/categorial), and
  2. what the value inside this range is (numeric).

Therefore, decomposing the variable into those pieces might make sense. For a starters, you could encode the range the value belongs to as one categorial variable, then use the difference between the value and the mean/median/mode value of the associated range to encode this as a second, numeric variable.

I understand that this requires you to know the amount of ranges you are dealing with beforehand - so if you don't know those feel free to comment, I'll try to update my answer then.

  • $\begingroup$ I actually don't know the amount of ranges beforehand but I can always guess or use a clustering algorithm if the exact range is important, right? $\endgroup$ – Mercury Bench Jun 30 '16 at 15:25
  • $\begingroup$ @FasEtNefas Exactly, but I assume this will become an own question if you should encounter problems when doing so ;) $\endgroup$ – geekoverdose Jun 30 '16 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.