24
$\begingroup$

I have a data set with a set of features. Some of them are binary $(1=$ active or fired, $0=$ inactive or dormant), and the rest are real valued, e.g. $4564.342$.

I want to feed this data to a machine learning algorithm, so I $z$-score all the real-valued features. I get them between ranges $3$ and $-2$ approximately. Now the binary values are also $z$-scored, therefore the zeros become $-0.222$ and the ones become $0.5555$.

Does standardising binary variables like this make sense?

$\endgroup$
14
$\begingroup$

Standardizing binary variables does not make any sense. The values are arbitrary; they don't mean anything in and of themselves. There may be a rationale for choosing some values like 0 & 1, with respect to numerical stability issues, but that's it.

$\endgroup$
  • $\begingroup$ what if they were between 0-100. As I said, they mean stuff like "recognized a face" and "not recognized face", and 0-100 means the confidence level. Does it make sense to z-score that? $\endgroup$ – siamii May 18 '13 at 18:36
  • $\begingroup$ Your 0-100 example sounds like an ordinal rating. There's a bit of detail regarding how to best deal w/ that situation & it's been discussed on CV quite a bit. Search on the ordinal tag to learn more. $\endgroup$ – gung May 18 '13 at 18:45
  • $\begingroup$ well, the problem is that only some of the variables are 0-100. Others are for example -400 - +400 $\endgroup$ – siamii May 18 '13 at 18:49
  • $\begingroup$ What is the problem w/ that? Is this a numerical stability issue? $\endgroup$ – gung May 18 '13 at 18:52
  • $\begingroup$ perhaps, do you suggest I don't z-score? $\endgroup$ – siamii May 18 '13 at 22:24
14
$\begingroup$

A binary variable with values 0, 1 can (usually) be scaled to (value - mean) / SD, which is presumably your z-score.

The most obvious constraint on that is that if you happen to get all zeros or all ones then plugging in SD blindly would mean that the z-score is indeterminate. There is a case for assigning zero too in so far as value - mean is identically zero. But many statistical things won't make much sense if a variable is really a constant. More generally, however, if the SD is small, there is more risk that scores are unstable and/or not well determined.

A problem over giving a better answer to your question is precisely what "machine learning algorithm" you are considering. It sounds as if it's an algorithm that combines data for several variables, and so it usually will make sense to supply them on similar scales.

(LATER) As the original poster adds comments one by one, their question is morphing. I still consider that (value - mean) / SD makes sense (i.e. is not nonsensical) for binary variables so long as the SD is positive. However, logistic regression was later named as the application and for this there is no theoretical or practical gain (and indeed some loss of simplicity) to anything other than feeding in binary variables as 0, 1. Your software should be able to cope well with that; if not, abandon that software in favour of a program that can. In terms of the title question: can, yes; should, no.

$\endgroup$
  • 3
    $\begingroup$ Short answer is that it means nothing different and I see no reason why changing 0, 1 to z-scores will help anything in this situation. To convince yourself, try it both ways and see that nothing important changes. $\endgroup$ – Nick Cox May 18 '13 at 17:40
  • 3
    $\begingroup$ On the contrary, I think most people would use 0, 1 here. $\endgroup$ – Nick Cox May 18 '13 at 18:06
  • 1
    $\begingroup$ When you're doing logistic regression, the software will almost surely perform the standardization under the hood anyway (to achieve better numerical properties). Thus it's a good idea to keep the binary indicator expressed in a meaningful way. Standardizing it doesn't sound either good or useful. $\endgroup$ – whuber May 18 '13 at 19:17
  • 1
    $\begingroup$ Any machine learning method that requires you to "standardize" binary predictors is suspect. $\endgroup$ – Frank Harrell May 18 '13 at 19:57
  • 2
    $\begingroup$ Since it's your own implementation, then nobody else has any basis to give you an objective answer! You need to examine how your software treats the data in order to decide whether prior standardization make sense. $\endgroup$ – whuber May 19 '13 at 14:59
2
$\begingroup$

One nice example where it can be useful to standardize in a slightly different way is given in section 4.2 of Gelman and Hill (http://www.stat.columbia.edu/~gelman/arm/). This is mostly when the interpretation of the coefficients is of interest, and perhaps when there are not many predictors.

There, they standardize a binary variable (with equal proportion of 0 and 1) by $$ \frac{x-\mu_x}{2\sigma_x}, $$ instead of the normal $\sigma$. Then these standardized coefficients take on values $\pm 0.5 $ and then the coefficients reflect comparisons between $x=0$ and $x=1$ directly. If scaled by $\sigma$ instead then the coefficient would correspond to half the difference between the possible values of $x$.

$\endgroup$
  • $\begingroup$ Please explain "with equal proportion of 0 and 1" as the binary variables I see are rarely like that. $\endgroup$ – Nick Cox Mar 13 '18 at 7:27
  • $\begingroup$ I don't think the proportion will actually make a difference, they just use it to make the example cleaner. $\endgroup$ – Gosset's Student Mar 13 '18 at 13:27
1
$\begingroup$

What do you want to standardize, a binary random variable, or a proportion?

It makes no sense to standardize a binary random variable. A random variable is a function that assigns a real value to an event $Y:S\rightarrow \mathbb{R} $. In this case 0 for failure and 1 to success, i.e. $Y\in \lbrace 0,1\rbrace$.

In the case of a proportion, this is not a binary random variable, this is a continuous variable $X\in[0,1]$, $x\in \mathbb{R}^+$.

$\endgroup$
0
$\begingroup$

In logistic regression binary variables may be standardise for combining them with continuos vars when you want to give to all of them a non informative prior such as N~(0,5) or Cauchy~(0,5). The standardisation is adviced to be as follows: Take the total count and give

1 = proportion of 1's

0 = 1 - proportion of 1's.

-----

Edit: Actually I was not right at all, it is not an standardisation but a shifting to be centered at 0 and differ by 1 in the lower and upper condition, lets say that a population is 30% with company A and 70% other, we can define centered "Company A" variable to take on the values -0.3 and 0.7.

$\endgroup$
  • $\begingroup$ Can;t make sense of this as a standardization. $\endgroup$ – Michael Chernick Mar 12 '18 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.