Since binary variables are 0 or 1 to begin with, it doesn't make sense to normalize them.
On the other hand, we normalize continuous variables which usually results in values greater than 0 and less than 1.
There seems to be an asymmetry here. The continuous variables are normalized, but the binary variables are not. The magnitude of the binary variables is either as high or as low as it can possibly be, not because of any information in the variable itself but because that is our convention. On the other hand, the continuous variable can take on any value in the range 0 or 1. Doesn't this make it inappropriate to use both continuous variables and binary variables as predictors in a model? Am I overthinking this?