67 votes
Accepted

Is it a good practice to always scale/normalize data for machine learning?

First things first, I don't think there are many questions of the form "Is it a good practice to always X in machine learning" where the answer is going to be definitive. Always? Always always? Across ...
one_observation's user avatar
61 votes
Accepted

Why are log probabilities useful?

The log of $1$ is just $0$ and the limit as $x$ approaches $0$ (from the positive side) of $\log x$ is $-\infty$. So the range of values for log probabilities is $(-\infty, 0]$. The real advantage is ...
Greg Snow's user avatar
  • 51.5k
50 votes
Accepted

One-hot vs dummy encoding in Scikit-learn

Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to ...
Matthew Drury's user avatar
45 votes
Accepted

How to perform isometric log-ratio transformation

The ILR (Isometric Log-Ratio) transformation is used in the analysis of compositional data. Any given observation is a set of positive values summing to unity, such as the proportions of chemicals in ...
whuber's user avatar
  • 321k
39 votes
Accepted

From uniform distribution to exponential distribution and vice-versa

It is not the case that exponentiating a uniform random variable gives an exponential, nor does taking the log of an exponential random variable yield a uniform. Let $U$ be uniform on $(0,1)$ and let ...
Glen_b's user avatar
  • 281k
33 votes
Accepted

What are the myths associated with linear regression, data transformations?

There are three myths that bother me. Predictor variables need to be normal. The pooled/marginal distribution of $Y$ has to be normal. Predictor variables should not be correlated, and if they are, ...
32 votes
Accepted

How does entropy depend on location and scale?

Since the probability element of $X$ is $f(x)\mathrm{d}x,$ the change of variable $y = x\sigma + \mu$ is equivalent to $x = (y-\mu)/\sigma,$ whence from $$f(x)\mathrm{d}x = f\left(\frac{y-\mu}{\sigma}\...
whuber's user avatar
  • 321k
27 votes

Why are log probabilities useful?

I would like to add that taking the log of a probability or probability density can often simplify certain computations, such as calculating the gradient of the density given some of its parameters. ...
John Madden's user avatar
  • 4,085
24 votes

Is it a good practice to always scale/normalize data for machine learning?

Well, I believe a more geometric point of view will help better decide whether normalization helps or not. Imagine your problem of interest has only two features and they range differently. Then ...
Amir's user avatar
  • 674
24 votes

What are the myths associated with linear regression, data transformations?

Myth A linear regression model can only model linear relationships between the outcome $y$ and the explanatory variables. Fact Despite the name, linear regression models can easily accomodate ...
24 votes
Accepted

If a data set appears to be normal after some transformation is applied, is it really normal?

NO It means that the transformed distribution is normal (at least roughly). Depending on the transformation, it might suggest a lack of normality of the original distribution. For instance, if a log-...
Dave's user avatar
  • 61.1k
22 votes

Are log difference time series models better than growth rates?

One major advantage of log-differences is symmetry: if you have a log difference of $0.1$ today and one of $-0.1$ tomorrow, you are back from where you started. In contrast, 10% growth today and 10% ...
Christoph Hanck's user avatar
22 votes
Accepted

Predicting y from log y as the dependent variable

The underlying model is $$E[\log Y] = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k$$ or, in terms of error terms $\varepsilon_i,$ $$\log Y_i = \beta_0 + \beta_1 x_{1i} + \cdots + \beta_k x_{ki} + \...
whuber's user avatar
  • 321k
21 votes
Accepted

Why feature scaling only to training set?

Not quite. You learn the means and standard deviation of the training set, and then: Standardize the training set using the training set means and standard deviations. Standardize any test set using ...
Matthew Drury's user avatar
21 votes

Will log transformation always mitigate heteroskedasticity?

No; sometimes it will make it worse. Heteroskedasticity where the spread is close to proportional to the conditional mean will tend to be improved by taking log(y), but if it's not increasing with ...
Glen_b's user avatar
  • 281k
20 votes

How is the Box-Cox transformation valid?

One statement and six questions here. But first on behalf of namesakes everywhere and the continuing history of statistics, please note that the proper-cased name "Box-Cox" is standard. The Box-...
Nick Cox's user avatar
  • 55.5k
20 votes

From uniform distribution to exponential distribution and vice-versa

You almost have it back to front. You asked: "If $X$ has a uniform distribution, does it mean that $e^X$ follows an exponential distribution?" "Similarly, if $Y$ follows an exponential distribution,...
Henry's user avatar
  • 38.8k
20 votes
Accepted

Quantile Transformation with Gaussian Distribution - Sklearn Implementation

Yes, it appears to be described in a few different places, with no link to any papers. The class documentation summarises the algorithm as follows: The transformation is applied on each feature ...
A. G.'s user avatar
  • 2,151
20 votes

What are the myths associated with linear regression, data transformations?

@Dave's answers are excellent. Here are some more myths. The original scale/transformation for Y is one you should use in the model. The central limit theorem means you don't have to worry about any ...
19 votes

How do I find a variance-stabilizing transformation?

Since the question partly concerns notation and basic concepts, I will be expansive in the following answer, making sure to motivate, describe, and explain the notation, the statistical reasoning, and ...
whuber's user avatar
  • 321k
18 votes
Accepted

What to do with GLM (Gamma) when residuals are not normally distributed?

Residuals in glm's such as with the gamma family is not normally distributed, so simply a QQ plot against the normal distribution isn't very helpful. To understand this, note that the usual linear ...
kjetil b halvorsen's user avatar
18 votes

Using Decibels in Statistics

Strictly we need to see your data to have any chance of giving definitive advice, but it is possible to guess. As you say, decibels are already on a logarithmic scale. That is likely to mean, for a ...
Nick Cox's user avatar
  • 55.5k
18 votes

Interpreting log-log regression with log(1+x) as independent variable

Definitely not, except when $x$ is much larger than $1.$ This is one reason why the automatic reflex to "just add $1$ to values that might be zero before taking the log" is difficult to ...
whuber's user avatar
  • 321k
18 votes
Accepted

For linear regression, why do people usually standardize the X variables and log transform Y variables to make them normally distributed?

All 3 of your points are incorrect. The log transformation does not necessarily make the dependent variable normally distributed. But more importantly, it doesn't matter if the dependent variable is ...
mkt's user avatar
  • 18.1k
18 votes
Accepted

Interpreting the Lambdas of Yeo Johnson Transformation?

A table adds little, but a picture can add a lot more to our understanding. I offer two pictures. Unlike the Box-Cox transformation, which applies to positive numbers, the Yeo-Johnson transformation ...
whuber's user avatar
  • 321k
18 votes
Accepted

An interesting observation regarding the log transformation of data

This is independent of $\epsilon$ because it’s independent of the $y$’s entirely, and depends only on $n$. Assuming that $S_y$ means $\sqrt{\sum(y_i-\bar{y})^2/(n-1)}$: \begin{align} \log\phi(z) &=...
Matt F.'s user avatar
  • 4,266
17 votes

Interpretation of log transformed predictor and/or response

The main purpose of linear regression is to estimate a mean difference of outcomes comparing adjacent levels of a regressor. There are many types of means. We are most familiar with the arithmetic ...
AdamO's user avatar
  • 62k
17 votes
Accepted

Why does log-transformation of the RNA-seq data reduce the amount of explained variance in PCA?

Based on the size of your dataset, I suspect you are working with the single cell RNA-seq data. If so, I can confirm your observation: with scRNA-seq data, PCA explained variances after log-transform ...
amoeba's user avatar
  • 104k
17 votes

Linear regression with "hour of the day"

Dummy encoding would destroy any proximity measure (and ordering) among hours. For example, the distance between 1 PM and 9 PM would be the same as the distance between 1 PM and 1 AM. It'd be harder ...
gunes's user avatar
  • 57k
16 votes
Accepted

What is the most appropriate way to transform proportions when they are an independent variable?

The main question about transforming proportions (I'll use $x$ as symbol, similarly but not identically to your notation) allows some general comments. In what follows I take it that the main motive ...
Nick Cox's user avatar
  • 55.5k

Only top scored, non community-wiki answers of a minimum length are eligible