# Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

2,472 questions
Filter by
Sorted by
Tagged with
376k views

### When (and why) should you take the log of a distribution (of numbers)?

Say I have some historical data e.g., past stock prices, airline ticket price fluctuations, past financial data of the company... Now someone (or some formula) comes along and says "let's take/use ...
• 14.8k
214k views

### How should I transform non-negative data including zeros?

If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used: $\log(x+1)$ which has the ...
• 57.7k
543k views

### In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
• 2,411
726k views

### How to summarize data by group in R? [closed]

I have R data frame like this: ...
• 4,139
85k views

### One-hot vs dummy encoding in Scikit-learn

There are two different ways to encoding categorical variables. Say, one categorical variable has n values. One-hot encoding converts it into n variables, while dummy encoding converts it into n-1 ...
• 2,095
32k views

### How small a quantity should be added to x to avoid taking the log of zero?

I have analysed my data as they are. Now I want to look at my analyses after taking the log of all variables. Many variables contain many zeros. Therefore I add a small quantity to avoid taking the ...
• 3,774
91k views

### When are Log scales appropriate?

I've read that using log scales when charting/graphing is appropriate in certain circumstances, like the y-axis in a time series chart. However, I've not been able to find a definitive explanation as ...
• 1,551
31k views

### Why is the square root transformation recommended for count data?

It is often recommended to take the square root when you have count data. (For some examples on CV, see @HarveyMotulsky's answer here, or @whuber's answer here.) On the other hand, when fitting a ...
60k views

### Does it ever make sense to treat categorical data as continuous?

In answering this question on discrete and continuous data I glibly asserted that it rarely makes sense to treat categorical data as continuous. On the face of it that seems self-evident, but ...
• 1,918
81k views

### Interpretation of log transformed predictor and/or response

I'm wondering if it makes a difference in interpretation whether only the dependent, both the dependent and independent, or only the independent variables are log transformed. Consider the case of <...
• 3,157
154k views

### Is it a good practice to always scale/normalize data for machine learning? [duplicate]

My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect ...
39k views

### Box-Cox like transformation for independent variables?

Is there a Box-Cox like transformation for independent variables? That is, a transformation that optimizes the $x$ variable so that the y~f(x) will make a more ...
• 21.8k
32k views

### How to apply standardization/normalization to train- and testset if prediction is the goal?

Do I transform all my data or folds (if CV is applied) at the same time? e.g. (allData - mean(allData)) / sd(allData) Do I transform trainset and testset ...
• 807
83k views

### Normalization vs. scaling

What is the difference between data 'Normalization' and data 'Scaling'? Till now I thought both terms refers to same process but now I realize there is something more that I don't know/understand. ...
• 931
53k views

### Alternatives to one-way ANOVA for heteroskedastic data

I have data from 3 groups of algae biomass ($A$, $B$, $C$) which contain unequal sample sizes ($n_A=15$, $n_B=13$, $n_C=12$) and I would like compare if these groups are from the same population. One-...
• 561
44k views

### Regression: Transforming Variables

When transforming variables, do you have to use all of the same transformation? For example, can I pick and choose differently transformed variables, as in: Let, $x_1,x_2,x_3$ be age, length of ...
• 7,332
23k views

### Analysis with complex data, anything different?

Say for example you are doing a linear model, but the data $y$ is complex. $y = x \beta + \epsilon$ My data set is complex, as in all the numbers in $y$ are of the form $(a + bi)$. Is there ...
• 2,851
22k views

### Why are log probabilities useful?

Probabilities of a random variable's observations are in the range $[0,1]$, whereas log probabilities transform them to the log scale. What then is the corresponding range of log probabilities, i.e. ...
• 4,015
48k views

### ANOVA on binomial data

I am analyzing an experimental data set. The data consists of a paired vector of treatment type and a binomial outcome: ...
• 563
111k views

### What is the reason the log transformation is used with right-skewed distributions?

I once heard that log transformation is the most popular one for right-skewed distributions in linear regression or quantile regression I would like to know is there any reason underlying this ...
• 5,232
158k views

### Changing the scale of a variable to 0-100

I have constructed a social capital index using PCA technique. This index comprises values both positive and negative. I want to transform / convert this index to 0-100 scale to make it easy to ...
• 379
75k views

### When to log transform a time series before fitting an ARIMA model

I have previously used forecast pro to forecast univariate time series, but am switching my workflow over to R. The forecast package for R contains a lot of useful functions, but one thing it doesn't ...
• 24.2k
27k views

### How to perform isometric log-ratio transformation

I have data on movement behaviours (time spent sleeping, sedentary, and doing physical activity) that sums to approximately 24 (as in hours per day). I want to create a variable that captures the ...
• 373
31k views

### Is whitening always good?

A common pre-processing step for machine learning algorithms is whitening of data. It seems like it is always good to do whitening since it de-correlates the data, making it simpler to model. When ...
• 1,626
104k views

### What does "normalization" mean and how to verify that a sample or a distribution is normalized?

I have a question in which it asks to verify whether if the Uniform distribution (${\rm Uniform}(a,b)$) is normalized. For one, what does it mean for any distribution to be normalized? And two, how ...
• 509
31k views

### What are the assumptions of negative binomial regression?

I'm working with a large data set (confidential, so I can't share too much), and came to the conclusion a negative binomial regression would be necessary. I've never done a glm regression before, and ...
• 489
62k views

### How to change data between wide and long formats in R? [closed]

You can have data in wide format or in long format. This is quite an important thing, as the useable methods are different, depending on the format. I know you have to work with ...
• 729
4k views

### What are the myths associated with linear regression, data transformations?

I have been encountering many assumptions associated with linear regression (especially ordinary least squares regression) which are untrue or unnecessary. For example: independent variables must ...
61k views

### From uniform distribution to exponential distribution and vice-versa

This is probably a trivial question, but my search has been fruitless so far, including this wikipedia article, and the "Compendium of Distributions" document. If $X$ has a uniform distribution, does ...
• 2,748
97k views

### Transforming variables for multiple regression in R

I am trying to perform a multiple regression in R. However, my dependent variable has the following plot: Here is a scatterplot matrix with all my variables (...
• 869