Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
242 votes
4 answers
369k views

When (and why) should you take the log of a distribution (of numbers)?

Say I have some historical data e.g., past stock prices, airline ticket price fluctuations, past financial data of the company... Now someone (or some formula) comes along and says "let's take/use ...
PhD's user avatar
  • 14.6k
241 votes
14 answers
210k views

How should I transform non-negative data including zeros?

If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used: $\log(x+1)$ which has the ...
Rob Hyndman's user avatar
  • 56.8k
222 votes
8 answers
532k views

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
d_2's user avatar
  • 2,381
188 votes
9 answers
725k views

How to summarize data by group in R? [closed]

I have R data frame like this: ...
Yuriy Petrovskiy's user avatar
80 votes
3 answers
84k views

One-hot vs dummy encoding in Scikit-learn

There are two different ways to encoding categorical variables. Say, one categorical variable has n values. One-hot encoding converts it into n variables, while dummy encoding converts it into n-1 ...
Munichong's user avatar
  • 2,095
76 votes
5 answers
30k views

How small a quantity should be added to x to avoid taking the log of zero?

I have analysed my data as they are. Now I want to look at my analyses after taking the log of all variables. Many variables contain many zeros. Therefore I add a small quantity to avoid taking the ...
miura's user avatar
  • 3,684
70 votes
3 answers
90k views

When are Log scales appropriate?

I've read that using log scales when charting/graphing is appropriate in certain circumstances, like the y-axis in a time series chart. However, I've not been able to find a definitive explanation as ...
dav's user avatar
  • 1,551
68 votes
1 answer
31k views

Why is the square root transformation recommended for count data?

It is often recommended to take the square root when you have count data. (For some examples on CV, see @HarveyMotulsky's answer here, or @whuber's answer here.) On the other hand, when fitting a ...
gung - Reinstate Monica's user avatar
64 votes
3 answers
80k views

Interpretation of log transformed predictor and/or response

I'm wondering if it makes a difference in interpretation whether only the dependent, both the dependent and independent, or only the independent variables are log transformed. Consider the case of <...
upabove's user avatar
  • 3,137
64 votes
8 answers
59k views

Does it ever make sense to treat categorical data as continuous?

In answering this question on discrete and continuous data I glibly asserted that it rarely makes sense to treat categorical data as continuous. On the face of it that seems self-evident, but ...
walkytalky's user avatar
  • 1,898
63 votes
7 answers
149k views

Is it a good practice to always scale/normalize data for machine learning? [duplicate]

My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect ...
Juan Antonio Gomez Moriano's user avatar
61 votes
4 answers
39k views

Box-Cox like transformation for independent variables?

Is there a Box-Cox like transformation for independent variables? That is, a transformation that optimizes the $x$ variable so that the y~f(x) will make a more ...
Tal Galili's user avatar
  • 21.6k
61 votes
1 answer
32k views

How to apply standardization/normalization to train- and testset if prediction is the goal?

Do I transform all my data or folds (if CV is applied) at the same time? e.g. (allData - mean(allData)) / sd(allData) Do I transform trainset and testset ...
DerTom's user avatar
  • 807
56 votes
4 answers
83k views

Normalization vs. scaling

What is the difference between data 'Normalization' and data 'Scaling'? Till now I thought both terms refers to same process but now I realize there is something more that I don't know/understand. ...
d.putto's user avatar
  • 921
55 votes
1 answer
52k views

Alternatives to one-way ANOVA for heteroskedastic data

I have data from 3 groups of algae biomass ($A$, $B$, $C$) which contain unequal sample sizes ($n_A=15$, $n_B=13$, $n_C=12$) and I would like compare if these groups are from the same population. One-...
Rick L.'s user avatar
  • 551
53 votes
1 answer
43k views

Regression: Transforming Variables

When transforming variables, do you have to use all of the same transformation? For example, can I pick and choose differently transformed variables, as in: Let, $x_1,x_2,x_3$ be age, length of ...
Brandon Bertelsen's user avatar
45 votes
5 answers
22k views

Analysis with complex data, anything different?

Say for example you are doing a linear model, but the data $y$ is complex. $ y = x \beta + \epsilon $ My data set is complex, as in all the numbers in $y$ are of the form $(a + bi)$. Is there ...
bill_e's user avatar
  • 2,831
42 votes
3 answers
47k views

ANOVA on binomial data

I am analyzing an experimental data set. The data consists of a paired vector of treatment type and a binomial outcome: ...
speciousfool's user avatar
41 votes
4 answers
19k views

Why are log probabilities useful?

Probabilities of a random variable's observations are in the range $[0,1]$, whereas log probabilities transform them to the log scale. What then is the corresponding range of log probabilities, i.e. ...
develarist's user avatar
  • 3,927
40 votes
5 answers
107k views

What is the reason the log transformation is used with right-skewed distributions?

I once heard that log transformation is the most popular one for right-skewed distributions in linear regression or quantile regression I would like to know is there any reason underlying this ...
user3269's user avatar
  • 5,162
37 votes
6 answers
157k views

Changing the scale of a variable to 0-100

I have constructed a social capital index using PCA technique. This index comprises values both positive and negative. I want to transform / convert this index to 0-100 scale to make it easy to ...
Sohail Akram's user avatar
37 votes
4 answers
74k views

When to log transform a time series before fitting an ARIMA model

I have previously used forecast pro to forecast univariate time series, but am switching my workflow over to R. The forecast package for R contains a lot of useful functions, but one thing it doesn't ...
Zach's user avatar
  • 23.8k
37 votes
3 answers
25k views

How to perform isometric log-ratio transformation

I have data on movement behaviours (time spent sleeping, sedentary, and doing physical activity) that sums to approximately 24 (as in hours per day). I want to create a variable that captures the ...
Nicole's user avatar
  • 373
35 votes
3 answers
31k views

Is whitening always good?

A common pre-processing step for machine learning algorithms is whitening of data. It seems like it is always good to do whitening since it de-correlates the data, making it simpler to model. When ...
Ran's user avatar
  • 1,626
34 votes
2 answers
30k views

What are the assumptions of negative binomial regression?

I'm working with a large data set (confidential, so I can't share too much), and came to the conclusion a negative binomial regression would be necessary. I've never done a glm regression before, and ...
Carly's user avatar
  • 489
33 votes
5 answers
62k views

How to change data between wide and long formats in R? [closed]

You can have data in wide format or in long format. This is quite an important thing, as the useable methods are different, depending on the format. I know you have to work with ...
Mien's user avatar
  • 729
32 votes
3 answers
59k views

From uniform distribution to exponential distribution and vice-versa

This is probably a trivial question, but my search has been fruitless so far, including this wikipedia article, and the "Compendium of Distributions" document. If $X$ has a uniform distribution, does ...
luchonacho's user avatar
  • 2,742
32 votes
3 answers
102k views

What does "normalization" mean and how to verify that a sample or a distribution is normalized?

I have a question in which it asks to verify whether if the Uniform distribution (${\rm Uniform}(a,b)$) is normalized. For one, what does it mean for any distribution to be normalized? And two, how ...
Ada's user avatar
  • 479
31 votes
10 answers
3k views

What are the myths associated with linear regression, data transformations?

I have been encountering many assumptions associated with linear regression (especially ordinary least squares regression) which are untrue or unnecessary. For example: independent variables must ...
31 votes
2 answers
96k views

Transforming variables for multiple regression in R

I am trying to perform a multiple regression in R. However, my dependent variable has the following plot: Here is a scatterplot matrix with all my variables (...
zgall1's user avatar
  • 869
29 votes
7 answers
70k views

Why is gender typically coded 0/1 rather than 1/2, for example?

I understand the logic of coding for data analysis. My question below is on the use of a specific code. Is there a reason why gender is often coded as 0 for female and 1 for male? Why is this coding ...
Adhesh Josh's user avatar
  • 3,255
29 votes
1 answer
3k views

How does entropy depend on location and scale?

The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals $$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\...
whuber's user avatar
  • 323k
28 votes
2 answers
58k views

Transform Data to Desired Mean and Standard Deviation

I am looking for a method to transform my dataset from its current mean and standard deviation to a target mean and a target standard deviation. Basically, I want to shrink/expand the dispersion and ...
zdebruine's user avatar
  • 241
27 votes
3 answers
7k views

How to model this odd-shaped distribution (almost a reverse-J)

My dependent variable shown below doesn't fit any stock distribution that I know of. Linear regression produces somewhat non-normal, right-skewed residuals that relate to predicted Y in an odd way (...
rolando2's user avatar
  • 12.5k
27 votes
1 answer
20k views

Standardized VS centered variables

I have found many useful posts about standardized independent variables and centered independent variables on stats.stackexchange.com, but I am still a bit confused. I am asking you an evaluation of ...
Fuca26's user avatar
  • 1,097
26 votes
3 answers
59k views

Column-wise matrix normalization in R [closed]

I would like to perform column-wise normalization of a matrix in R. Given a matrix m, I want to normalize each column by dividing each element by the sum of the ...
mavam's user avatar
  • 690
26 votes
2 answers
31k views

Why log-transforming the data before performing principal component analysis?

Im following a tutorial here: http://www.r-bloggers.com/computing-and-visualizing-pca-in-r/ to gain a better understanding of PCA. The tutorial uses the Iris dataset and applies a log transform prior ...
Marc van der Peet's user avatar
26 votes
4 answers
26k views

"Normalizing" variables for SVD / PCA

Suppose we have $N$ measurable variables, $(a_1, a_2, \ldots, a_N)$, we do a number $M > N$ of measurements, and then wish to perform singular value decomposition on the results to find the axes of ...
Szabolcs's user avatar
  • 1,388
26 votes
1 answer
46k views

Is a log transformation a valid technique for t-testing non-normal data?

In reviewing a paper, the authors state, "Continuous outcome variables exhibiting a skewed distribution were transformed, using the natural logarithms, before t tests were conducted to satisfy the ...
CLS's user avatar
  • 361
25 votes
3 answers
4k views

Why aren't power or log transformations taught much in machine learning?

Machine learning (ML) uses linear and logistic regression techniques heavily. It also relies on feature engineering techniques (feature transform, ...
WeiChing 林煒清's user avatar
24 votes
3 answers
27k views

How to interpret regression coefficients when response was transformed by the 4th root?

I'm using fourth root (1/4) power transformation on my response variable, as a result of heteroscedasticity. But now I'm not sure how to interpret my regression ...
user13968's user avatar
  • 351
24 votes
5 answers
61k views

What could be the reason for using square root transformation on data?

What is the primary reason that someone would apply the square root transformation to their data? I always observe that doing this always increases the $R^2$. However, this is probably just due to ...
MarkDollar's user avatar
  • 5,955
24 votes
2 answers
73k views

How to transform negative values to logarithms?

I would like to know how to transform negative values to log(), since I have heteroskedastic data. I've read that log(x+1) ...
Darwin PC's user avatar
  • 541
24 votes
6 answers
2k views

Advanced regression modeling examples

I'm looking for an advanced linear regression case study illustrating the steps required to model complex, multiple non-linear relationships using GLM or OLS. It is surprisingly difficult to find ...
Robert Kubrick's user avatar
23 votes
2 answers
32k views

Transforming proportion data: when arcsin square root is not enough

Is there a (stronger?) alternative to the arcsin square root transformation for percentage/proportion data? In the data set I'm working on at the moment, marked heteroscedasticity remains after I ...
Freya Harrison's user avatar
23 votes
2 answers
23k views

Are log difference time series models better than growth rates?

Often I see authors estimate a "log difference" model, e.g. $\log (y_t)-\log(y_{t-1}) = \log(y_t/y_{t-1}) = \alpha + \beta x_t$ I agree this is appropriate to relate $x_t$ to a percentage change in $...
A. Smith's user avatar
  • 231
22 votes
5 answers
47k views

Interpretation of log transformed predictors in logistic regression

One of the predictors in my logistic model has been log transformed. How do you interpret the estimated coefficient of the log transformed predictor and how do you calculate the impact of that ...
user avatar
22 votes
3 answers
9k views

Comparing AIC of a model and its log-transformed version

The essence of my question is this: Let $Y \in \mathbb{R}^n$ be a multivariate normal random variable with mean $\mu$ and covariance matrix $\Sigma$. Let $Z := \log(Y)$, i.e. $Z_i = \log(Y_i), i \...
Stijn's user avatar
  • 1,880
21 votes
2 answers
9k views

Why is GLM different than an LM with transformed variable

As explained in this course handout (page 1), a linear model can be written in the form: $$ y = \beta_1 x_{1} + \cdots + \beta_p x_{p} + \varepsilon_i,$$ where $y$ is the response variable and $x_{...
Remi.b's user avatar
  • 5,112
21 votes
4 answers
17k views

Transformation to increase kurtosis and skewness of normal r.v

I'm working on an algorithm that relies on the fact that observations $Y$s are normally distributed, and I would like to test the robustness of the algorithm to this assumption empirically. To do ...
Matteo Fasiolo's user avatar

1
2 3 4 5
49