Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
210
votes
4answers
293k views

When (and why) should you take the log of a distribution (of numbers)?

Say I have some historical data e.g., past stock prices, airline ticket price fluctuations, past financial data of the company... Now someone (or some formula) comes along and says "let's take/use ...
187
votes
8answers
380k views

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
51
votes
3answers
58k views

Interpretation of log transformed predictor and/or response

I'm wondering if it makes a difference in interpretation whether only the dependent, both the dependent and independent, or only the independent variables are log transformed. Consider the case of <...
61
votes
5answers
20k views

How small a quantity should be added to x to avoid taking the log of zero?

I have analysed my data as they are. Now I want to look at my analyses after taking the log of all variables. Many variables contain many zeros. Therefore I add a small quantity to avoid taking the ...
207
votes
13answers
157k views

How should I transform non-negative data including zeros?

If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used: $\log(x+1)$ which has the ...
57
votes
3answers
33k views

Box-Cox like transformation for independent variables?

Is there a Box-Cox like transformation for independent variables? That is, a transformation that optimizes the $x$ variable so that the y~f(x) will make a more ...
42
votes
1answer
41k views

Alternatives to one-way ANOVA for heteroskedastic data

I have data from 3 groups of algae biomass ($A$, $B$, $C$) which contain unequal sample sizes ($n_A=15$, $n_B=13$, $n_C=12$) and I would like compare if these groups are from the same population. One-...
47
votes
1answer
37k views

Regression: Transforming Variables

When transforming variables, do you have to use all of the same transformation? For example, can I pick and choose differently transformed variables, as in: Let, $x_1,x_2,x_3$ be age, length of ...
26
votes
3answers
6k views

How to model this odd-shaped distribution (almost a reverse-J)

My dependent variable shown below doesn't fit any stock distribution that I know of. Linear regression produces somewhat non-normal, right-skewed residuals that relate to predicted Y in an odd way (...
28
votes
2answers
81k views

Transforming variables for multiple regression in R

I am trying to perform a multiple regression in R. However, my dependent variable has the following plot: Here is a scatterplot matrix with all my variables (...
58
votes
8answers
47k views

Does it ever make sense to treat categorical data as continuous?

In answering this question on discrete and continuous data I glibly asserted that it rarely makes sense to treat categorical data as continuous. On the face of it that seems self-evident, but ...
24
votes
3answers
12k views

How to perform isometric log-ratio transformation

I have data on movement behaviours (time spent sleeping, sedentary, and doing physical activity) that sums to approximately 24 (as in hours per day). I want to create a variable that captures the ...
51
votes
7answers
97k views

Is it a good practice to always scale/normalize data for machine learning? [duplicate]

My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect ...
64
votes
1answer
23k views

Why is the square root transformation recommended for count data?

It is often recommended to take the square root when you have count data. (For some examples on CV, see @HarveyMotulsky's answer here, or @whuber's answer here.) On the other hand, when fitting a ...
22
votes
2answers
27k views

Transforming proportion data: when arcsin square root is not enough

Is there a (stronger?) alternative to the arcsin square root transformation for percentage/proportion data? In the data set I'm working on at the moment, marked heteroscedasticity remains after I ...
16
votes
1answer
5k views

Back transformation of an MLR model [duplicate]

I've obtained a multiple linear regression model in the form $$ \mathrm{log}(Y) = \beta_0 + \beta_1x_1 + \dots + \beta_4x_4 + \beta_5x_1x_2 + \dots + \beta_{10}x_3x_4 + \beta_{11}x_1^2 + \dots + \...
21
votes
3answers
21k views

How to interpret regression coefficients when response was transformed by the 4th root?

I'm using fourth root (1/4) power transformation on my response variable, as a result of heteroscedasticity. But now I'm not sure how to interpret my regression ...
2
votes
2answers
349 views

log transform fixed PH in Cox model - how?

I have survival data to which I am fitting a Cox model with a continuous predictor. The cumulative martingale residual method (supremum test) of Lin, Wei and Ying suggested that both proportional ...
56
votes
1answer
28k views

How to apply standardization/normalization to train- and testset if prediction is the goal?

Do I transform all my data or folds (if CV is applied) at the same time? e.g. (allData - mean(allData)) / sd(allData) Do I transform trainset and testset ...
24
votes
5answers
54k views

What is the reason the log transformation is used with right-skewed distributions?

I once heard that log transformation is the most popular one for right-skewed distributions in linear regression or quantile regression I would like to know is there any reason underlying this ...
10
votes
2answers
25k views

How to choose the best transformation to achieve linearity?

I want to do multiple linear regression and then to predict new values with little extrapolation. I have my response variable in the range from -2 to +7, and three predictors (the ranges about +10 - +...
8
votes
3answers
3k views

Choosing c such that log(x + c) would remove skew from the population

I have data for which I would like to take the log transformation before doing OLS. The data include zeros. Thus, I want to do a log(x + c). I know a traditional c to choose is 1. I am wondering ...
20
votes
1answer
1k views

How does entropy depend on location and scale?

The entropy of a continuous distribution with density function $f$ is defined to be the negative of the expectation of $\log(f),$ and therefore equals $$H_f = -\int_{-\infty}^{\infty} \log(f(x)) f(x)\...
2
votes
1answer
7k views

What to do with GLM (Gamma) when residuals are not normally distributed?

Until now I have only done very basic/simple simple stats, but now I got stuck in all the literature/tips/forums ... It's about the following problem: I have the following data: ...
8
votes
2answers
8k views

Transformation Chi-squared to Normal distribution

The relationship between the standard normal and the chi-squared distributions is well known. I was wondering though, is there a transformation that can lead from a $\chi^2 (1)$ back to a standard ...
3
votes
2answers
3k views

Mboxcox, interpreting difficult regressions

Mboxcox in Stata suggests transforming my variables using a power of 0.1 for the independent variable, and 0.4 for the dependent variable. I have run the model, and it fixes problems associated with ...
38
votes
4answers
16k views

Analysis with complex data, anything different?

Say for example you are doing a linear model, but the data $y$ is complex. $ y = x \beta + \epsilon $ My data set is complex, as in all the numbers in $y$ are of the form $(a + bi)$. Is there ...
25
votes
3answers
69k views

What does “normalization” mean and how to verify that a sample or a distribution is normalized?

I have a question in which it asks to verify whether if the Uniform distribution (${\rm Uniform}(a,b)$) is normalized. For one, what does it mean for any distribution to be normalized? And two, how ...
18
votes
2answers
17k views

Back-transformation of regression coefficients

I'm doing a linear regression with a transformed dependent variable. The following transformation was done so that the assumption of normality of residuals would hold. The untransformed dependant ...
13
votes
1answer
10k views

What is the most appropriate way to transform proportions when they are an independent variable?

I thought I understood this issue, but now I'm not as sure and I'd like to check with others before I proceed. I have two variables, X and ...
26
votes
7answers
48k views

Why is gender typically coded 0/1 rather than 1/2, for example?

I understand the logic of coding for data analysis. My question below is on the use of a specific code. Is there a reason why gender is often coded as 0 for female and 1 for male? Why is this coding ...
9
votes
2answers
13k views

Testing whether data follows T-Distribution

I am involved in a project where I need to check whether my data follows a T-distribution with N degrees of freedom for a given value of N. I know that Kolmogorov-Smirnoff can be used, but is there ...
7
votes
1answer
1k views

Can I replace NAs based on response variable?

My data consists of 1 response variable 'Age' and 1 feature (beta). The feature contains some missing values (NA) so I want to replace them. I've been replacing them with the median of the feature. ...
21
votes
4answers
12k views

Transformation to increase kurtosis and skewness of normal r.v

I'm working on an algorithm that relies on the fact that observations $Y$s are normally distributed, and I would like to test the robustness of the algorithm to this assumption empirically. To do ...
8
votes
2answers
7k views

Reporting regression statistics after logarithmic transformation

I'm a bit troubled about how to report linear regression statistics after log transformation of the dependent variable. I suppose I should report the transformed coefficient, but would they be easily ...
6
votes
1answer
1k views

How do I find a variance-stabilizing transformation?

I wonder how to solve this classical problem: Recall that for a binomial proportion $\hat p$ based on a sample of size $n$ we have $$E(\hat p)=p$$ and $$\operatorname{Var}(\hat p) = p(1-p)/n.$$ ...
15
votes
3answers
17k views

What is the effect of dichotomising variables?

When dichotomising variables, what information is lost in the process? How does a dichotomisation help in the analyses?
18
votes
4answers
31k views

Interpretation of log transformed predictors in logistic regression

One of the predictors in my logistic model has been log transformed. How do you interpret the estimated coefficient of the log transformed predictor and how do you calculate the impact of that ...
10
votes
4answers
20k views

How to interpret logarithmically transformed coefficients in linear regression?

My situation is: I have 1 continuous dependent and 1 continuous predictor variable that I've logarithmically transformed to normalise their residuals for simple linear regression. I would ...
68
votes
3answers
77k views

When are Log scales appropriate?

I've read that using log scales when charting/graphing is appropriate in certain circumstances, like the y-axis in a time series chart. However, I've not been able to find a definitive explanation as ...
31
votes
4answers
51k views

When to log transform a time series before fitting an ARIMA model

I have previously used forecast pro to forecast univariate time series, but am switching my workflow over to R. The forecast package for R contains a lot of useful functions, but one thing it doesn't ...
17
votes
2answers
6k views

Why is GLM different than an LM with transformed variable

As explained in this course handout (page 1), a linear model can be written in the form: $$ y = \beta_1 x_{1} + \cdots + \beta_p x_{p} + \varepsilon_i,$$ where $y$ is the response variable and $x_{...
12
votes
2answers
12k views

Clustering of very skewed, count data: any suggestions to go about (transform etc)?

Basic problem Here is my basic problem: I am trying to cluster a dataset containing some very skewed variables with counts. The variables contain many zeros and are therefore not very informative for ...
10
votes
4answers
12k views

Why not log-transform all variables that are not of main interest?

Books and discussions often state that when facing problems (of which there are a few) with a predictor, log-transformimg it is a possibility. Now, I understand that this depends on distributions and ...
16
votes
2answers
603 views

Intuition behind Box-Cox transform

For features that are heavily skewed, the Transformation technique is useful to stabilize variance, make the data more normal distribution-like, improve the validity of measures of association. I am ...
19
votes
4answers
21k views

“Normalizing” variables for SVD / PCA

Suppose we have $N$ measurable variables, $(a_1, a_2, \ldots, a_N)$, we do a number $M > N$ of measurements, and then wish to perform singular value decomposition on the results to find the axes of ...
15
votes
2answers
15k views

Transforming Data: All variables or just the non-normal ones?

In Andy Field's Discovering Statistics Using SPSS he states that all variables have to be transformed. However in the publication: "Examining spatially varying relationships between land use and ...
10
votes
5answers
14k views

What other normalizing transformations are commonly used beyond the common ones like square root, log, etc.?

In the analysis of test scores (e.g., in Education or Psychology), common analysis techniques often assume that data are normally distributed. However, perhaps more often than not, scores tend to ...
12
votes
3answers
18k views

How to transform leptokurtic distribution to normality?

Suppose I have a leptokurtic variable that I would like to transform to normality. What transformations can accomplish this task? I am well aware that transforming data may not always be desirable, ...
7
votes
1answer
6k views

Confusion related to which transformation to use

I have this confusion about which transformation to use in my data. The histogram of my original data looks like this Now I have seen at most of the places to take log transformation in case the ...

1
2 3 4 5
7