Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
4
votes
1answer
5k views

PDF of cosine of a uniform random variable

There is a formula for the density of the cosine of random variable that's a uniform on $(-\pi,\pi)$ as discussed in this page: $f_{Y}(y) = \dfrac{1}{\pi \sin(\cos^{-1}y)}, y \in\ [-1,1]$ Can anyone ...
4
votes
1answer
2k views

Should quantitative predictors be transformed to be normally distributed?

I am always struggling with normality testing for quantitative predictors (no factors) and transforming them to normality. If I am running a GLMM and my predictors are really non-normal, should I ...
3
votes
4answers
2k views

Appropriate data transformation

I have two dependent variables y1 and y2 with highly skewed distributions. In order to do ANOVA, I was trying to transform the ...
3
votes
2answers
301 views

Natural log approximation

I've got an equation that contains $$x^p - 1$$ $x$ is any positive number (such as 2) and $p$ is a small positive number close to 0 (such as 0.001). For some reason (that I may have known in High ...
61
votes
2answers
51k views

One-hot vs dummy encoding in Scikit-learn

There are two different ways to encoding categorical variables. Say, one categorical variable has n values. One-hot encoding converts it into n variables, while dummy encoding converts it into n-1 ...
21
votes
2answers
32k views

Is a log transformation a valid technique for t-testing non-normal data?

In reviewing a paper, the authors state, "Continuous outcome variables exhibiting a skewed distribution were transformed, using the natural logarithms, before t tests were conducted to satisfy the ...
15
votes
2answers
487 views

Intuition behind Box-Cox transform

For features that are heavily skewed, the Transformation technique is useful to stabilize variance, make the data more normal distribution-like, improve the validity of measures of association. I am ...
35
votes
4answers
33k views

ANOVA on binomial data

I am analyzing an experimental data set. The data consists of a paired vector of treatment type and a binomial outcome: ...
11
votes
3answers
3k views

Are these formulas for transforming P, LSD, MSD, HSD, CI, to SE as an exact or inflated/conservative estimate of $\hat{\sigma}$ correct?

Background I am conducting a meta-analysis that includes previously published data. Often, differences between treatments are reported with P-values, least significant differences (LSD), and other ...
9
votes
1answer
11k views

Standardizing features when using LDA as a pre-processing step

If a multi-class Linear Discriminant Analysis (or I also read Multiple Discriminant Analysis sometimes) is used for dimensionality reduction (or transformation after dimensionality reduction via PCA), ...
9
votes
2answers
5k views

Help me fit this non-linear multiple regression that has defied all previous efforts

EDIT: Since making this post, I have followed up with an additional post here. Summary of the text below: I am working on a model and have tried linear regression, Box Cox transformations and GAM but ...
8
votes
1answer
5k views

How is the Box-Cox transformation valid?

The Box-Cox transformation transforms our data into a normal distribution. How is that even a proper technique? What if our data didn't come from a normal distribution? How could someone just blindly ...
9
votes
4answers
20k views

Box Cox Transforms for regression

I'm trying to fit a linear model on some data with just one predictor (say (x,y)). The data is such that for small values of x, the y values give a tight fit to a straight line, however as x values ...
7
votes
2answers
7k views

Back-transformation and interpretation of $\log(X+1)$ estimates in multiple linear regression

I have performed multiple linear regression analyses with different combinations of transformed and untransformed variables--both explanatory (independent) and response (dependent) variables. All ...
1
vote
2answers
475 views

How to identify models as linear or non-linear?

Identify the following models as linear or non-linear. In case of a non-linear model, reduce the model into a linear model by a suitable transformation. $$\eqalign{ (a)\quad&y=\beta_0+\beta_1 x+\...
5
votes
2answers
4k views

How to transform data to normality?

We have financial some data (500-1000 samples), which is not normally distributed (well known fact from the literature). I have some ideas to do parametric transformations of this data (using some ...
2
votes
1answer
195 views

log transform fixed PH in Cox model - how?

I have survival data to which I am fitting a Cox model with a continuous predictor. The cumulative martingale residual method (supremum test) of Lin, Wei and Ying suggested that both proportional ...
1
vote
3answers
333 views

How is missing data calculated on Likert Scale?

I have the following scales: 1 (Strongly Disagree); 2 (Disagree); 3 (Not Applicable); 4 (Agree); 5 (Strongly Agree); Questionnaire responses: Question Item1: 2; Question Item2: 3; Question Item3: 1;...
1
vote
1answer
393 views

Skewness transformation for one but not the other variable?

I have two dependent variables, one is positively skewed (significantly), the other is negatively skewed (not significant). I can apply a log10 transformation to improve the skewness on the first one. ...
32
votes
5answers
60k views

How to change data between wide and long formats in R? [closed]

You can have data in wide format or in long format. This is quite an important thing, as the useable methods are different, depending on the format. I know you have to work with ...
15
votes
1answer
21k views

How to deal with a mix of binary and continuous inputs in neural networks? [duplicate]

I'm using the nnet package in R to attempt to build an ANN to predict real estate prices for condos (personal project). I am new to this and don't have a math background so please bare with me. I ...
4
votes
1answer
11k views

transformation to normality of the dependent variable in multiple regression

Is it really important to normalize dependent variables in multiple regression or are there any exceptions? My model is providing better results with more significant hypothesis when the DVs are not ...
17
votes
5answers
45k views

What could be the reason for using square root transformation on data?

What is the primary reason that someone would apply the square root transformation to their data? I always observe that doing this always increases the $R^2$. However, this is probably just due to ...
13
votes
1answer
10k views

Standardized VS centered variables

I have found many useful posts about standardized independent variables and centered independent variables on stats.stackexchange.com, but I am still a bit confused. I am asking you an evaluation of ...
11
votes
1answer
23k views

When to transform predictor variables when doing multiple regression?

I'm currently taking my first applied linear regression class at the graduate level, and am struggling with predictor variable transformations in multiple linear regression. The text I'm using, Kutner ...
13
votes
2answers
23k views

Var(X) is known, how to calculate Var(1/X)?

If I have only $\mathrm{Var}(X)$, how can I calculate $\mathrm{Var}(\frac{1}{X})$? I do not have any information about the distribution of $X$, so I cannot use transformation, or any other methods ...
9
votes
3answers
11k views

Linear regression effect sizes when using transformed variables

When performing linear regression, it is often useful to do a transformation such as log-transformation for the dependent variable to achieve better normal distribution conformation. Often it is also ...
19
votes
2answers
5k views

Comparing AIC of a model and its log-transformed version

The essence of my question is this: Let $Y \in \mathbb{R}^n$ be a multivariate normal random variable with mean $\mu$ and covariance matrix $\Sigma$. Let $Z := \log(Y)$, i.e. $Z_i = \log(Y_i), i \...
17
votes
3answers
5k views

Extract data points from moving average?

Is it possible to extract data points from moving average data? In other words, if a set of data only has simple moving averages of the previous 30 points, is it possible to extract the original data ...
12
votes
1answer
41k views

Why use logged variables?

Probably, this is a very basic question but I don't seem to be able to find a solid answer for it. I hope here, I can. I'm currently reading papers as a preparation for my own master's thesis. ...
13
votes
2answers
5k views

Covariance of transformed random variables

I have two random variables $X > 0$ and $Y > 0$. Given that I can estimate $$\text{Cov}(X, Y),$$ how can I estimate $$\text{Cov}(\log(X), \log(Y))?$$
11
votes
4answers
2k views

Using Decibels in Statistics

I'm working on a project that involves reading RFID Tags and comparing the signal strength the reader sees when you change the antenna configuration (number of antenna, position, etc...). As part of ...
10
votes
1answer
6k views

I log transformed my dependent variable, can I use GLM normal distribution with LOG link function?

I have a question concerning Generalized Linear Models (GLM).My dependent variable (DV) is continuous and not normal. So I log transformed it (still not normal but improved it). I want to relate the ...
8
votes
3answers
1k views

Are ecologists the only ones who didn't know that the arcsine is asinine?

Proportion, ratio, and percentage data is very common in ecology (eg, % of flowers pollinated, male:female sex ratio, % mortality in response to a treatment, % of leaf eaten by an herbivore). An ...
7
votes
1answer
9k views

How to use time dependent covariates with cox regression in R

I don't know how to generate time dependent covariates in R for use cox regression. I know you need to reorganize your dataset into intervals between event times. This I believe I can do with the ...
7
votes
1answer
5k views

Non-normally distributed data - Box-Cox transformation?

I have data that is not normally distributed. The problem seems to be that there are too many of one value relative to other values. What I have tried to make data normal: I have tried a log ...
13
votes
4answers
8k views

Boxplot equivalent for heavy-tailed distributions?

For approximately normally distributed data, boxplots are a great way to quickly visualize the median and spread of the data, as well as the presence of any outliers. However for more heavy-tailed ...
7
votes
1answer
1k views

Do the principal components change if we apply PCA more than once (recursively) on data?

Consider a set $X=(X_1; \dots; X_n)$ of $n$ data points such that $X_i \in \mathbb{R}^d$ is a column vector. Let $Y = \text{pca_proj}(X)$ denote the projection of points in $X$ according to the PCA ...
6
votes
2answers
3k views

Is it possible to convert a Rayleigh distribution into a Gaussian distribution?

...and how might we do this? If possible, I am curious if outliers in the Rayleigh distributed data would also remain outliers in the new Gaussian distributed data. Thanks.
3
votes
1answer
610 views

Why there is no uniform prior for Box-Cox Power Transformed Normal Models

I am trying to get intuition why uniform prior like below will not work for the box-cox model. Box-cox model: $y^{(\phi)}_i \sim N(\mu, \sigma^{2})$ where $y^{(\phi)}_i = (y^{\phi}_i-1)/\phi $ if ...
6
votes
1answer
6k views

How to transform continuous data with extreme bimodal distribution

Is there a way to transform a continuous predictor variable (grant) that has a bimodal distribution into a normal distribution (see density plot below)? I have tried log(x+c), z-score and inverse ...
4
votes
1answer
2k views

Linear regression and prediction on transformed data

I have a 7 points dataset that looks like this: ...
2
votes
1answer
172 views

Freeman and Tukey interval

If $Y$ is Poisson$(\lambda)$ distributed, how do we obtain this confidence interval for $\lambda$: $$ \left (\frac{(\sqrt Y+\sqrt{Y + 1}-z_{(α⁄2)} )^2 - 1}{4},\ \frac{(\sqrt Y+\sqrt{Y +1}+z_{(α⁄2)} ...
186
votes
9answers
684k views

How to summarize data by group in R? [closed]

I have R data frame like this: ...
32
votes
2answers
22k views

What are the assumptions of negative binomial regression?

I'm working with a large data set (confidential, so I can't share too much), and came to the conclusion a negative binomial regression would be necessary. I've never done a glm regression before, and ...
23
votes
6answers
2k views

Advanced regression modeling examples

I'm looking for an advanced linear regression case study illustrating the steps required to model complex, multiple non-linear relationships using GLM or OLS. It is surprisingly difficult to find ...
17
votes
2answers
20k views

Why log-transforming the data before performing principal component analysis?

Im following a tutorial here: http://www.r-bloggers.com/computing-and-visualizing-pca-in-r/ to gain a better understanding of PCA. The tutorial uses the Iris dataset and applies a log transform prior ...
15
votes
4answers
1k views

Pitfalls to avoid when transforming data?

I achieved a strong linear relationship between my $X$ and $Y$ variable after doubly transforming the response. The model was $Y\sim X$ but I transformed it to $\sqrt{\frac{Y}{X}}\sim \sqrt{X}$ ...
24
votes
3answers
4k views

Why aren't power or log transformations taught much in machine learning?

Machine learning (ML) uses linear and logistic regression techniques heavily. It also relies on feature engineering techniques (feature transform, ...
5
votes
2answers
10k views

Interpreting test results on log-transformed data

I have data that is not normally distributed. I can log-transform it to be normally distributed, and then perform, for example, a t-test. But how do I interpret the results of the t-test? Do I have ...

1
2
3 4 5
7