# Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

838 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1k views

### Feature scaling for non-negative sparse data

Imagine you have many observations on which you want to run a classification algorithm. Each observation is characterized by a matrix of non-negative values. For all observations 90-98% of the values ...
• 5,126
700 views

### Transformation bias with non-linear functions

This a more general question: I often deal with experimental data (subject to uncertainties in the measurements) that have to be transformed using a function, to calculate a parameter (which can, for ...
• 179
479 views

### T tests on proportions - Wrong, but how wrong?

Background: In psychology, and probably a number of other disciplines, it's common practice to test between-groups effects on a binary variable, such as accuracy, by aggregating data within ...
• 9,475
410 views

### What, if any, dissimilarity is preserved in partial least squares (PLS)?

When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
• 48.9k
970 views

### Box constraints with BFGS algorithm

I've been a long time adept of the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS), which I trusted to be a pretty efficient local optimisation technique. And indeed it is. The problem I usually ...
• 1,556
149 views

### Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
• 13.1k
9k views

### Is there a way to prevent forecasting negative values with ARIMA (or add constrains) in R?

Currently I'm using the ARIMA provided in R, the training series is a seasonal time series, with some values close to zero in each period, and I find that when the training series have a descending ...
• 245
495 views

### Algorithms for data symmetrization

There are statistical methods (e.g. by Box-Cox or Yeo-Johnson, see references below) to automatically bring data vectors as close as possible to symmetry/normality using optimal power transformations. ...
• 12k
2k views

### Why would SVD be 'unstable' if you don't standardize your data first?

I'm reading an article about Direct Linear Transformation which processes data using SVD, and the data set is standardized so that it has zero mean and unit standard deviation (n.b., some people call ...
• 3,633
824 views

### Factor analysis across different levels of data aggregation

I have survey data for thousands of individuals from hundreds of towns. I want to identify factors underlying certain characteristics at the town level and the individual level. The individual level ...
• 1,312
6k views

### Linear mixed effects models: what to do when the residual QQ-plot looks non-normal?

I have four linear mixed effect models of similar structure: ...
114 views

### Techniques for addressing the homoskedasticity and normality assumption violations in mixed models with a non-all-positive response variable

I have a mixed model which the heteroskedasticity and normality assumptions for the residuals are violated. Up to this point, I have been addressing that by using the ...
• 273
2k views

### Methods to extract signal from three very noisy time series of same event

I have three time series of same length, all containing magnitude measurements of the same event "A". But each time series is using a different method of measurement. My goal is to merge the ...
• 83
802 views

### Data transformation : bimodal feature

I have a data feature that follows closely a bimodal distribution (mixture of two separate normal distributions with different mean, standard deviation and weights). Is it meaningful to transform that ...
• 51
2k views

### Quantile transform vs Power transformation to get normal distribution

I was introduced to the concept of quantile-based gaussian transform. To my understanding, it changes the value of the original data by each percentile to the matching percentile of gaussian ...
• 1,041
104 views

### Is there a name for a $y=\sqrt[k]{x}$-like data normalization?

I'm normalizing multivariate numeric data that has both negative and positive values. For the sake of the question let's assume a range of e.g. $[-10000,10000]$ with a lot of values in $[-1,1]$. I've ...
• 3,911
174 views

### More effective methods for transforming to normality

I'm in a field that is overly concerned with transforming non-normal variables in an attempt to make them normal. However, it's also generally recognized that the standard transformations (e.g. log, ...
• 594
2k views

### How to compare transformed and untransformed linear models?

I have a linear model which doesn't have any particular issues with its assumptions (diagnostics plots look well). However it has a slighly skewed response (skewness approx. 0.5) and few skew ...
• 215
58 views

### Is there a systematic procedure to do data cleaning or preprocessing?

I do data cleaning / data preprocessing everyday, using various tools to remove outliers and to keep normality However, I feel what I am doing is more a handcraft rather than systematic: I don't ...
• 1,499
47 views

### Data Transformation Question - Multiplying data proportional to demographics

I have a bunch of data that is tied to demographic variables (Age, Sex, Income, Education, etc.). However, the data is sent by one person in a household for the entire house. It's numerical data and I ...
• 41
140 views

### Categorical PCA: Merge categories based on Transformation Plots?

A tutorial on categorical pca (CATPCA) (Linting et al. 2012) explains that a decision to merge categories of an ordinal variable can be made based on the category quantification ("none of the ...
• 453
163 views

### Reduced degrees of freedom using transformed data in one-way ANOVA (SAS PROC MIXED)

On pg. 88 of Design and Analysis of Experiments (8th Ed.) by Montgomery, he's analyzing square root transformed data in a one-way ANOVA. He provides an ANOVA table (SS, d.f., MS, F, p) for these data, ...
• 1,873
6k views

### Box cox for mixed models in R

Consider a mixed model generated using the lme function in R. How can I consider the Box-cox transformations of this model in R? I have seen similar questions being asked before but they did not give ...
• 41
1k views

### Variable Transformation using Cumulative Distribution Function (CDF)

Consider two different data time-series, Data1 and Data2, expressed using inhomogeneous scales (units). Each of these two data series is itself a weighted-average of a bunch of standardized ...
• 957
126 views

### Splitting a variable with nominal and numeric values

I have a variable that has both numeric and nominal components. The source has a documentation which helps in identifying which is which and for splitting into their proper components. I will do ...
• 147
450 views

• 58.3k
103 views

### Weibull regression sensitive to scaling of predictors?

I'm running a Weibull regression and decided to simulate data to assure myself that the model is able to recover the true parameters. ...
• 205
102 views

### Projecting an image matrix onto another image matrix

This question is an attempt to re-frame another question on this forum that was of interest to a user dealing with two-dimensional image data from two scanners. If I have misunderstood the linked ...
• 129k
175 views

### Robust regression after using Box cox transformation

Is that making sense to apply robust regression after using Box Cox transformation. In my data, it seems by using log transformation I can improve the model since I have the violation of the ...
• 61
709 views

### How to untransform/interpret results after a Lambert W transformation?

Suppose I have some heavy-tailed data that I want to transform so it's roughly normal in order to perform a t-test. ...
• 415