# Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

700 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
786 views

### Feature scaling for non-negative sparse data

Imagine you have many observations on which you want to run a classification algorithm. Each observation is characterized by a matrix of non-negative values. For all observations 90-98% of the values ...
362 views

### T tests on proportions - Wrong, but how wrong?

Background: In psychology, and probably a number of other disciplines, it's common practice to test between-groups effects on a binary variable, such as accuracy, by aggregating data within ...
393 views

### What, if any, dissimilarity is preserved in partial least squares (PLS)?

When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
374 views

### Transformation bias with non-linear functions

This a more general question: I often deal with experimental data (subject to uncertainties in the measurements) that have to be transformed using a function, to calculate a parameter (which can, for ...
112 views

### Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
330 views

### Algorithms for data symmetrization

There are statistical methods (e.g. by Box-Cox or Yeo-Johnson, see references below) to automatically bring data vectors as close as possible to symmetry/normality using optimal power transformations. ...
512 views

### Box constraints with BFGS algorithm

I've been a long time adept of the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS), which I trusted to be a pretty efficient local optimisation technique. And indeed it is. The problem I usually ...
51 views

### Is there a systematic procedure to do data cleaning or preprocessing?

I do data cleaning / data preprocessing everyday, using various tools to remove outliers and to keep normality However, I feel what I am doing is more a handcraft rather than systematic: I don't ...
46 views

### Data Transformation Question - Multiplying data proportional to demographics

I have a bunch of data that is tied to demographic variables (Age, Sex, Income, Education, etc.). However, the data is sent by one person in a household for the entire house. It's numerical data and I ...
109 views

### Categorical PCA: Merge categories based on Transformation Plots?

A tutorial on categorical pca (CATPCA) (Linting et al. 2012) explains that a decision to merge categories of an ordinal variable can be made based on the category quantification ("none of the ...
82 views

### Reduced degrees of freedom using transformed data in one-way ANOVA (SAS PROC MIXED)

On pg. 88 of Design and Analysis of Experiments (8th Ed.) by Montgomery, he's analyzing square root transformed data in a one-way ANOVA. He provides an ANOVA table (SS, d.f., MS, F, p) for these data, ...
630 views

130 views

### clustering for histogram shapes

I am trying to get a start on a clustering problem. The sample data is trade volume at a particular price. Some notes about the data: number of bins vary from sample to sample (larger price range ...