Questions tagged [data-transformation]
Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.
862
questions with no upvoted or accepted answers
16
votes
2
answers
932
views
Visualizing many left-skewed distributions
I have a series of left-skewed/heavy tailed distributions that I would like to show. There are 42 distributions across three factors (labeled as A, ...
- 5,960
8
votes
0
answers
1k
views
Feature scaling for non-negative sparse data
Imagine you have many observations on which you want to run a classification algorithm. Each observation is characterized by a matrix of non-negative values. For all observations 90-98% of the values ...
- 5,056
7
votes
0
answers
663
views
Transformation bias with non-linear functions
This a more general question: I often deal with experimental data (subject to uncertainties in the measurements) that have to be transformed using a function, to calculate a parameter (which can, for ...
- 179
7
votes
0
answers
474
views
T tests on proportions - Wrong, but how wrong?
Background: In psychology, and probably a number of other disciplines, it's common practice to test between-groups effects on a binary variable, such as accuracy, by aggregating data within ...
- 8,227
7
votes
0
answers
410
views
What, if any, dissimilarity is preserved in partial least squares (PLS)?
When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
- 45.7k
6
votes
0
answers
927
views
Box constraints with BFGS algorithm
I've been a long time adept of the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS), which I trusted to be a pretty efficient local optimisation technique. And indeed it is.
The problem I usually ...
- 1,526
5
votes
1
answer
3k
views
What is the most appropriate transformation method for performing analyses on species composition data?
I would like to compare differences in fish diets between sampling sites using a Bray-Curtis dissimilarity matrix and non-metric multidimensional scaling techniques. My raw data consists of counts of ...
- 51
5
votes
0
answers
149
views
Censored logit transform for (ad hoc) exploratory data analysis
In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
- 12.6k
5
votes
0
answers
479
views
Algorithms for data symmetrization
There are statistical methods (e.g. by Box-Cox or Yeo-Johnson, see references below) to automatically bring data vectors as close as possible to symmetry/normality using optimal power transformations.
...
- 11.6k
5
votes
0
answers
2k
views
Why would SVD be 'unstable' if you don't standardize your data first?
I'm reading an article about Direct Linear Transformation which processes data using SVD, and the data set is standardized so that it has zero mean and unit standard deviation (n.b., some people call ...
- 3,479
5
votes
0
answers
794
views
Factor analysis across different levels of data aggregation
I have survey data for thousands of individuals from hundreds of towns. I want to identify factors underlying certain characteristics at the town level and the individual level. The individual level ...
- 1,272
5
votes
1
answer
6k
views
Linear mixed effects models: what to do when the residual QQ-plot looks non-normal?
I have four linear mixed effect models of similar structure:
...
4
votes
0
answers
104
views
Techniques for addressing the homoskedasticity and normality assumption violations in mixed models with a non-all-positive response variable
I have a mixed model which the heteroskedasticity and normality assumptions for the residuals are violated. Up to this point, I have been addressing that by using the ...
- 263
4
votes
0
answers
735
views
Data transformation : bimodal feature
I have a data feature that follows closely a bimodal distribution (mixture of two separate normal distributions with different mean, standard deviation and weights).
Is it meaningful to transform that ...
- 51
4
votes
0
answers
2k
views
Quantile transform vs Power transformation to get normal distribution
I was introduced to the concept of quantile-based gaussian transform. To my understanding, it changes the value of the original data by each percentile to the matching percentile of gaussian ...
- 1,011
4
votes
0
answers
100
views
Is there a name for a $y=\sqrt[k]{x}$-like data normalization?
I'm normalizing multivariate numeric data that has both negative and positive values. For the sake of the question let's assume a range of e.g. $[-10000,10000]$ with a lot of values in $[-1,1]$. I've ...
- 3,891
4
votes
0
answers
161
views
More effective methods for transforming to normality
I'm in a field that is overly concerned with transforming non-normal variables in an attempt to make them normal. However, it's also generally recognized that the standard transformations (e.g. log, ...
- 594
4
votes
0
answers
2k
views
How to compare transformed and untransformed linear models?
I have a linear model which doesn't have any particular issues with its assumptions (diagnostics plots look well). However it has a slighly skewed response (skewness approx. 0.5) and few skew ...
- 215
4
votes
0
answers
54
views
Is there a systematic procedure to do data cleaning or preprocessing?
I do data cleaning / data preprocessing everyday, using various tools to remove outliers and to keep normality
However, I feel what I am doing is more a handcraft rather than systematic: I don't ...
- 1,489
4
votes
0
answers
47
views
Data Transformation Question - Multiplying data proportional to demographics
I have a bunch of data that is tied to demographic variables (Age, Sex, Income, Education, etc.). However, the data is sent by one person in a household for the entire house. It's numerical data and I ...
- 41
4
votes
0
answers
8k
views
Is there a way to prevent forecasting negative values with ARIMA (or add constrains) in R?
Currently I'm using the ARIMA provided in R, the training series is a seasonal time series, with some values close to zero in each period, and I find that when the training series have a descending ...
- 235
4
votes
0
answers
138
views
Categorical PCA: Merge categories based on Transformation Plots?
A tutorial on categorical pca (CATPCA) (Linting et al. 2012) explains that a decision to merge categories of an ordinal variable can be made based on the category quantification ("none of the ...
- 443
4
votes
1
answer
153
views
Reduced degrees of freedom using transformed data in one-way ANOVA (SAS PROC MIXED)
On pg. 88 of Design and Analysis of Experiments (8th Ed.) by Montgomery, he's analyzing square root transformed data in a one-way ANOVA. He provides an ANOVA table (SS, d.f., MS, F, p) for these data, ...
- 1,823
4
votes
0
answers
6k
views
Box cox for mixed models in R
Consider a mixed model generated using the lme function in R. How can I consider the Box-cox transformations of this model in R? I have seen similar questions being asked before but they did not give ...
- 41
4
votes
0
answers
1k
views
Variable Transformation using Cumulative Distribution Function (CDF)
Consider two different data time-series, Data1 and Data2, expressed using inhomogeneous scales (units). Each of these two data series is itself a weighted-average of a bunch of standardized ...
- 957
4
votes
0
answers
126
views
Splitting a variable with nominal and numeric values
I have a variable that has both numeric and nominal components. The source has a documentation which helps in identifying which is which and for splitting into their proper components. I will do ...
- 147
4
votes
0
answers
447
views
On nonlinear regression, fits, and transformations
I am trying to fit a nonlinear regression model in R using nls(). I have a form of the equation I want to fit to:
$$y = (a \times x_{1}^c +b \times x_{2}^d) (x_{3}^...
- 41
4
votes
0
answers
309
views
Methodology for validation of stochastic simulations with Kolmogorov-Smirnov test
I'm a phd student in Geography, i need some help (or good ressources) to understand why and when i need to use PIT (Probability integral transform) in my validation program for simulation.
I explain ...
- 175
4
votes
0
answers
912
views
Interpretation of a log likelihood function for PROC NLMIXED in SAS
I have a data set of skewed nutrient intake values, from around 7800 individuals, of whom around 3000 had two measures of daily nutrient intake (the others only had one measure), so this is a repeated ...
- 3,700
4
votes
1
answer
143
views
Remove Outlying Data with a Different Trend
I currently have many sets of data that display more or less the trend in the image, which may be due to abnormalities of the data source. The series "splits" into two different trends, with one ...
- 41
3
votes
0
answers
49
views
Is SVM rotation invariant?
Let's say we have some data X and we want to find a linear separator using soft SVM with l2 regularization, and then we want to solve the same problem after applying some rotation matrix Q to the data ...
- 31
3
votes
0
answers
74
views
How to apply the Jacobian correction to AIC for a transformed dependent variable when the transformation includes an independent variable?
I am comparing several OLS multivariate regression models of a dependent variable (we'll call it $Y$) using various transformations, some of which also involve one of the independent variables ($X_1$)....
3
votes
3
answers
1k
views
Methods to extract signal from three very noisy time series of same event
I have three time series of same length, all containing magnitude measurements of the same event "A". But each time series is using a different method of measurement.
My goal is to merge the ...
- 43
3
votes
1
answer
71
views
Why is a non-linear transformation a parameter?
This in reference to my answer at What model should I use to prove statistical significance?.
I test the correlation between x and ...
- 4,110
3
votes
0
answers
355
views
Normalizing zero inflated predictors for multiple regression
Hope I got it right, as this is my first active post :-)
I was trying to find a solution the whole day for my problem.
I am trying to predict a continuos variable based on 20 different predictors. The ...
- 31
3
votes
0
answers
169
views
log transform in linear regression
Assume we have a data set and the theory suggests to model $Y \sim X$. We apply a simple linear regression and get the following:
Next, let us make a log transform of both $X$ and $Y$. The result is ...
- 426
3
votes
0
answers
25
views
How to deal with data from different centers?
I want to examine the psychometric properties (e.g. internal consistency, factorial validity) of a scale. The data (N=700) comes from 16 different locations. Central means and variances did not differ ...
- 63
3
votes
0
answers
51
views
(G)LM prediction interval with heteroscedasticity
I am trying to get prediction intervals from some non-linear data which also exhibits heteroscedasticity.
...
- 7,438
3
votes
0
answers
2k
views
How to back transform from log10 ~ log10 regression in order to predict?
I would like to understand the results of this paper (they supply all of their R code and raw data here). The idea is to regress home range on body mass for a range of taxa.
For one group of birds, ...
- 899
3
votes
1
answer
57
views
Combining data with differing dependent variables
Suppose we have two feature matrices, $X_1$ and $X_2$, with response variables $Y_1$ and $Y_2.$ Where $X_1$ and $X_2$ have the same feature columns, but distinct observations. Furthermore, $Y_1$ and $...
3
votes
0
answers
771
views
Are Box-Cox and differencing redundant or complementary?
I was always under the impression that differencing and Box-Cox were two ways to achieve the same goal: Making a time series stationary so that it can be modeled using an ARMA process.
However, ...
- 1,371
3
votes
1
answer
97
views
why has author divided by 1.5 in hands on machine learning with scikit learn
I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow (76/718), and the author is talking about dividing the dataset into a test set which i follow, but then goes on to talk about ...
- 31
3
votes
2
answers
1k
views
How to weight observations to transform a distribution into normal?
Suppose X is a variable which follows some distribution (non normal) then how to define $f(X(k))$ (f is some functions of the variable X) such that
$$f(X)X$$ is normally distributed and $0\leq f(X(k))...
- 93
3
votes
0
answers
889
views
High proportion of zero values and PCA
My aim is to perform PCA since I have 76 variables in my dataset. Problem is that most of my variables are highly skewed as you can see in the histogram below.
These variables are proportions derived ...
- 41
3
votes
1
answer
317
views
Dealing with a dataset having target values on different scales?
I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
3
votes
0
answers
1k
views
Obtain within-group Gram matrix out of distance matrix
Gram matrix
Let $\bf X$ be a n x p dataset with columns (variables) centered. Then p x p $\bf X'X$ is the total scatter matrix ...
- 56.5k
3
votes
0
answers
103
views
Weibull regression sensitive to scaling of predictors?
I'm running a Weibull regression and decided to simulate data to assure myself that the model is able to recover the true parameters.
...
- 95
3
votes
0
answers
99
views
Projecting an image matrix onto another image matrix
This question is an attempt to re-frame another question on this forum that was of interest to a user dealing with two-dimensional image data from two scanners. If I have misunderstood the linked ...
- 118k
3
votes
1
answer
3k
views
How to shift my dataset to all positive values before applying BoxCox
I am trying to apply boxcox() function to my dataset but it shows an error saying data must be positive. How can I make my data positive? I have 1009 observations ...
3
votes
0
answers
163
views
Robust regression after using Box cox transformation
Is that making sense to apply robust regression after using Box Cox transformation. In my data, it seems by using log transformation I can improve the model since I have the violation of the ...
- 61