Skip to main content

Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

838 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
0 answers
1k views

Feature scaling for non-negative sparse data

Imagine you have many observations on which you want to run a classification algorithm. Each observation is characterized by a matrix of non-negative values. For all observations 90-98% of the values ...
pir's user avatar
  • 5,126
7 votes
0 answers
700 views

Transformation bias with non-linear functions

This a more general question: I often deal with experimental data (subject to uncertainties in the measurements) that have to be transformed using a function, to calculate a parameter (which can, for ...
Martin's user avatar
  • 179
7 votes
0 answers
479 views

T tests on proportions - Wrong, but how wrong?

Background: In psychology, and probably a number of other disciplines, it's common practice to test between-groups effects on a binary variable, such as accuracy, by aggregating data within ...
Eoin's user avatar
  • 9,475
7 votes
0 answers
410 views

What, if any, dissimilarity is preserved in partial least squares (PLS)?

When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
Gavin Simpson's user avatar
6 votes
0 answers
970 views

Box constraints with BFGS algorithm

I've been a long time adept of the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS), which I trusted to be a pretty efficient local optimisation technique. And indeed it is. The problem I usually ...
Quantuple's user avatar
  • 1,556
5 votes
0 answers
149 views

Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
GeoMatt22's user avatar
  • 13.1k
5 votes
0 answers
9k views

Is there a way to prevent forecasting negative values with ARIMA (or add constrains) in R?

Currently I'm using the ARIMA provided in R, the training series is a seasonal time series, with some values close to zero in each period, and I find that when the training series have a descending ...
宇宙人's user avatar
  • 245
5 votes
0 answers
495 views

Algorithms for data symmetrization

There are statistical methods (e.g. by Box-Cox or Yeo-Johnson, see references below) to automatically bring data vectors as close as possible to symmetry/normality using optimal power transformations. ...
Michael M's user avatar
  • 12k
5 votes
0 answers
2k views

Why would SVD be 'unstable' if you don't standardize your data first?

I'm reading an article about Direct Linear Transformation which processes data using SVD, and the data set is standardized so that it has zero mean and unit standard deviation (n.b., some people call ...
avocado's user avatar
  • 3,633
5 votes
0 answers
824 views

Factor analysis across different levels of data aggregation

I have survey data for thousands of individuals from hundreds of towns. I want to identify factors underlying certain characteristics at the town level and the individual level. The individual level ...
Dr. Beeblebrox's user avatar
5 votes
1 answer
6k views

Linear mixed effects models: what to do when the residual QQ-plot looks non-normal?

I have four linear mixed effect models of similar structure: ...
Alison Fairbrass's user avatar
4 votes
0 answers
114 views

Techniques for addressing the homoskedasticity and normality assumption violations in mixed models with a non-all-positive response variable

I have a mixed model which the heteroskedasticity and normality assumptions for the residuals are violated. Up to this point, I have been addressing that by using the ...
Katie's user avatar
  • 273
4 votes
4 answers
2k views

Methods to extract signal from three very noisy time series of same event

I have three time series of same length, all containing magnitude measurements of the same event "A". But each time series is using a different method of measurement. My goal is to merge the ...
litmus's user avatar
  • 83
4 votes
0 answers
802 views

Data transformation : bimodal feature

I have a data feature that follows closely a bimodal distribution (mixture of two separate normal distributions with different mean, standard deviation and weights). Is it meaningful to transform that ...
Arnaud's user avatar
  • 51
4 votes
0 answers
2k views

Quantile transform vs Power transformation to get normal distribution

I was introduced to the concept of quantile-based gaussian transform. To my understanding, it changes the value of the original data by each percentile to the matching percentile of gaussian ...
Eric Kim's user avatar
  • 1,041
4 votes
0 answers
104 views

Is there a name for a $y=\sqrt[k]{x}$-like data normalization?

I'm normalizing multivariate numeric data that has both negative and positive values. For the sake of the question let's assume a range of e.g. $[-10000,10000]$ with a lot of values in $[-1,1]$. I've ...
geekoverdose's user avatar
  • 3,911
4 votes
0 answers
174 views

More effective methods for transforming to normality

I'm in a field that is overly concerned with transforming non-normal variables in an attempt to make them normal. However, it's also generally recognized that the standard transformations (e.g. log, ...
TPM's user avatar
  • 594
4 votes
1 answer
2k views

How to compare transformed and untransformed linear models?

I have a linear model which doesn't have any particular issues with its assumptions (diagnostics plots look well). However it has a slighly skewed response (skewness approx. 0.5) and few skew ...
jakes's user avatar
  • 215
4 votes
0 answers
58 views

Is there a systematic procedure to do data cleaning or preprocessing?

I do data cleaning / data preprocessing everyday, using various tools to remove outliers and to keep normality However, I feel what I am doing is more a handcraft rather than systematic: I don't ...
user152503's user avatar
  • 1,499
4 votes
0 answers
47 views

Data Transformation Question - Multiplying data proportional to demographics

I have a bunch of data that is tied to demographic variables (Age, Sex, Income, Education, etc.). However, the data is sent by one person in a household for the entire house. It's numerical data and I ...
Skys_'s user avatar
  • 41
4 votes
0 answers
140 views

Categorical PCA: Merge categories based on Transformation Plots?

A tutorial on categorical pca (CATPCA) (Linting et al. 2012) explains that a decision to merge categories of an ordinal variable can be made based on the category quantification ("none of the ...
user116948's user avatar
4 votes
1 answer
163 views

Reduced degrees of freedom using transformed data in one-way ANOVA (SAS PROC MIXED)

On pg. 88 of Design and Analysis of Experiments (8th Ed.) by Montgomery, he's analyzing square root transformed data in a one-way ANOVA. He provides an ANOVA table (SS, d.f., MS, F, p) for these data, ...
Meg's user avatar
  • 1,873
4 votes
0 answers
6k views

Box cox for mixed models in R

Consider a mixed model generated using the lme function in R. How can I consider the Box-cox transformations of this model in R? I have seen similar questions being asked before but they did not give ...
thankyou's user avatar
4 votes
0 answers
1k views

Variable Transformation using Cumulative Distribution Function (CDF)

Consider two different data time-series, Data1 and Data2, expressed using inhomogeneous scales (units). Each of these two data series is itself a weighted-average of a bunch of standardized ...
Mayou's user avatar
  • 957
4 votes
0 answers
126 views

Splitting a variable with nominal and numeric values

I have a variable that has both numeric and nominal components. The source has a documentation which helps in identifying which is which and for splitting into their proper components. I will do ...
Final Litiu's user avatar
4 votes
0 answers
450 views

On nonlinear regression, fits, and transformations

I am trying to fit a nonlinear regression model in R using nls(). I have a form of the equation I want to fit to: $$y = (a \times x_{1}^c +b \times x_{2}^d) (x_{3}^...
JamesCodella's user avatar
4 votes
0 answers
309 views

Methodology for validation of stochastic simulations with Kolmogorov-Smirnov test

I'm a phd student in Geography, i need some help (or good ressources) to understand why and when i need to use PIT (Probability integral transform) in my validation program for simulation. I explain ...
reyman64's user avatar
  • 175
4 votes
0 answers
928 views

Interpretation of a log likelihood function for PROC NLMIXED in SAS

I have a data set of skewed nutrient intake values, from around 7800 individuals, of whom around 3000 had two measures of daily nutrient intake (the others only had one measure), so this is a repeated ...
Michelle's user avatar
  • 3,920
4 votes
1 answer
150 views

Remove Outlying Data with a Different Trend

I currently have many sets of data that display more or less the trend in the image, which may be due to abnormalities of the data source. The series "splits" into two different trends, with one ...
Kev W.'s user avatar
  • 41
3 votes
0 answers
53 views

Is there a by-group interaction issue after the Box-Cox transformation?

I've come across a question that has me a bit stumped and hope to seek your valuable insights. Specifically, I've been working with the Box-Cox Transformation to normalize dependent variables within ...
Elizabeth's user avatar
3 votes
0 answers
55 views

Is SVM rotation invariant?

Let's say we have some data X and we want to find a linear separator using soft SVM with l2 regularization, and then we want to solve the same problem after applying some rotation matrix Q to the data ...
user3917631's user avatar
3 votes
0 answers
97 views

How to apply the Jacobian correction to AIC for a transformed dependent variable when the transformation includes an independent variable?

I am comparing several OLS multivariate regression models of a dependent variable (we'll call it $Y$) using various transformations, some of which also involve one of the independent variables ($X_1$)....
Andrew G. Benson's user avatar
3 votes
1 answer
73 views

Why is a non-linear transformation a parameter?

This in reference to my answer at What model should I use to prove statistical significance?. I test the correlation between x and ...
dariober's user avatar
  • 4,764
3 votes
0 answers
396 views

Normalizing zero inflated predictors for multiple regression

Hope I got it right, as this is my first active post :-) I was trying to find a solution the whole day for my problem. I am trying to predict a continuos variable based on 20 different predictors. The ...
Eugen Cuic's user avatar
3 votes
0 answers
20 views

Trade-off between explaining variance and correcting overdispersion

I am fitting linear model. I happen to be working in R, and the specific model I'm fitting is a generalized additive model using the package mgcv, but I think all that is incidental to my question. ...
dbspon's user avatar
  • 371
3 votes
0 answers
184 views

log transform in linear regression

Assume we have a data set and the theory suggests to model $Y \sim X$. We apply a simple linear regression and get the following: Next, let us make a log transform of both $X$ and $Y$. The result is ...
ABK's user avatar
  • 678
3 votes
0 answers
25 views

How to deal with data from different centers?

I want to examine the psychometric properties (e.g. internal consistency, factorial validity) of a scale. The data (N=700) comes from 16 different locations. Central means and variances did not differ ...
Tina Kanina 's user avatar
3 votes
0 answers
54 views

(G)LM prediction interval with heteroscedasticity

I am trying to get prediction intervals from some non-linear data which also exhibits heteroscedasticity. ...
user2974951's user avatar
  • 7,868
3 votes
0 answers
2k views

How to back transform from log10 ~ log10 regression in order to predict?

I would like to understand the results of this paper (they supply all of their R code and raw data here). The idea is to regress home range on body mass for a range of taxa. For one group of birds, ...
adkane's user avatar
  • 1,021
3 votes
1 answer
57 views

Combining data with differing dependent variables

Suppose we have two feature matrices, $X_1$ and $X_2$, with response variables $Y_1$ and $Y_2.$ Where $X_1$ and $X_2$ have the same feature columns, but distinct observations. Furthermore, $Y_1$ and $...
Peter DeWeirdt's user avatar
3 votes
0 answers
886 views

Are Box-Cox and differencing redundant or complementary?

I was always under the impression that differencing and Box-Cox were two ways to achieve the same goal: Making a time series stationary so that it can be modeled using an ARMA process. However, ...
Akaike's Children's user avatar
3 votes
1 answer
98 views

why has author divided by 1.5 in hands on machine learning with scikit learn

I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow (76/718), and the author is talking about dividing the dataset into a test set which i follow, but then goes on to talk about ...
bhoolabhoola's user avatar
3 votes
2 answers
1k views

How to weight observations to transform a distribution into normal?

Suppose X is a variable which follows some distribution (non normal) then how to define $f(X(k))$ (f is some functions of the variable X) such that $$f(X)X$$ is normally distributed and $0\leq f(X(k))...
Kotoll's user avatar
  • 103
3 votes
0 answers
925 views

High proportion of zero values and PCA

My aim is to perform PCA since I have 76 variables in my dataset. Problem is that most of my variables are highly skewed as you can see in the histogram below. These variables are proportions derived ...
Tomas's user avatar
  • 41
3 votes
0 answers
1k views

Obtain within-group Gram matrix out of distance matrix

Gram matrix Let $\bf X$ be a n x p dataset with columns (variables) centered. Then p x p $\bf X'X$ is the total scatter matrix ...
ttnphns's user avatar
  • 58.3k
3 votes
0 answers
103 views

Weibull regression sensitive to scaling of predictors?

I'm running a Weibull regression and decided to simulate data to assure myself that the model is able to recover the true parameters. ...
Ben's user avatar
  • 205
3 votes
0 answers
102 views

Projecting an image matrix onto another image matrix

This question is an attempt to re-frame another question on this forum that was of interest to a user dealing with two-dimensional image data from two scanners. If I have misunderstood the linked ...
Ben's user avatar
  • 129k
3 votes
1 answer
175 views

Robust regression after using Box cox transformation

Is that making sense to apply robust regression after using Box Cox transformation. In my data, it seems by using log transformation I can improve the model since I have the violation of the ...
Jack's user avatar
  • 61
3 votes
0 answers
709 views

How to untransform/interpret results after a Lambert W transformation?

Suppose I have some heavy-tailed data that I want to transform so it's roughly normal in order to perform a t-test. ...
Grint's user avatar
  • 415
3 votes
0 answers
361 views

Correlation of nonstationary time series (levels vs differences)

I wonder what the relationship between the empirical sample correlation of two time series in levels and the one of the differenced series is. I know that for nonstationary variables, it makes little ...
Kuma's user avatar
  • 467

1
2 3 4 5
17