Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

700 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
786 views

Feature scaling for non-negative sparse data

Imagine you have many observations on which you want to run a classification algorithm. Each observation is characterized by a matrix of non-negative values. For all observations 90-98% of the values ...
6
votes
0answers
362 views

T tests on proportions - Wrong, but how wrong?

Background: In psychology, and probably a number of other disciplines, it's common practice to test between-groups effects on a binary variable, such as accuracy, by aggregating data within ...
6
votes
0answers
393 views

What, if any, dissimilarity is preserved in partial least squares (PLS)?

When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
5
votes
0answers
374 views

Transformation bias with non-linear functions

This a more general question: I often deal with experimental data (subject to uncertainties in the measurements) that have to be transformed using a function, to calculate a parameter (which can, for ...
5
votes
0answers
112 views

Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
5
votes
0answers
330 views

Algorithms for data symmetrization

There are statistical methods (e.g. by Box-Cox or Yeo-Johnson, see references below) to automatically bring data vectors as close as possible to symmetry/normality using optimal power transformations. ...
4
votes
0answers
512 views

Box constraints with BFGS algorithm

I've been a long time adept of the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS), which I trusted to be a pretty efficient local optimisation technique. And indeed it is. The problem I usually ...
4
votes
0answers
51 views

Is there a systematic procedure to do data cleaning or preprocessing?

I do data cleaning / data preprocessing everyday, using various tools to remove outliers and to keep normality However, I feel what I am doing is more a handcraft rather than systematic: I don't ...
4
votes
0answers
46 views

Data Transformation Question - Multiplying data proportional to demographics

I have a bunch of data that is tied to demographic variables (Age, Sex, Income, Education, etc.). However, the data is sent by one person in a household for the entire house. It's numerical data and I ...
4
votes
0answers
109 views

Categorical PCA: Merge categories based on Transformation Plots?

A tutorial on categorical pca (CATPCA) (Linting et al. 2012) explains that a decision to merge categories of an ordinal variable can be made based on the category quantification ("none of the ...
4
votes
1answer
82 views

Reduced degrees of freedom using transformed data in one-way ANOVA (SAS PROC MIXED)

On pg. 88 of Design and Analysis of Experiments (8th Ed.) by Montgomery, he's analyzing square root transformed data in a one-way ANOVA. He provides an ANOVA table (SS, d.f., MS, F, p) for these data, ...
4
votes
0answers
630 views

Bayesian Analysis of Box-Cox Transformation

This problem is problem 5 in Chapter 7 of Bayesian Data Analysis, 3rd edition. Consider the Box-Cox transformation: $y_i^{(\lambda)} \sim \mathcal{N}(\mu, \sigma^2)$ where $y_i^{(\lambda)} = (y_i^{\...
4
votes
0answers
4k views

Box cox for mixed models in R

Consider a mixed model generated using the lme function in R. How can I consider the Box-cox transformations of this model in R? I have seen similar questions being asked before but they did not give ...
4
votes
2answers
804 views

What methods can be used to transform data?

I am solving a binary classification problem with 4 predictor variables. The variables didn't seem to be linearly separable. I have used Neural Networks and Kernel SVM which work and give desired ...
4
votes
0answers
125 views

Splitting a variable with nominal and numeric values

I have a variable that has both numeric and nominal components. The source has a documentation which helps in identifying which is which and for splitting into their proper components. I will do ...
4
votes
0answers
991 views

Why would SVD be 'unstable' if you don't standardize your data first?

I'm reading an article about Direct Linear Transformation which processes data using SVD, and the data set is standardized so that it has zero mean and unit standard deviation (n.b., some people call ...
4
votes
0answers
375 views

On nonlinear regression, fits, and transformations

I am trying to fit a nonlinear regression model in R using nls(). I have a form of the equation I want to fit to: $$y = (a \times x_{1}^c +b \times x_{2}^d) (x_{3}^...
4
votes
0answers
633 views

Factor analysis across different levels of data aggregation

I have survey data for thousands of individuals from hundreds of towns. I want to identify factors underlying certain characteristics at the town level and the individual level. The individual level ...
4
votes
0answers
288 views

Methodology for validation of stochastic simulations with Kolmogorov-Smirnov test

I'm a phd student in Geography, i need some help (or good ressources) to understand why and when i need to use PIT (Probability integral transform) in my validation program for simulation. I explain ...
4
votes
0answers
806 views

Interpretation of a log likelihood function for PROC NLMIXED in SAS

I have a data set of skewed nutrient intake values, from around 7800 individuals, of whom around 3000 had two measures of daily nutrient intake (the others only had one measure), so this is a repeated ...
4
votes
1answer
105 views

Remove Outlying Data with a Different Trend

I currently have many sets of data that display more or less the trend in the image, which may be due to abnormalities of the data source. The series "splits" into two different trends, with one ...
4
votes
1answer
4k views

Linear mixed effects models: what to do when the residual QQ-plot looks non-normal?

I have four linear mixed effect models of similar structure: ...
3
votes
0answers
31 views

Normalizing zero inflated predictors for multiple regression

Hope I got it right, as this is my first active post :-) I was trying to find a solution the whole day for my problem. I am trying to predict a continuos variable based on 20 different predictors. The ...
3
votes
0answers
46 views

log transform in linear regression

Assume we have a data set and the theory suggests to model $Y \sim X$. We apply a simple linear regression and get the following: Next, let us make a log transform of both $X$ and $Y$. The result is ...
3
votes
0answers
21 views

How to deal with data from different centers?

I want to examine the psychometric properties (e.g. internal consistency, factorial validity) of a scale. The data (N=700) comes from 16 different locations. Central means and variances did not differ ...
3
votes
0answers
35 views

(G)LM prediction interval with heteroscedasticity

I am trying to get prediction intervals from some non-linear data which also exhibits heteroscedasticity. ...
3
votes
1answer
102 views

GLM on non-integer data

I'm looking for a recommendation on what GLM I could do with non-integer data. Brief background of what I am doing: I'm wanting to combine calculated herbivory rates with abundance data, to compare ...
3
votes
0answers
487 views

Quantile transform vs Power transformation to get normal distribution

I was introduced to the concept of quantile-based gaussian transform. To my understanding, it changes the value of the original data by each percentile to the matching percentile of gaussian ...
3
votes
0answers
69 views

Is there a name for a $y=\sqrt[k]{x}$-like data normalization?

I'm normalizing multivariate numeric data that has both negative and positive values. For the sake of the question let's assume a range of e.g. $[-10000,10000]$ with a lot of values in $[-1,1]$. I've ...
3
votes
1answer
83 views

why has author divided by 1.5 in hands on machine learning with scikit learn

I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow (76/718), and the author is talking about dividing the dataset into a test set which i follow, but then goes on to talk about ...
3
votes
0answers
605 views

How to compare transformed and untransformed linear models?

I have a linear model which doesn't have any particular issues with its assumptions (diagnostics plots look well). However it has a slighly skewed response (skewness approx. 0.5) and few skew ...
3
votes
0answers
61 views

Weibull regression sensitive to scaling of predictors?

I'm running a Weibull regression and decided to simulate data to assure myself that the model is able to recover the true parameters. ...
3
votes
0answers
42 views

Projecting an image matrix onto another image matrix

This question is an attempt to re-frame another question on this forum that was of interest to a user dealing with two-dimensional image data from two scanners. If I have misunderstood the linked ...
3
votes
0answers
112 views

Robust regression after using Box cox transformation

Is that making sense to apply robust regression after using Box Cox transformation. In my data, it seems by using log transformation I can improve the model since I have the violation of the ...
3
votes
0answers
188 views

Correlation of nonstationary time series (levels vs differences)

I wonder what the relationship between the empirical sample correlation of two time series in levels and the one of the differenced series is. I know that for nonstationary variables, it makes little ...
3
votes
1answer
132 views

Common variable transformations

I want to predict a variable $Y$ given a set of variables $X_i$. To account for nonlinearity, my $X_i$ are put in several quantile dummies, so that I prefer transforming my $y$. My $Y$ variable are ...
3
votes
0answers
156 views

Is the conditional densities of X given Y and X given a bijective function of Y equivalent?

Suppose that I know the conditional density $$ f_{X\lvert Z}(x\lvert Z=z) $$ It is the same as knowing $$ f_{X\lvert Y}(x\lvert Y = y) $$ if I assume that $Z=g(Y)$? Let say I have two continuous ...
3
votes
0answers
881 views

Box-Cox transformation vs. predictor transformation in multiple linear regression

I first tried to transform my predictors one by one, but I couldn't find a satisfactory transformation. Then I tried Box-Cox which solved a lot of linearity and constant variance issues, but I still ...
3
votes
0answers
277 views

Why am I required to conduct a reverse transformation on my predicted variables?

I'm having difficulty wrapping my head around an instruction in my assignment. I would appreciate any pointers. After transforming the mean predicted values of my simple linear regression model, I ...
3
votes
0answers
103 views

Multiple Regression Assumptions

This may seem like a basic question, but I'm verifying the assumptions for a multiple regression and have some trouble wrapping my head around homoscedasticity. I have a few questions listed below: 1)...
3
votes
0answers
113 views

Mean and Standard Error of True Percentages (Not Binomial Proportions)

In my line of work I occasionally deal with data on the percent lipid of fish fillet (environmental sampling). The question came up today about how to calculate the mean and standard error and ...
3
votes
0answers
55 views

How to interpret these distributions?

I'm analysing some stack overflow data and trying to characterise four variables with appropriate distributions. The histograms below show each variable untransformed. The common theme is that the ...
3
votes
0answers
129 views

Comparison between regression of $a = bc^t$ and $\log a = \log b +t \log c$

This question is more qualitative then about the maths behind the equation. Variables: a = month (1, 2, 3, ) t = shipments of a product in that month You wish to derive the relationship between $a$ ...
3
votes
0answers
64 views

Regression: Should I use the prediction interval obtained given n=9 and an outlier (Cook's D= 0.558) present?

The data I'm working with has 9 observations. I'm using only one predictor variable. Using SAS, I fit the model and checked the residuals. The typical model assumptions appear to be met, but there ...
3
votes
0answers
251 views

Relation between variance stabilizing transformations and effect sizes?

When researching effect size for proportions, in particular the paper Effect-Size Indices for Dichotomized Outcomes in Meta-Analysis, that at least two of the usual effect sizes are realy variance ...
3
votes
0answers
464 views

How to estimate parameters of a nonlinear function with log-normal error?

Consider you have some nonlinear function \begin{align} y_i&=\epsilon_i f(\beta,x_i) \end{align} where $\epsilon_i$ is log-normally distributed with mean 1, and \begin{equation} f(x,\beta)=\...
3
votes
0answers
208 views

effects of Box-Cox transformation on covariance

I'm trying to synthesize data for a Monte Carlo simulation. I have a stationary random process $x$ and can readily estimate its covariance matrix $S$. I know that if the increments of the process are ...
3
votes
0answers
130 views

clustering for histogram shapes

I am trying to get a start on a clustering problem. The sample data is trade volume at a particular price. Some notes about the data: number of bins vary from sample to sample (larger price range ...
3
votes
0answers
360 views

How deal with exponential distribution of data during principal component analysis

I am trying to do PCA on a series of variables (all are positive, real numbers) using correlation and varimax rotation. All the computation is done in R. Although I got high loadings for all ...
3
votes
0answers
2k views

OLS with a heavily skewed independent variable

I am regressing a log-normally distributed dependent variable (wage) on a heavily skewed independent variable and I want to make sure I handle it in the best ...

1
2 3 4 5
14