# Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

2,424 questions
Filter by
Sorted by
Tagged with
7 views

### how to properly calculate statistics with data (time series) repeated in the same time stamp

We have made almost 400 laser experiments where the measurements are done through a photocell, registering voltage variations corresponding to variations in the intensity of the laser beam. When the ...
8 views

### Any reasons I shouldn't calculate the difference between CLR transformed variables to analyze my data in a time-independent way?

First time posting, so apologies for any missing info and the like. I have some microbiome data collected from two different treatment groups over two timepoints. I want to look at the compositional ...
• 1
60 views

### Normalizing Power Law Data [closed]

I am working with a dataset of ordinal data, denoted as $O$, to which I am applying a power law distribution (Zeta, Pareto, Yule-Simon, etc.) resulting in a dataset $P = \zeta(O)$ that appears roughly ...
• 101
15 views

### Total generalized variance for Box-Cox transformed components

I have a couple Gaussian mixture models where each component comes from (component-wise) Box-Cox transformed data. These models do not describe the same data: the individual components are selected ...
1 vote
30 views

### Show: scaling transformation $Y_i=D^{-1}(X_i-\bar{X})$ can be written as $Y = HX_{n\times p}D^{-1}$

Show that the scaling transformation $\textbf{y}_i=D^{-1}(\textbf{x}_i-\bar{\textbf{x}})$ can be written as $Y = HX_{n\times p}D^{-1}$ where $H$ is the centering matrix $(I_n-\frac{1}{n}\textbf{11}')$....
• 187
18 views

### Why am I getting negative components with my custom NIPALS algorithm

I've recently been learning about the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm for computing the principal components of a dataset. I am trying to code a NIPALS class from scratch ...
• 41
44 views

### Can you do a log transformation for excess kurtosis, or is that mainly used for skewness?

I am planning on doing a regression analysis on STATA on the financial performance of private equity funds. On my descriptive statistics, I saw higher levels of kurtosis and skewness. I decreased ...
• 1
61 views

### WOE (Weight of evidence) cross-validation bias

I have a task to create credit scoring model using WOE encoding. I have a very small dataset, so I wont be able to perform testing on test and out-of-time samples. Thus, I am going to use cross-...
• 33
15 views

### How can I back transform the residuals of a decomposed time series , where I used log(x+c) transformation on the original data?

I did a time series decomposition on a series of Twitter activity data into trend, seasonal and residual component. I checked the distribution of the residuals when fitting a linear model to the time ...
10 views

### How can I best correct / handle a slight right skew distribution in my residuals plot using stats.models mixed effects model?

I am using statsmodels mixedlm as follows: ...
• 1
32 views

### How to convert six point likert scale to four point likert scale

I would like to convert the mean score of a sample that was administered a questionnaire using a six point likert scale into a mean that is comparable to the same questionnaire using a four point ...
24 views

### Calculating percentage change from emmeans

A related question was asked on this thread How to calculate percentage difference of geometric means with emmeans?, but I still need some help. Instead of calculating the absolute difference between ...
46 views

### Reducing Variance in Estimating the Exponential Average of Random Variables

Imagine we have a random variable called X, and the function form of the probability density for X is unknown. Now, I'm interested in finding the average value of the exponential of X, denoted as E[...
• 121
6 views

### How to transform ratio input features in deep learning

I am a recommender system study which predict how likely a user browsing a product A will but another product B. One of the features is the price ratio of A and B, i.e., PriceB/PriceA. The assumption ...
• 2,065
15 views

### Questions on calculating probabilities from user choice data for GLM training when two objects are equivalent

I’ve designed a GLM to predict user preference when presented with a choice between two buttons. However, there is some disagreement about how I’m providing collected user data to the model, and I ...
16 views

### Mixture of Conditional Random Variables by Sampling

I am struggling to put my transformation of data into mathematical contexts. My goal is to define a mapping that transforms the original data into some awkwardly mixed data. In my simulation study, I ...
• 1
17 views

### Should I use log transformation on Target Encoding values ​to avoid heteroscedasticity?

The dataset I'm working with contains categorical variables with several classes. To do its pre-processing I chose to use Target Encoder. With numerical variables I used MinMaxScaler. When training ...
22 views

### Anova violates normality assumption for error data

I have a 2 x 2 repeated measures ANOVA (N = 51) with Error Rate data as the dependent measure. The error data violates the normality assumption, even when outliers are removed. I have looked at the ...
• 103
22 views

### What to do after violation of homogeneity of regression in ANCOVA

I wanted to run an ANCOVA. My independent variable is field of study (3 groups: science, humanities, and business). My dependent variable is IQ measured on a continuous scale. There are two covariates:...
15 views

### Log transforming + a constant for negative data- ANOVA

I have data (change in mass) ranging from ~ -400 to + 4,000. I'd like to run an ANOVA comparing change in mass in different treatments. When I plot the residuals they are not normal and the residual ...
• 23
1 vote
56 views

### Transforming Data for Linear Models

I am relatively new to statistics and need some help understanding some concepts relating to linear models and their assumptions. My question refers to one assumption, but I am using this as an ...
• 11
1 vote
64 views

### Is there a way to stretch sigmoid function output

I have an array of values and a value that lies outside of array's max value: arr = [10, 15, 20, 30] value = 150 and I want to make that value less of an outlier, ...
• 11
14 views

### Decomposing into Gaussian components using Bhattacharya from topFishR

I am working with fishery data. I have a data vector called SFL that contains the sizes of the fish caught. Here is some sample data: ...
• 113
1 vote
25 views

### Filtration Performance Variable

Update 7/26/23 Sorry for the lack of data in the original post. I put together a dummy data set in the image below. In this case the target for contaminant out is ~2, although there are diminishing ...
53 views

### What model would be best to analyze bimodal distribution?

I have some data that is causing a bit of a pickle. We ran a dietary experiment on 500 invertebrates for 4 weeks; 250 invertebrates were given a Phosphorus-enriched diet and the other 250 were given a ...
• 23
10 views

### Comparing data/means from different surveys - same question, different likert scales

I am trying to compare data from the same series of questions that most recently was used with a 5 point scale, but the previous iteration of it used a 7 point scale. Is there a way for me to convert/...
42 views

### When do numerical transforms need sample-weighting before applying linear regression

Just wondering if anyone knows of any good resources for sample-weighting in numerical transformations. That is with the intent of using the transformed data in a linear regression model I work with ...
• 101
288 views

### How do I transform an extremely skewed distribution to use it for linear regression?

I'm currently working on a data set where the goal is to predict the number of rented bikes in Seoul, given information about the weather at the time. The data set can be downloaded here: https://...
1 vote
29 views

### How can I tell if my residual standard error is too high? Also, what transformation to linearize my parameters that are contributing to high RSE?

I regressed a model in R with the following functional form/code (where sqrt, log, and squared refer to the transformations): model<-lm(sqrt(y)~x1 + x2 + x3 + x4_squared + log_x5 + x6 + x7 + log_x8 ...
• 11
43 views

183 views

### GAM negative binomial model improved by log-transforming the dependent variable

I fit a negative binomial GAM model with the R mgcv package. I noticed some heteroskedasticity in the fitted vs response plots. Then I noticed that if I log transform the dependent variable before ...
• 141
1 vote
26 views

### A good reference for power and Box-Cox transformation

I would appreciate if someone could introduce me a good reference for power transformations and Box-Cox transformation. I'm elementary in statistics and I want to learn them properly. Thank you very ...
18 views

### Should the data transformation to the uniform [0,1] always be performed in copula modeling, even for Archimedean copula families?

Is the data transformation to the uniform [0,1] always required in copula modeling, even for Archimedean copula? I have read some sources stating that the first step in copula modeling involves ...
93 views

### Inverting diff() and BoxCox() in R

i'm doing a project with a non-stationary time series. i used BoxCox trasformation and differences to make it stationary ...
• 41
1 vote
67 views

### How do you interpret the PCA Score from data that has previously been logarithmic transformed?

I want to make financial inclusion by using PCA. Before I did PCA, I had done a logarithmic transformation on the data because my data was positive and highly skewed (I saw this from Jolliffe's book ...
1 vote
36 views

### How can I normalize features while preserving information about the original values?

I am trying to feed a neural network information about a stock (100s of features concerning price, MA, volume etc.). To ensure training stability, I normalize the features to have ...
• 281
24 views

### Finding the right transformation in linear regression model

I would like to consider municipal levy rates for taxes as well as percentage utilization factors, which can range from 15-40 percent, as part of a regression. The minimum of the tax rates is 200 ...
• 75
1 vote
26 views

### What is the influence of transforming explanatory variables in regression? [closed]

Let's say I have a participant who performs a test and I am measuring EMG (multiple trials/epochs). This participant has to come in for multiple sessions and every time the EMG is measured. Now I want ...
12 views

### I used different transformations on all the variables in multiple regression analysis

So, I used different transformations on different variables, to create a normal distribution. I did the multiple linear regression with it. But now I am coming to the part in which I need to interpret ...
1 vote
16 views

### Changing scale and rounding off Target

I am training a regression machine learning model to predict an airplane's maximum take-off weight based on some pre-project features. The airplane weights in the dataset I'm working with are ...
49 views

### Does this need to be transformed? If, yes, how?

My data is collecting deposition of particles from the atmosphere once a month for 11 months at two sites. I am testing to see if my two sites' data are normally distributed so I can determine what T-...
57 views

### Variable (frequency) transformation in R from (0;1) range

I have variable with a range (0;1). I want to do a transformation, so that later I can use this variable as a dependent variable in my multiple linear model. For example, can I use logit ...
• 1
1 vote