Questions tagged [data-transformation]
Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.
2,424
questions
0
votes
0
answers
8
views
Any reasons I shouldn't calculate the difference between CLR transformed variables to analyze my data in a time-independent way?
First time posting, so apologies for any missing info and the like.
I have some microbiome data collected from two different treatment groups over two timepoints. I want to look at the compositional ...
- 1
0
votes
0
answers
60
views
Normalizing Power Law Data [closed]
I am working with a dataset of ordinal data, denoted as $O$, to which I am applying a power law distribution (Zeta, Pareto, Yule-Simon, etc.) resulting in a dataset $P = \zeta(O)$ that appears roughly ...
- 101
0
votes
0
answers
15
views
Total generalized variance for Box-Cox transformed components
I have a couple Gaussian mixture models where each component comes from (component-wise) Box-Cox transformed data. These models do not describe the same data: the individual components are selected ...
- 11
1
vote
0
answers
30
views
Show: scaling transformation $Y_i=D^{-1}(X_i-\bar{X})$ can be written as $Y = HX_{n\times p}D^{-1}$
Show that the scaling transformation $\textbf{y}_i=D^{-1}(\textbf{x}_i-\bar{\textbf{x}})$ can be written as $Y = HX_{n\times p}D^{-1}$ where $H$ is the centering matrix $(I_n-\frac{1}{n}\textbf{11}')$....
- 187
0
votes
0
answers
18
views
Why am I getting negative components with my custom NIPALS algorithm
I've recently been learning about the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm for computing the principal components of a dataset. I am trying to code a NIPALS class from scratch ...
- 41
0
votes
1
answer
43
views
Can you do a log transformation for excess kurtosis, or is that mainly used for skewness?
I am planning on doing a regression analysis on STATA on the financial performance of private equity funds. On my descriptive statistics, I saw higher levels of kurtosis and skewness. I decreased ...
- 1
0
votes
1
answer
60
views
WOE (Weight of evidence) cross-validation bias
I have a task to create credit scoring model using WOE encoding. I have a very small dataset, so I wont be able to perform testing on test and out-of-time samples. Thus, I am going to use cross-...
- 33
0
votes
0
answers
15
views
How can I back transform the residuals of a decomposed time series , where I used log(x+c) transformation on the original data?
I did a time series decomposition on a series of Twitter activity data into trend, seasonal and residual component. I checked the distribution of the residuals when fitting a linear model to the time ...
- 41
0
votes
0
answers
9
views
How can I best correct / handle a slight right skew distribution in my residuals plot using stats.models mixed effects model?
I am using statsmodels mixedlm as follows:
...
- 1
0
votes
1
answer
32
views
How to convert six point likert scale to four point likert scale
I would like to convert the mean score of a sample that was administered a questionnaire using a six point likert scale into a mean that is comparable to the same questionnaire using a four point ...
- 1
0
votes
1
answer
24
views
Calculating percentage change from emmeans
A related question was asked on this thread How to calculate percentage difference of geometric means with emmeans?, but I still need some help.
Instead of calculating the absolute difference between ...
- 1
2
votes
1
answer
46
views
Reducing Variance in Estimating the Exponential Average of Random Variables
Imagine we have a random variable called X, and the function form of the probability density for X is unknown. Now, I'm interested in finding the average value of the exponential of X, denoted as E[...
- 121
0
votes
0
answers
6
views
How to transform ratio input features in deep learning
I am a recommender system study which predict how likely a user browsing a product A will but another product B. One of the features is the price ratio of A and B, i.e., PriceB/PriceA. The assumption ...
- 2,065
0
votes
0
answers
15
views
Questions on calculating probabilities from user choice data for GLM training when two objects are equivalent
I’ve designed a GLM to predict user preference when presented with a choice between two buttons. However, there is some disagreement about how I’m providing collected user data to the model, and I ...
- 1
0
votes
0
answers
16
views
Mixture of Conditional Random Variables by Sampling
I am struggling to put my transformation of data into mathematical contexts. My goal is to define a mapping that transforms the original data into some awkwardly mixed data. In my simulation study, I ...
- 1
0
votes
0
answers
17
views
Should I use log transformation on Target Encoding values to avoid heteroscedasticity?
The dataset I'm working with contains categorical variables with several classes. To do its pre-processing I chose to use Target Encoder.
With numerical variables I used MinMaxScaler. When training ...
- 1
0
votes
0
answers
22
views
Anova violates normality assumption for error data
I have a 2 x 2 repeated measures ANOVA (N = 51) with Error Rate data as the dependent measure. The error data violates the normality assumption, even when outliers are removed. I have looked at the ...
- 103
0
votes
1
answer
22
views
What to do after violation of homogeneity of regression in ANCOVA
I wanted to run an ANCOVA. My independent variable is field of study (3 groups: science, humanities, and business). My dependent variable is IQ measured on a continuous scale. There are two covariates:...
0
votes
0
answers
15
views
Log transforming + a constant for negative data- ANOVA
I have data (change in mass) ranging from ~ -400 to + 4,000. I'd like to run an ANOVA comparing change in mass in different treatments. When I plot the residuals they are not normal and the residual ...
- 23
1
vote
2
answers
56
views
Transforming Data for Linear Models
I am relatively new to statistics and need some help understanding some concepts relating to linear models and their assumptions. My question refers to one assumption, but I am using this as an ...
- 11
1
vote
1
answer
64
views
Is there a way to stretch sigmoid function output
I have an array of values and a value that lies outside of array's max value:
arr = [10, 15, 20, 30]
value = 150
and I want to make that value less of an outlier, ...
- 11
0
votes
0
answers
14
views
Decomposing into Gaussian components using Bhattacharya from topFishR
I am working with fishery data. I have a data vector called SFL that contains the sizes of the fish caught. Here is some sample data:
...
- 113
1
vote
0
answers
25
views
Filtration Performance Variable
Update 7/26/23
Sorry for the lack of data in the original post. I put together a dummy data set in the image below. In this case the target for contaminant out is ~2, although there are diminishing ...
- 11
0
votes
0
answers
53
views
What model would be best to analyze bimodal distribution?
I have some data that is causing a bit of a pickle. We ran a dietary experiment on 500 invertebrates for 4 weeks; 250 invertebrates were given a Phosphorus-enriched diet and the other 250 were given a ...
- 23
0
votes
0
answers
10
views
Comparing data/means from different surveys - same question, different likert scales
I am trying to compare data from the same series of questions that most recently was used with a 5 point scale, but the previous iteration of it used a 7 point scale. Is there a way for me to convert/...
- 1
0
votes
0
answers
42
views
When do numerical transforms need sample-weighting before applying linear regression
Just wondering if anyone knows of any good resources for sample-weighting in numerical transformations. That is with the intent of using the transformed data in a linear regression model
I work with ...
- 101
4
votes
1
answer
287
views
How do I transform an extremely skewed distribution to use it for linear regression?
I'm currently working on a data set where the goal is to predict the number of rented bikes in Seoul, given information about the weather at the time.
The data set can be downloaded here: https://...
1
vote
1
answer
29
views
How can I tell if my residual standard error is too high? Also, what transformation to linearize my parameters that are contributing to high RSE?
I regressed a model in R with the following functional form/code (where sqrt, log, and squared refer to the transformations):
model<-lm(sqrt(y)~x1 + x2 + x3 + x4_squared + log_x5 + x6 + x7 + log_x8 ...
- 11
2
votes
0
answers
43
views
Comparing many event studies
I am studying the impact of an event on around 100 timeseries $i$ with weekly observations $t$.
$y_{i,t} = \beta_{1}\text{post-event} + \beta_{2} \text{week number} + \beta_{3} \text{(post-event X ...
- 121
0
votes
0
answers
91
views
Carrying out statistics on -log(p)
I have an $n \times m$ matrix, where each row $\mathbf{v}_i$, for $i \in \{1, \ldots , n\}$, consists of some permutation of the set $\{1, \ldots, m\}$ (and so in particular, each $\mathbf{v}_{i, j} \...
- 1
3
votes
1
answer
183
views
GAM negative binomial model improved by log-transforming the dependent variable
I fit a negative binomial GAM model with the R mgcv package. I noticed some heteroskedasticity in the fitted vs response plots. Then I noticed that if I log transform the dependent variable before ...
- 141
1
vote
0
answers
26
views
A good reference for power and Box-Cox transformation
I would appreciate if someone could introduce me a good reference for power transformations and Box-Cox transformation. I'm elementary in statistics and I want to learn them properly. Thank you very ...
0
votes
0
answers
18
views
Should the data transformation to the uniform [0,1] always be performed in copula modeling, even for Archimedean copula families?
Is the data transformation to the uniform [0,1] always required in copula modeling, even for Archimedean copula? I have read some sources stating that the first step in copula modeling involves ...
4
votes
1
answer
93
views
Inverting diff() and BoxCox() in R
i'm doing a project with a non-stationary time series. i used BoxCox trasformation and differences to make it stationary
...
- 41
1
vote
1
answer
67
views
How do you interpret the PCA Score from data that has previously been logarithmic transformed?
I want to make financial inclusion by using PCA. Before I did PCA, I had done a logarithmic transformation on the data because my data was positive and highly skewed (I saw this from Jolliffe's book ...
1
vote
1
answer
36
views
How can I normalize features while preserving information about the original values?
I am trying to feed a neural network information about a stock (100s of features concerning price, MA, volume etc.). To ensure training stability, I normalize the features to have ...
- 281
0
votes
0
answers
24
views
Finding the right transformation in linear regression model
I would like to consider municipal levy rates for taxes as well as percentage utilization factors, which can range from 15-40 percent, as part of a regression. The minimum of the tax rates is 200 ...
- 75
1
vote
0
answers
26
views
What is the influence of transforming explanatory variables in regression? [closed]
Let's say I have a participant who performs a test and I am measuring EMG (multiple trials/epochs). This participant has to come in for multiple sessions and every time the EMG is measured. Now I want ...
- 11
0
votes
0
answers
12
views
I used different transformations on all the variables in multiple regression analysis
So, I used different transformations on different variables, to create a normal distribution. I did the multiple linear regression with it. But now I am coming to the part in which I need to interpret ...
- 1
1
vote
1
answer
16
views
Changing scale and rounding off Target
I am training a regression machine learning model to predict an airplane's maximum take-off weight based on some pre-project features. The airplane weights in the dataset I'm working with are ...
0
votes
0
answers
49
views
Does this need to be transformed? If, yes, how?
My data is collecting deposition of particles from the atmosphere once a month for 11 months at two sites. I am testing to see if my two sites' data are normally distributed so I can determine what T-...
- 1
0
votes
0
answers
57
views
Variable (frequency) transformation in R from (0;1) range
I have variable with a range (0;1). I want to do a transformation, so that later I can use this variable as a dependent variable in my multiple linear model. For example, can I use logit ...
- 1
1
vote
0
answers
25
views
Transform Second Difference Predictions from VAR model back to levels
I am currently working with a VAR model in R using the second difference of some variables (it only becomes stationary after differencing twice). So far I'm trying to see if the model fits one of the ...
- 11
0
votes
0
answers
44
views
Nonlinear transformation in simple linear regression and almost inverse function
Let $X$ be an independent variable and $Y$ the dependent variable. Suppose we have the relationship $Y = f(X) + \epsilon$ for some unknown function $f(x)$ and some noise $\epsilon \sim N(0,1)$. If $f(...
1
vote
1
answer
56
views
Can input variable be leaking data?
I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet ...
- 2,894
0
votes
1
answer
67
views
Confidence intervals for a transformed
We are trying to calculate a CI around an inverse proportion.
The sample is women who have had one caesarean section in the past and are now having a normal (vaginal) birth of another baby
The ...
- 31
4
votes
3
answers
1k
views
A way to train a model on data with a very large number of features
I have standard data: where rows are observations, and columns are features.
...
- 249
1
vote
0
answers
37
views
How to transform percentage cover data into species abundance data in algae
I have collected percentage data for Rocky shore and summarised it per algae type like the example below:
...
- 11
0
votes
1
answer
76
views
Numerically stable transformation of log-likelihoods to probability [closed]
I have the following problem: I have log-likelihoods that need to be transformed to probabilities. One thing I have attempted is the following. Define $\kappa_{s} := \log \int p (\theta | X_s) \text{d}...
0
votes
0
answers
56
views
Differencing, taking logs or squaring does not fix nonstationarity. What to do?
I want to test for correlation in a time series model. However, all four independent variables and the dependent variable are non-stationary. I tried taking first, second and third differences but my ...
