Questions tagged [data-transformation]
Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.
2,438
questions
3
votes
3
answers
264
views
How to Handle 0 and 1 in Logit Transformation? [closed]
I am planning to analyze experimental data using statistical methods, and I intend to perform analysis on repeated measurements using GEE (Generalized Estimating Equations) or RM ANOVA. Some of the ...
0
votes
0
answers
15
views
Normalization by more than one variable [closed]
I want to generate a variable $real \_income$ by normalizing income by population and consumer price index. To do that, I would have to divide by the product of population and consumer price index. ...
0
votes
2
answers
59
views
simple ANN as a set of linear transformations
We cannot classify the points of the XOR problem with a single perceptron in the hidden layer. However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, ...
1
vote
0
answers
11
views
Is there a correction for samples from a (linear) Prophet model when trained on an inhomogenous Poisson point process?
Facebook's Prophet is a popular modelling choice for time series forecasting in production due to many steps being automated (and thus convenient). This can sometimes lead to over-reliance on it when ...
3
votes
1
answer
40
views
Forecasting excess mortality with ARIMA model
I am using the forecast package by Prof Hyndman, and have had success fitting ARIMA models to excess mortality (from the COVID-19 pandemic) data. I am currently trying to produce plots for cumulative ...
2
votes
0
answers
27
views
Is it possible to fit a linear model of y in log scale but with offset in the original scale?
Let's start with simple linear regression with log transformation of the response variable y:
$$ \log(y_i) = \beta_0 + \beta_1x_i + e_i$$
(btw, how is this model called? log-linear regression or ...
3
votes
1
answer
61
views
Transforming data for ANOVA or GLM
I am working with ecological count data in order to analyze differences/any contrast in species composition between warm and cold year communities. The abundances of species were recorded from ...
0
votes
1
answer
31
views
Count data and proportion covariates: best practices
I'm working with spatial data and I have the following log-linear model for count data. Let $y \sim Poisson(\lambda_{i})$ such that
$$
\log \lambda_{i} = \text{x}_i^\top\beta_{} + \epsilon_{i}
$$
such ...
2
votes
1
answer
149
views
What kind of data that I need to do my PCA?
So I have a not normal data (I did saphiro test and the result said it's not normal)
Then, I did data transform with log. So the data went normal.
Does the log data can work for my pca?
Or should I ...
3
votes
0
answers
45
views
Is there a by-group interaction issue after the Box-Cox transformation?
I've come across a question that has me a bit stumped and hope to seek your valuable insights. Specifically, I've been working with the Box-Cox Transformation to normalize dependent variables within ...
0
votes
1
answer
35
views
Detrending and data transformation to logarithm can be done together?
I want to get the effect of bitcoin price changes on foreign currency price. The third variable is inflation, which is an explanatory variable.
Should variables be detrended before regressing? Is it ...
3
votes
1
answer
78
views
Box-Cox transformation formula
I am reading some resources about the Box-Cox transformation. Almost all of the websites I found give the formula of the transformation formula as
$$y^{(\lambda )} =\begin{cases}\frac{y^\lambda-1}{\...
1
vote
2
answers
66
views
Best way to format this data for exploratory factor analysis, using R?
I originally asked this on StackOverflow, but it's more of a stats question than a coding question.
My question is about data formatting. I have this dataset (well, this is just the first two of ...
1
vote
0
answers
39
views
How do I transform this regression?
I am researching bladderwrack and whether they can adjust their amount of bladders (small inflated bags of air that develop on their skin) depending on how wave-exposed the surroundings are.
Hence I'...
2
votes
0
answers
100
views
Adapting to Changes in Thai Medical School Entrance Exam Scoring System: Seeking Statistical Adjustment Solutions
The Thai medical school entrance exam comprises three aptitude tests and other subject tests, each with their own weight. The table below shows how to calculate the total score.
Students rely on past ...
0
votes
1
answer
25
views
How to Handle Infinite Values in Feature Engineering for Machine Learning Models
I'm currently working on a machine learning project where I am creating new features related to the ratio of bytes sent and received in a communications network. However, I'm facing a challenge: when ...
2
votes
1
answer
68
views
Effect on regression coefficients by multiplying a constant to a feature [duplicate]
I was solving one quiz question on Coursera and I found an interesting question.
If you double the value of a given feature (i.e. a specific column of
the feature matrix), what happens to the least-...
1
vote
1
answer
89
views
Is Linear Regression a good algorithm or even applicable with the distribution shown in the scatter plot I have shared in this question?
I am trying to use Linear Regression on a dataset using scikit-learn with python. And my understanding is that Linear Regression requires "some linearity" to exist between independent and ...
0
votes
0
answers
31
views
Versatility and validity of aggregate algorithm for a multiple criteria decision problem (averaging)
I'm trying to aggregate different variables in multiple scales and distributions. The distributions are not known beforehand, but I want to generate a general statistical value that represents all the ...
5
votes
2
answers
399
views
Can I use different transformations on features to ensure my data follow Gaussian Distribution
Suppose I am doing linear regression on a dataset. My dataset contains columns $f_1, f_2, f_3, f_4, f_5, \text{target}$. Features (independent variables) are the column names starting with "f&...
9
votes
4
answers
889
views
An interesting observation regarding the log transformation of data
I stumbled upon something interesting while attempting to do a log transformation for some data (with zeros) today. It seems that there must be a good reason for this that I'm just not seeing. I'm ...
0
votes
0
answers
18
views
Performing PCA on count data with many true 0´s [duplicate]
I have a dataset with behavioural observations that are split into different types within each category.
For example one category would be: "Boldness". Within "Boldness" 7 ...
2
votes
0
answers
43
views
Does applying a variable transformation to experiment data result in p-hacking?
From formal statistics classwork and some research experience, I understand that it is widely considered standard practice to apply transformations to predictor variables for conducting significance ...
1
vote
0
answers
133
views
Applying PCA to Time-Series Emotional Data: Validity and Interpretation Concerns
I'm currently exploring the application of Principal Component Analysis (PCA) to time-series data representing various "facial emotional expression" states (e.g., anger, happiness, sadness, ...
1
vote
0
answers
72
views
Optimal method for estimating geometric mean ratio using Bayesian log transformed data
I'm working on a Bayesian analysis with a categorical variable involving two groups (A vs B). I'm seeking advice on the best method to compute the geometric mean ratio (GMR) together with the highest ...
1
vote
1
answer
28
views
How to model a standardized index in a regression?
I have a standardized index, $x$ (variations in s.d.) and I want to regress my dependent variable, $y$ on it.
In my dataset, the index ranges from approximately -2 to 2, but there is no constraint.
...
0
votes
0
answers
18
views
Resample random variable to fit different variance
Suppose I have samples drawn from a random variable, and I want to multiply that random variable with a scalar constant. How should I transform the samples such that they would have been drawn from ...
0
votes
0
answers
54
views
Log transformation leading to extremely negative R-squared and extremle values for MSE
I have a set of data and I am using this code on it:
...
2
votes
1
answer
16
views
How to interpret this model diagnostics?
A model was fit as below:
m1 <- lmer(log (ld50) ~ var * strain * time + (1|rep) + (1|rep:var) + (1|strain:env), dt)
The response ld50 ranges from 0.15 (lower ...
1
vote
0
answers
22
views
How to Handle Non-Multinormality in the Context of Exploratory Factor Analysis for Logistic Regression
I'm trying to follow the book A Step-by-Step Guide to Exploratory Factor Analysis with R and Rstudio, by Marley W. Watkins, and apply the principles in the book to a real-world data set. Ultimately, ...
2
votes
1
answer
85
views
My dataset includes multiple variable and all of these variables have sub-variables. How to visualise & test which segment is significant?
So, I have survey responses from users. Just to make it clear, if you select an issue like Poor UI then you are prompted with 4-5 specific issues about the UI to select from. Poor UI is the main ...
0
votes
0
answers
47
views
Should data be centered at $1$ before applying Box-Cox transformation?
Let's suppose that we perform Box-Cox transformation in R for the following data
...
1
vote
1
answer
43
views
Transforming data with a fitted distribution function
I have a bivariate dataset on $[0,1]^2$ in which I am interested in fitting a joint distribution. I fit a Gaussian copula but am unsure how to judge if it's a good fit. I tried transforming my data ...
0
votes
0
answers
20
views
R: boundary (singular) fit: see help('isSingular') with lrem model - only when transforming data to log
I am trying to run a lmer model on my dataset.
My dataset :
str(tabfi)
...
5
votes
1
answer
111
views
Should I orthogonalize variables before regression?
If I have several correlated variables in my dataset which I would like to include as predictors in my model. For example with this simulated dataset:
...
7
votes
2
answers
763
views
How to report my log transformed (+1) data?
Say that I have a variable with lots of 0 values that needs log-transforming so I do log(variable+1) to transform it. How do I write that in my methods section as opposed to just 'the data was log-...
0
votes
0
answers
17
views
Lowered dependent variable in one period
I have a problem with a dependent variable in my model.
The dependent variable is the saving rate which is savings/disposable income. However, disposable income is income - taxes. So the problem is ...
1
vote
1
answer
101
views
Transformations to meet heteroscedasticity
I have a dataset containing angles. They represent the bending angle that a seedling makes to go toward light. I have two factors: treatment and genotype, so I use a two way ANOVA. However, the ...
1
vote
1
answer
59
views
Interpretation of betareg coefficients where observations transformed to account for y=0 or y=1
I am running a beta-regression using betareg in R (with default logit link function). My response variable is a proportion, and may include 0 and/or 1. I've transformed the data following the betareg ...
0
votes
0
answers
17
views
log-log regression as reward function in optimization problem
Consider the model $\hat{y}_t = e^{\text{trend} + \text{seasonality}} \prod_k^K x_{k, t}^{b_k}$
where $K$ denotes different investment alternatives. You can think that trend and seasonality are ...
5
votes
2
answers
753
views
Always higher R squared after log transformation
lately I lost access to SPSS and instead of using Python or R, I tend to perform analysis using a free software called Jamovi.
The thing is, this software doesn't have the different non-linear ...
0
votes
0
answers
90
views
copula and categorical data transformation
I plan to model copulas with data that has categorical variables. Copula modeling with categorical variables requires a transformation of categorical variables into continuous (link: https://hal....
1
vote
0
answers
32
views
When is sufficiency and completeness of a statistic preserved?
This question has been asked in math stack but no one has replied.
I have been given these definitions in my statistical inference class:
Let $(X_1,...,X_n)$ be a simple random sampling of $X\...
0
votes
0
answers
27
views
Optimising a multivariate table of counts based on marginals
I've been stuck on a problem for a very long time now so I decided to post on this forum for the first time. Although I am using code to perform this task, I believe it uses some statistics and I ...
1
vote
1
answer
50
views
How to transform STDEV of a numerical value to logarithmic value? [closed]
I have the following question:
I have the average and STDEV values of variable. To transform this to Log10 scale, I directly apply the logarithm to the average. Can I do the same with the STDEV?
...
2
votes
2
answers
84
views
Is it valid to take the midpoint of discrete outcome variables that express a range e.g. (1-20%, 21-40%) then calculate the mean of those midpoints?
I work in a health service and have a PhD student who was told by my boss (her primary supervisor) to measure clinicians' responses concerning what proportion of their clients had a certain condition ...
0
votes
0
answers
28
views
Distribution of positive semidefinite matrices that are generated by uniformly distributed positive definite matrices
Let $\mathcal{A}=\{ A_1,A_2,\dots,A_n \} \subseteq \mathcal{S}^p_{++}$ be a set of real positive-definite matrices sampled uniformly with a fixed trace (say, using this algorithm).
To convert each $...
0
votes
0
answers
20
views
how to properly calculate statistics with data (time series) repeated in the same time stamp
We have made almost 400 laser experiments where the measurements are done through a photocell, registering voltage variations corresponding to variations in the intensity of the laser beam. When the ...
0
votes
0
answers
22
views
Any reasons I shouldn't calculate the difference between CLR transformed variables to analyze my data in a time-independent way?
First time posting, so apologies for any missing info and the like.
I have some microbiome data collected from two different treatment groups over two timepoints. I want to look at the compositional ...
0
votes
0
answers
26
views
Total generalized variance for Box-Cox transformed components
I have a couple Gaussian mixture models where each component comes from (component-wise) Box-Cox transformed data. These models do not describe the same data: the individual components are selected ...