Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
3 votes
3 answers
264 views

How to Handle 0 and 1 in Logit Transformation? [closed]

I am planning to analyze experimental data using statistical methods, and I intend to perform analysis on repeated measurements using GEE (Generalized Estimating Equations) or RM ANOVA. Some of the ...
soobinism's user avatar
0 votes
0 answers
15 views

Normalization by more than one variable [closed]

I want to generate a variable $real \_income$ by normalizing income by population and consumer price index. To do that, I would have to divide by the product of population and consumer price index. ...
Katharina K's user avatar
0 votes
2 answers
59 views

simple ANN as a set of linear transformations

We cannot classify the points of the XOR problem with a single perceptron in the hidden layer. However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, ...
Mag's user avatar
  • 1
1 vote
0 answers
11 views

Is there a correction for samples from a (linear) Prophet model when trained on an inhomogenous Poisson point process?

Facebook's Prophet is a popular modelling choice for time series forecasting in production due to many steps being automated (and thus convenient). This can sometimes lead to over-reliance on it when ...
Galen's user avatar
  • 8,370
3 votes
1 answer
40 views

Forecasting excess mortality with ARIMA model

I am using the forecast package by Prof Hyndman, and have had success fitting ARIMA models to excess mortality (from the COVID-19 pandemic) data. I am currently trying to produce plots for cumulative ...
Jina A.'s user avatar
  • 31
2 votes
0 answers
27 views

Is it possible to fit a linear model of y in log scale but with offset in the original scale?

Let's start with simple linear regression with log transformation of the response variable y: $$ \log(y_i) = \beta_0 + \beta_1x_i + e_i$$ (btw, how is this model called? log-linear regression or ...
Sofie's user avatar
  • 21
3 votes
1 answer
61 views

Transforming data for ANOVA or GLM

I am working with ecological count data in order to analyze differences/any contrast in species composition between warm and cold year communities. The abundances of species were recorded from ...
user390865's user avatar
0 votes
1 answer
31 views

Count data and proportion covariates: best practices

I'm working with spatial data and I have the following log-linear model for count data. Let $y \sim Poisson(\lambda_{i})$ such that $$ \log \lambda_{i} = \text{x}_i^\top\beta_{} + \epsilon_{i} $$ such ...
BelwarDissengulp's user avatar
2 votes
1 answer
149 views

What kind of data that I need to do my PCA?

So I have a not normal data (I did saphiro test and the result said it's not normal) Then, I did data transform with log. So the data went normal. Does the log data can work for my pca? Or should I ...
Nimas Pertiwi's user avatar
3 votes
0 answers
45 views

Is there a by-group interaction issue after the Box-Cox transformation?

I've come across a question that has me a bit stumped and hope to seek your valuable insights. Specifically, I've been working with the Box-Cox Transformation to normalize dependent variables within ...
Elizabeth's user avatar
0 votes
1 answer
35 views

Detrending and data transformation to logarithm can be done together?

I want to get the effect of bitcoin price changes on foreign currency price. The third variable is inflation, which is an explanatory variable. Should variables be detrended before regressing? Is it ...
user405402's user avatar
3 votes
1 answer
78 views

Box-Cox transformation formula

I am reading some resources about the Box-Cox transformation. Almost all of the websites I found give the formula of the transformation formula as $$y^{(\lambda )} =\begin{cases}\frac{y^\lambda-1}{\...
Matata's user avatar
  • 671
1 vote
2 answers
66 views

Best way to format this data for exploratory factor analysis, using R?

I originally asked this on StackOverflow, but it's more of a stats question than a coding question. My question is about data formatting. I have this dataset (well, this is just the first two of ...
JeniFav's user avatar
  • 21
1 vote
0 answers
39 views

How do I transform this regression?

I am researching bladderwrack and whether they can adjust their amount of bladders (small inflated bags of air that develop on their skin) depending on how wave-exposed the surroundings are. Hence I'...
Telefondemonen_se's user avatar
2 votes
0 answers
100 views

Adapting to Changes in Thai Medical School Entrance Exam Scoring System: Seeking Statistical Adjustment Solutions

The Thai medical school entrance exam comprises three aptitude tests and other subject tests, each with their own weight. The table below shows how to calculate the total score. Students rely on past ...
nitsir's user avatar
  • 21
0 votes
1 answer
25 views

How to Handle Infinite Values in Feature Engineering for Machine Learning Models

I'm currently working on a machine learning project where I am creating new features related to the ratio of bytes sent and received in a communications network. However, I'm facing a challenge: when ...
Camilo Piñón's user avatar
2 votes
1 answer
68 views

Effect on regression coefficients by multiplying a constant to a feature [duplicate]

I was solving one quiz question on Coursera and I found an interesting question. If you double the value of a given feature (i.e. a specific column of the feature matrix), what happens to the least-...
teddcp's user avatar
  • 155
1 vote
1 answer
89 views

Is Linear Regression a good algorithm or even applicable with the distribution shown in the scatter plot I have shared in this question?

I am trying to use Linear Regression on a dataset using scikit-learn with python. And my understanding is that Linear Regression requires "some linearity" to exist between independent and ...
letdatado's user avatar
  • 325
0 votes
0 answers
31 views

Versatility and validity of aggregate algorithm for a multiple criteria decision problem (averaging)

I'm trying to aggregate different variables in multiple scales and distributions. The distributions are not known beforehand, but I want to generate a general statistical value that represents all the ...
gabriel's user avatar
  • 93
5 votes
2 answers
399 views

Can I use different transformations on features to ensure my data follow Gaussian Distribution

Suppose I am doing linear regression on a dataset. My dataset contains columns $f_1, f_2, f_3, f_4, f_5, \text{target}$. Features (independent variables) are the column names starting with "f&...
letdatado's user avatar
  • 325
9 votes
4 answers
889 views

An interesting observation regarding the log transformation of data

I stumbled upon something interesting while attempting to do a log transformation for some data (with zeros) today. It seems that there must be a good reason for this that I'm just not seeing. I'm ...
knrumsey's user avatar
  • 7,527
0 votes
0 answers
18 views

Performing PCA on count data with many true 0´s [duplicate]

I have a dataset with behavioural observations that are split into different types within each category. For example one category would be: "Boldness". Within "Boldness" 7 ...
JanB's user avatar
  • 1
2 votes
0 answers
43 views

Does applying a variable transformation to experiment data result in p-hacking?

From formal statistics classwork and some research experience, I understand that it is widely considered standard practice to apply transformations to predictor variables for conducting significance ...
rdos84's user avatar
  • 21
1 vote
0 answers
133 views

Applying PCA to Time-Series Emotional Data: Validity and Interpretation Concerns

I'm currently exploring the application of Principal Component Analysis (PCA) to time-series data representing various "facial emotional expression" states (e.g., anger, happiness, sadness, ...
AltunE's user avatar
  • 111
1 vote
0 answers
72 views

Optimal method for estimating geometric mean ratio using Bayesian log transformed data

I'm working on a Bayesian analysis with a categorical variable involving two groups (A vs B). I'm seeking advice on the best method to compute the geometric mean ratio (GMR) together with the highest ...
mat's user avatar
  • 541
1 vote
1 answer
28 views

How to model a standardized index in a regression?

I have a standardized index, $x$ (variations in s.d.) and I want to regress my dependent variable, $y$ on it. In my dataset, the index ranges from approximately -2 to 2, but there is no constraint. ...
Oalvinegro's user avatar
0 votes
0 answers
18 views

Resample random variable to fit different variance

Suppose I have samples drawn from a random variable, and I want to multiply that random variable with a scalar constant. How should I transform the samples such that they would have been drawn from ...
mroelofs's user avatar
0 votes
0 answers
54 views

Log transformation leading to extremely negative R-squared and extremle values for MSE

I have a set of data and I am using this code on it: ...
Coding_noob's user avatar
2 votes
1 answer
16 views

How to interpret this model diagnostics?

A model was fit as below: m1 <- lmer(log (ld50) ~ var * strain * time + (1|rep) + (1|rep:var) + (1|strain:env), dt) The response ld50 ranges from 0.15 (lower ...
Rabin KC's user avatar
1 vote
0 answers
22 views

How to Handle Non-Multinormality in the Context of Exploratory Factor Analysis for Logistic Regression

I'm trying to follow the book A Step-by-Step Guide to Exploratory Factor Analysis with R and Rstudio, by Marley W. Watkins, and apply the principles in the book to a real-world data set. Ultimately, ...
Adrian Keister's user avatar
2 votes
1 answer
85 views

My dataset includes multiple variable and all of these variables have sub-variables. How to visualise & test which segment is significant?

So, I have survey responses from users. Just to make it clear, if you select an issue like Poor UI then you are prompted with 4-5 specific issues about the UI to select from. Poor UI is the main ...
doodle2611's user avatar
0 votes
0 answers
47 views

Should data be centered at $1$ before applying Box-Cox transformation?

Let's suppose that we perform Box-Cox transformation in R for the following data ...
Adam Bogdański's user avatar
1 vote
1 answer
43 views

Transforming data with a fitted distribution function

I have a bivariate dataset on $[0,1]^2$ in which I am interested in fitting a joint distribution. I fit a Gaussian copula but am unsure how to judge if it's a good fit. I tried transforming my data ...
Bpe's user avatar
  • 13
0 votes
0 answers
20 views

R: boundary (singular) fit: see help('isSingular') with lrem model - only when transforming data to log

I am trying to run a lmer model on my dataset. My dataset : str(tabfi) ...
stillamistery's user avatar
5 votes
1 answer
111 views

Should I orthogonalize variables before regression?

If I have several correlated variables in my dataset which I would like to include as predictors in my model. For example with this simulated dataset: ...
locus's user avatar
  • 1,563
7 votes
2 answers
763 views

How to report my log transformed (+1) data?

Say that I have a variable with lots of 0 values that needs log-transforming so I do log(variable+1) to transform it. How do I write that in my methods section as opposed to just 'the data was log-...
sausageroll888's user avatar
0 votes
0 answers
17 views

Lowered dependent variable in one period

I have a problem with a dependent variable in my model. The dependent variable is the saving rate which is savings/disposable income. However, disposable income is income - taxes. So the problem is ...
RainBe's user avatar
  • 1
1 vote
1 answer
101 views

Transformations to meet heteroscedasticity

I have a dataset containing angles. They represent the bending angle that a seedling makes to go toward light. I have two factors: treatment and genotype, so I use a two way ANOVA. However, the ...
Marius Audenis's user avatar
1 vote
1 answer
59 views

Interpretation of betareg coefficients where observations transformed to account for y=0 or y=1

I am running a beta-regression using betareg in R (with default logit link function). My response variable is a proportion, and may include 0 and/or 1. I've transformed the data following the betareg ...
branwen85's user avatar
  • 111
0 votes
0 answers
17 views

log-log regression as reward function in optimization problem

Consider the model $\hat{y}_t = e^{\text{trend} + \text{seasonality}} \prod_k^K x_{k, t}^{b_k}$ where $K$ denotes different investment alternatives. You can think that trend and seasonality are ...
pete lewis's user avatar
5 votes
2 answers
753 views

Always higher R squared after log transformation

lately I lost access to SPSS and instead of using Python or R, I tend to perform analysis using a free software called Jamovi. The thing is, this software doesn't have the different non-linear ...
gabriel's user avatar
  • 93
0 votes
0 answers
90 views

copula and categorical data transformation

I plan to model copulas with data that has categorical variables. Copula modeling with categorical variables requires a transformation of categorical variables into continuous (link: https://hal....
mutu's user avatar
  • 1
1 vote
0 answers
32 views

When is sufficiency and completeness of a statistic preserved?

This question has been asked in math stack but no one has replied. I have been given these definitions in my statistical inference class: Let $(X_1,...,X_n)$ be a simple random sampling of $X\...
José's user avatar
  • 111
0 votes
0 answers
27 views

Optimising a multivariate table of counts based on marginals

I've been stuck on a problem for a very long time now so I decided to post on this forum for the first time. Although I am using code to perform this task, I believe it uses some statistics and I ...
taek's user avatar
  • 3
1 vote
1 answer
50 views

How to transform STDEV of a numerical value to logarithmic value? [closed]

I have the following question: I have the average and STDEV values of variable. To transform this to Log10 scale, I directly apply the logarithm to the average. Can I do the same with the STDEV? ...
Zahari's user avatar
  • 11
2 votes
2 answers
84 views

Is it valid to take the midpoint of discrete outcome variables that express a range e.g. (1-20%, 21-40%) then calculate the mean of those midpoints?

I work in a health service and have a PhD student who was told by my boss (her primary supervisor) to measure clinicians' responses concerning what proportion of their clients had a certain condition ...
llewmills's user avatar
  • 2,111
0 votes
0 answers
28 views

Distribution of positive semidefinite matrices that are generated by uniformly distributed positive definite matrices

Let $\mathcal{A}=\{ A_1,A_2,\dots,A_n \} \subseteq \mathcal{S}^p_{++}$ be a set of real positive-definite matrices sampled uniformly with a fixed trace (say, using this algorithm). To convert each $...
12345's user avatar
  • 213
0 votes
0 answers
20 views

how to properly calculate statistics with data (time series) repeated in the same time stamp

We have made almost 400 laser experiments where the measurements are done through a photocell, registering voltage variations corresponding to variations in the intensity of the laser beam. When the ...
Arturo Ortiz Tapia's user avatar
0 votes
0 answers
22 views

Any reasons I shouldn't calculate the difference between CLR transformed variables to analyze my data in a time-independent way?

First time posting, so apologies for any missing info and the like. I have some microbiome data collected from two different treatment groups over two timepoints. I want to look at the compositional ...
Kam's user avatar
  • 1
0 votes
0 answers
26 views

Total generalized variance for Box-Cox transformed components

I have a couple Gaussian mixture models where each component comes from (component-wise) Box-Cox transformed data. These models do not describe the same data: the individual components are selected ...
ladislaw94's user avatar

1
2 3 4 5
49