Skip to main content

Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Level mean from natural logarithm mean [duplicate]

Suppose I have a random sample $X_i$ where the values $x_i$ are strictly positive. I calculate the mean of the transformed variable $t_i = \ln x_i$ as $\bar{t} = \frac{\sum_{i=1}^n t_i}{n} \approx 9....
Papayapap's user avatar
  • 363
1 vote
0 answers
18 views

Rescale measures of association for meta-analysis (e.g., log-transformed independent variables)

I am carrying out a meta-analysis of studies evaluating the association between blood levels of specific environmental pollutants and health outcomes (binary). Some studies reported OR/RR/HR for ...
msas's user avatar
  • 11
0 votes
2 answers
25 views

Different Transformation of the same IV [closed]

Suppose I've a panel data with 2 segments. And I want to run pooled OLS regression. I'm doing data transformation for each segment separately. Say for segment A, I'm doing log transformation and for ...
Beta's user avatar
  • 6,386
0 votes
0 answers
33 views

log transform left data in r

I am having trouble finding the transformation operation for left/negatively skewed data. The catch? All of my values are between 0 and 1. As such, trying the standard log10 transformation command ...
YouLocalRUser's user avatar
0 votes
0 answers
25 views

Confusion over BoxCox in R

I'm new to time series forecasting and not coming from a maths degree so please bear with me. I'm working from an example of how to implement ts forecasting in R and attempting to a) see if I need to ...
Magnetar's user avatar
0 votes
0 answers
23 views

How can i create a linear regression model not having the exact same dataset

i want to create a data regression between two financial indexes, but they don't have a perfect correspondence in the data of observation (for example one has the relevations for 17/6 18/6 19/6 but ...
ConfusedConsultant's user avatar
1 vote
1 answer
42 views

Poisson regression given multiple predictors on a repeating ID variable

I was wondering how a poisson regression would work given my dataset which describes a series of zip codes stratified by age groups, gender and death counts. The regression would use death counts as ...
Seyong Chang's user avatar
1 vote
0 answers
27 views

Closed-Form Lambda for Yeo-Johnson-Transformed Normal-Inverse-Gaussian-Distributed Random Variables

I would like to know whether there exists a closed-form solution for the $\lambda$-parameter that maximizes the log-likelihood function of Yeo-Johnson transformed random variables that (before the ...
Hiro's user avatar
  • 425
0 votes
0 answers
19 views

Finding the best way for combining features non-linearly within a linear regression

Problem Statement I have a set of two features, $X_1$ and $X_2$ that I combine to try and predict a target variable in a regression of the form: $ Y_0 = \frac{X_1 - X_2}{X_1 + X_2} $. You can think of ...
Joe's user avatar
  • 271
-1 votes
1 answer
55 views

Transformation of negatively skewed data [closed]

I asked a question and thanks to your feedback I noticed how unspecific and not thought through enough it was. So, I am trying to be more specific. I am currently trying to calculate a moderation ...
Laura's user avatar
  • 49
3 votes
1 answer
186 views

Back Transforming log-log Model for Prediction

I have a model that is log-log and I would like to make raw predictions of $Y$ with it: $\ln(Y) = B_0 + B_1\ln(X)$ All answers and articles I have found concerning back transforming for prediction ...
Oberon Quinn's user avatar
8 votes
4 answers
455 views

Estimate Box-Cox Transformation Lambda Using Skewness and Kurtosis

I would be interested in a method to find an appropriate Lambda parameter for the Box-Cox transformation based on only the skewness and the kurtosis of a given sample. I.e, if the skewness and ...
Hiro's user avatar
  • 425
0 votes
0 answers
32 views

What happens to the data when entity- and time-fixed effects are applied?

Let us assume that the true data generating process is as follows: Y_it = beta_0 + beta_1 X_it + u_it What happens exactly when I apply both entity- and time-fixed ...
TFT's user avatar
  • 345
2 votes
0 answers
43 views

Standardization of summary statistic of group-linked values

Assume that you measure a summary statistic (e.g., arithmetic mean) in measurement windows of a fixed size along a long sequence of values and that these values are grouped into regions belonging to ...
Michael Gruenstaeudl's user avatar
5 votes
2 answers
377 views

Correction for heavy-tailed distribution of residuals?

I'm interested in studying the effect of $x$ on $y$ using a fixed effects method. The residuals follow a heavy tail distribution, as the normal Q-Q plot suggests. For inference, I need a normal ...
TFT's user avatar
  • 345
4 votes
1 answer
174 views

Making linear to logistic regression with sigmoid function - why is a transformation of predicted y needed?

I noticed that one can run a linear regression for binary outcomes and get the same predictions as from a logistic regression after using a sigmoid function. That is what I awaited. But the surprising ...
LulY's user avatar
  • 341
1 vote
0 answers
20 views

GAMLSS - How to adjust AIC for log-transformation of variables [duplicate]

I have one GAMLSS model in which the independent and dependent variable are on the natural scale, and a second model in which both variables have been log-transformed. How do I adjust the AIC of the ...
Peder Holman's user avatar
0 votes
1 answer
27 views

Converting quarterly growth rate forecasts to yearly growth rate forecasts

I'm generating forecasts on a quarterly basis, focusing on metrics like the GDP growth rate for Brazil. These forecasts are presented as growth rates. For example, the forecast for 2020Q1 represents ...
Afiq Johari's user avatar
2 votes
0 answers
20 views

Construct transformations of random variables that are "more normal"

I am reading this page in the Encyclopedia of Mathematics about transformations of random variables. I am puzzled about the Example 2: Let $X_1,...,X_n,...$ be independent random variables, each ...
JJJuuu's user avatar
  • 21
1 vote
1 answer
13 views

Transforming data with two groups for multiple regression analysis

I have a dataset includes 2 groups (HC, PT) and unfortunately, variables (8) are not normal. I'm trying to run a multiple regression to see if the relationship between the variable of interest ("...
statnoob9's user avatar
1 vote
0 answers
29 views

Help needed in recreating ANOVA test results from study

I am aiming to recreate ANOVA tests as part of Study 1 from the following research paper related to some psychological factors: https://doi.org/10.3389/fpsyg.2021.738447 There are altogether 80 ...
Fluellen's user avatar
2 votes
1 answer
40 views

In linear regression, in what situation(s) would you transform the response variable BEFORE having checked the assumptions?

I have a non-normally distributed, right-skewed response (dependent) variable which will be used in an (OLS) linear regression model. Why would I want to transform this before having checked the ...
Jen's user avatar
  • 21
7 votes
2 answers
435 views

Advantage of GLMs over transformation models

According to the book I am currently reading, we should prefer GLMs over a simple transformation model (e.g. $\log(y) = x_i^T \beta + u_i$ ) The argument is derived by Jensen's inequality: $$ E(\log(\...
Marlon Brando's user avatar
3 votes
1 answer
46 views

Is it possible to perform a three-way ANOVA on non-normal data?

I am not very experienced in statistics so I'm not sure how to tackle this data. I have run an experiment where we recorded the proportion of time some beetles spent in 4 different habitats. Response ...
FredB's user avatar
  • 51
1 vote
1 answer
36 views

Why does transforming our independent variable improve forecasting?

When doing OLS regression, I've heard that transforming an explanatory (or dependent)* variables can (sometimes) improve the forecasting ability of the model. Theoretically, why does this work? I've ...
Michael Jones's user avatar
0 votes
0 answers
28 views

Detrending Time Series Data for Network Analysis

In order to run psychometric network analyses on (up to) 10 repeated measures for 13 variables in R, I need to detrend my 13 variables (B1 to B13) first. My problem is that not all subjects (id) have ...
Benjamin Telkamp's user avatar
2 votes
0 answers
49 views

How may I find the distribution of a transformation of multivariate random variables?

Forgive me if this question has already been asked on here, but I could only find posts if the multivariate random variables were multivariate Gaussian. Suppose we have two multivariate random ...
Ron Snow's user avatar
  • 2,103
3 votes
2 answers
323 views

Is the exponent of the MAD (Median Absolute Deviation) of log transformed Data measuring the relative distance from median in the untransformed data?

I want to confirm whether taking the Exponent of the MAD of Log Transformed Data gives me a measure of relative distance from median of the original untransformed data. So say I have a MAD of 0.2 for ...
Anon9001's user avatar
3 votes
1 answer
132 views

Are there any situations where orthogonality is not optimal?

Data reduction is often used to avoid overfitting and to enhance explainability. Popular data reduction techniques, such as SVD or PCA map/project high-dimensional data to a lower-dimensional ...
Chris M's user avatar
  • 39
2 votes
1 answer
39 views

Does OLS give the maximum likelihood estimation for a linear log model?

I'm fitting a model $y=a\times \log(x)+ b$ using standard scikit linear regression (wich uses OLS) and a transformation $x'=\log(x)$. My doubt is: the parameters I get for the model are the best one ...
Roger Danilo Figlie's user avatar
0 votes
0 answers
12 views

Modeling non-negative time series with square root decay?

Q: How should I model a non-negative time series $y_t$ which exhibits square-root decay? More specifically, a time series $y_t$ whose square-root differences $\sqrt{y_t}-\sqrt{y_{t-1}}$ are linear and ...
lowndrul's user avatar
  • 2,147
0 votes
0 answers
18 views

Leaving duplicated entries in a dataset at pretraining stage

I'm adopting a fine-tuning approach after having pretrained a deep learning model (transformer) on a source dataset (let's call it dataset A) and then fine-tuning it on a target dataset (B). Dataset A ...
James Arten's user avatar
1 vote
0 answers
28 views

Equivalent GLM's for common stabilising transformations

I'm familiar with applying a log-transform to a skewed outcome variable to improve model fit, but I've not thought further to link stabilising transforms to GLM's in general. Reading around it seems ...
LucaS's user avatar
  • 771
1 vote
1 answer
38 views

Transformation of a Random Variable

I am working on this problem for class, where the setup is the following: Let X be a single observation from the $beta(\theta,1)$ pdf. (a) Let $Y=-(logX)^{-1}$. Evaluate the confidence coefficient of ...
Harry Lofi's user avatar
0 votes
1 answer
48 views

plotting log-transformed data, but running statistics on raw data?

I intend to compare differences between means of eight groups. The differences between some of the means are only visible when I plot (in a box plot) the log-transformed data. However, I am unsure as ...
Rikki Franklin Frederiksen's user avatar
1 vote
0 answers
24 views

AIC correction for unusual transformations of the response

When using transformations of the response variable, one should correct the AIC of the transformed model in order to be possible to compare it to the unstransformed model (see here). The example used ...
droubi's user avatar
  • 11
1 vote
1 answer
35 views

Can log2 be substituted with ln in logDice association measure?

I am currently doing collocational analysis in the Russian National Corpus, to be precise the Russian national news subcorpus, to see what is the most significant collocates of the lemma "gay&...
pindakazen's user avatar
0 votes
0 answers
29 views

Fit on original data vs linear fit on transformed data

I asked a question yesterday (Better function to fit log-like data?) and the accepted answer got me thinking. For non-linear data, Is it better/more recommended to asses the goodness of fit on the ...
Gabriel's user avatar
  • 4,352
0 votes
0 answers
36 views

Transformations for extremely skewed data for Binary Logistic Regression

Good evening everyone, I have a dataset with 5000 observations, and 10 explanatory and 1 response variable (binary 0 or 1), and my task is to make a logistic regression model that can ideally predict ...
Alex Smirnov's user avatar
1 vote
1 answer
40 views

Question on Best Transformation (Negative, Zero, Positive Values) + Missing Data

I have a dataset with $5000$ observations, and 10 explanatory and 1 response variable (binary 0 or 1), and my task is to make a logistic regression model for prediction (but also needs to provide some ...
Alex Smirnov's user avatar
0 votes
0 answers
14 views

What is the standard performance metric for categorical data clustering?

I performed a categorical clustering with some selected UCI datasets. I one-hot encoded the features, then directly used Binomial Mixture Model and KModes using this one-hot encoded data. On the ...
NOT-A-CS-GUY's user avatar
4 votes
3 answers
385 views

How to Handle 0 and 1 in Logit Transformation? [closed]

I am planning to analyze experimental data using statistical methods, and I intend to perform analysis on repeated measurements using GEE (Generalized Estimating Equations) or RM ANOVA. Some of the ...
soobinism's user avatar
0 votes
2 answers
72 views

simple ANN as a set of linear transformations

We cannot classify the points of the XOR problem with a single perceptron in the hidden layer. However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, ...
Mag's user avatar
  • 1
2 votes
0 answers
14 views

Is there a correction for samples from a (linear) Prophet model when trained on an inhomogenous Poisson point process?

Facebook's Prophet is a popular modelling choice for time series forecasting in production due to many steps being automated (and thus convenient). This can sometimes lead to over-reliance on it when ...
Galen's user avatar
  • 9,421
3 votes
1 answer
43 views

Forecasting excess mortality with ARIMA model

I am using the forecast package by Prof Hyndman, and have had success fitting ARIMA models to excess mortality (from the COVID-19 pandemic) data. I am currently trying to produce plots for cumulative ...
Jina A.'s user avatar
  • 31
2 votes
0 answers
28 views

Is it possible to fit a linear model of y in log scale but with offset in the original scale?

Let's start with simple linear regression with log transformation of the response variable y: $$ \log(y_i) = \beta_0 + \beta_1x_i + e_i$$ (btw, how is this model called? log-linear regression or ...
Sofie's user avatar
  • 21
3 votes
1 answer
73 views

Transforming data for ANOVA or GLM

I am working with ecological count data in order to analyze differences/any contrast in species composition between warm and cold year communities. The abundances of species were recorded from ...
user390865's user avatar
1 vote
1 answer
40 views

Count data and proportion covariates: best practices

I'm working with spatial data and I have the following log-linear model for count data. Let $y \sim Poisson(\lambda_{i})$ such that $$ \log \lambda_{i} = \text{x}_i^\top\beta_{} + \epsilon_{i} $$ such ...
BelwarDissengulp's user avatar
2 votes
1 answer
151 views

What kind of data that I need to do my PCA?

So I have a not normal data (I did saphiro test and the result said it's not normal) Then, I did data transform with log. So the data went normal. Does the log data can work for my pca? Or should I ...
Nimas Pertiwi's user avatar
3 votes
0 answers
53 views

Is there a by-group interaction issue after the Box-Cox transformation?

I've come across a question that has me a bit stumped and hope to seek your valuable insights. Specifically, I've been working with the Box-Cox Transformation to normalize dependent variables within ...
Elizabeth's user avatar

1
2 3 4 5
50