Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
15
votes
3answers
78k views

Transforming extremely skewed distributions

Assume that I have a variable whose distribution is skewed positively to a very high degree, such that taking the log will not be sufficient in order to bring it within the range of skewness for a ...
10
votes
3answers
25k views

Linear model Heteroscedasticity

I have the following linear model: To address the residuals heteroscedasticity I have tried to apply a log transformation on the dependent variable as $\log(Y + 1)$ but I still see the same fan out ...
13
votes
3answers
5k views

Express answers in terms of original units, in Box-Cox transformed data

For some measurements, the results of an analysis are appropriately presented on the transformed scale. In most of the cases, however, it's desirable to present the results on the original scale of ...
9
votes
2answers
6k views

Empirical logit transformation on percentage data

I have already used the logit transform on my outcome variables (which are displayed in percentages). However, this obviously gives me -INF values and since my data includes a lot of zeros in some ...
7
votes
1answer
5k views

Interpretation of log(1 + x) transformed predictor

Interpretation of log transformed predictor neatly explains how to interpret a log transformed predictor in OLS. Does the interpretation change if there are 0s in ...
5
votes
2answers
16k views

How to log transform data with a large number of zeros

I have a dataset comprised of continuous values that have about 30-50% zeros and a large range (10^3 - 10^10). I believe these zeros are not a result of missing data and are the result of the ...
1
vote
1answer
11k views

making predictions with log-log regression model

Is it necessary to exponentiate the predicted values in a log-log regression model? For example my model is: $\log(y) = \log(x)$ $\log(y) = -0.5141 + 0.5377 \log(x)$ if I wanted to make a ...
7
votes
2answers
9k views

Best way to optimize MAPE

The MAPE is a metric that can be used for regression problems : $$\mbox{MAPE} = \frac{1}{n}\sum_{t=1}^n \left|\frac{A_t-F_t}{A_t}\right|$$ Where $A$ represents the actual value and $F$ the the ...
6
votes
1answer
10k views

Why use differencing and Box-Cox in time series?

Why use Differencing and Box-Cox transformation in a time series? From what I read the usefulness of the procedures are Differencing: Making a time series stationary and stabilize the mean Box-Cox: ...
5
votes
1answer
3k views

How can I estimate theta for the inverse hyperbolic sine transformation?

I would like help with R code to estimate theta for the Inverse Hyperbolic Sine Transformation. This transformation is useful to transform skewed data that contain negative values or zeros. There ...
4
votes
1answer
11k views

Interpreting Log-Transformed Percentages in OLS

In a log-log model, such as $\log(y) = b_0 + b_1 \log(x)$, I know that with OLS the standard interpretation is a "1% increase in x is associated with a $b_1$% increase in y." I have three related ...
24
votes
4answers
2k views

Why are log probabilities useful?

Probabilities of a random variable's observations are in the range $[0,1]$, whereas log probabilities transform them to the log scale. What then is the corresponding range of log probabilities, i.e. ...
15
votes
3answers
1k views

CDF raised to a power?

If $F_Z$ is a CDF, it looks like $F_Z(z)^\alpha$ ($\alpha \gt 0$) is a CDF as well. Q: Is this a standard result? Q: Is there a good way to find a function $g$ with $X \equiv g(Z)$ s.t. $F_X(x) = ...
8
votes
2answers
2k views

Using self organizing maps for dimensionality reduction

Over the past few days, I have been conducting some research on self organizing maps for a project at school. I have come to understand that self organizing maps can be used to reduce the ...
7
votes
2answers
3k views

Zero-inflated two-part models for semi-continuous data

I am trying to study predictors of companies' pollution output of some specific chemicals. The data I am using have many 0's (i.e., the company did not pollute at all with those chemicals) and then ...
5
votes
1answer
927 views

Does a log transform always bring a distribution closer to normal?

I have a highly right skewed data set with a large range of values (from 1 ~ 10^6) (can't share the actual data for work related reasons). When I plot the log of the data instead, the distribution ...
4
votes
1answer
3k views

Data transformation for Principal Components Analysis from different Likert scales

I have data from a survey comprised of several measures that used different Likert-type scaling (4-, 5-, and 6-point scales). I would like to run a principal components analysis using the data from ...
3
votes
2answers
21k views

How to split a numeric variable into a binary low-high variable

I have measured frequency of a certain behavior on 15 individuals. I would like to create two groups based on the amount of this behaviour that was observed (i.e., a group exhibiting high levels of ...
2
votes
1answer
7k views

Min-Max scaling on Z-score standardizd data?

For a specific task of score fusion I need to test my data on some different normalization techniques like typical Z-normalization or Sigmoid-normalization. This is my first step to do. For a second ...
13
votes
2answers
419 views

Is visualization sufficient rationale for transforming data?

Problem I would like to plot the variance explained by each of 30 parameters, for example as a barplot with a different bar for each parameter, and variance on the y axis: However, the variances are ...
8
votes
3answers
16k views

Exponential of a standard normal random variable

We know that $Z\sim N(0, 1)$. How do I prove that $e^Z$ has a mean of $e^{0.5}$? I have tried integrating $e^z$ times the pdf of $Z$ but I don't know why it isn't working out. Also what is the pdf ...
7
votes
2answers
10k views

Logistic regression with an log transformed variable, how to determine economic significance

I am using a logistic regression model with continuous independent variables and two log transformed size variables (total assets and total deposits). My question is how to interpret the results and ...
7
votes
3answers
10k views

Two-way ANOVA with count data

Can we report a Two-way ANOVA with count data? If Yes, What are your references? If No, Why? For example: Factor A in 4 level and Factor B in 3 level and our responses are number of patients.
6
votes
1answer
840 views

What is Box-Cox regression?

I am confused about the terminology: What is Box-Cox regression? Is it applying Box-Cox power transformation and then running a linear regression? Is there any relationship between "Box-Cox regression"...
5
votes
2answers
8k views

How to standardize data for hierarchical clustering?

When running hierarchical clustering analysis of a matrix of individuals x samples (e.g., employee performances across different days), there are several ...
4
votes
2answers
17k views

How to transform negative data to be homoscedastic

I have a bunch of data that's both positive and negative. Its calculated from the residuals of an ANOVA (i.e. specific leaf area calculated as the residuals of an ANOVA of leaf area with leaf blade ...
3
votes
2answers
2k views

Kriging on log transformed rainfall data

I am beginner in R. I had found in the literature that prior to performing kriging on the data, the distribution has to be investigated to check if it is Gaussian. So, in order to check if the data ...
2
votes
1answer
357 views

Including both transformed and original data (untransformed) in a multivariable linear regression.

This may have a quick response (i.e. don't do it). Just attended a lecture on multivariable linear regression where the outcome is forced expiratory volume (the amount of air you can push out of ...
2
votes
1answer
378 views

Variable frequency redistribution

I know that there is a way to "redistribute the frequencies of a variable" as stated here: Slide number 14 and 15 about redistribution and here: Dorian Pyle book chapter 7 section 2 paragraph 3 (7....
11
votes
2answers
13k views

Convert Poisson distribution to normal distribution

I primarily have a computer science background but now I am trying to teach myself basic stats. I have some data which I think has a Poisson distribution I have two questions: Is this a Poisson ...
11
votes
1answer
9k views

Back-transformed confidence intervals

Having come across this discussion I'm raising the question on the back-transformed confidence intervals conventions. According to this article the nominal coverage back-transformed CI for the mean ...
8
votes
2answers
351 views

How to find a suitable association of color with data value in a visualization?

I'm working on a software project that involves creating a visualizer for flood simulations. As part of this project, I've created a water gradient that shows water depth at particular points. To set ...
6
votes
3answers
12k views

How to log transform Z-scores?

The data for my variable is in the form of Z-scores only. I'd like to log transform the scores, but I don't know the mean or standard deviation in order to covert to raw scores. Can I assign an ...
5
votes
1answer
3k views

Including ordinal independent variables in a linear mixed effects model (using the lme4 package in R)

I am analyzing data from cohort of 500 calves investigating the impact of disease on growth. My outcome variables are normally distributed, continuous data. I am using hierarchical models with calf ...
5
votes
1answer
302 views

What if a transformed variable yields more normal and less heteroskedastic residuals but lower $R^2$?

I am trying to decide whether to use a square root transformed dependent variable in multiple linear regression. Transforming $y$ leads to more normally distributed residuals and also to less ...
5
votes
0answers
111 views

Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
4
votes
2answers
2k views

How can I prepare the input layer for recurrent neural network if there are many categorical variables?

I am building a recurrent neural network (RNN). The feature set contains many categorical variables. Some of them are like users and items. In this case, if I use one-hot encoding and concatenate ...
3
votes
2answers
2k views

Determine when time-series should be logged (or any other transformation) and applied automatically

Is there any way to test whether a series should be logged or transformed in another way? I have a code of which i use to run lots of different data through to forecast. Some of the data definitely ...
2
votes
1answer
881 views

Dimensionality reduction of multiple signals using Fourier transform

I have $N$ recorded signals, $x$, each of which have been sampled 672 times across a time period of a week (= 15 min intervals). I will denote $x_{ij}$ as the $j$th sample for the $i$th recorded ...
2
votes
3answers
2k views

Questions on standard deviation of a time series

If the standard deviation of a economic time series is approximately proportional to its level, that is, the standard deviation is well expressed as a percentage of the level of the series, then the ...
2
votes
2answers
6k views

Can I mix different data transformations in the same model?

I assume it is OK to mix different data transformations in the same analysis. I've had to transform some variables to squared and some to cubed in order to meet normal distribution requirements. I'm ...
1
vote
1answer
88 views

Help me about using ARIMA forecasting rainfall [closed]

I am currently using the ARIMA provided in R. I use the data as the rainfall time series in QuyNhon (Vietnam) from 2000 to 2017 to forecast rainfall for the next several years. I wish that the ...
4
votes
3answers
5k views

fitting a distribution to skewed data with negative values

I am trying to model data about altruistic behavior in a simple lab experiment. I have one value for each participant in the sample (N=479), describing how altruistic that person was. As you can see ...
4
votes
1answer
1k views

Prediction interval on untransformed scale

I am attempting to use simple linear regression to construct a 95% prediction interval for a continuous response variable (Y) using a continuous input variable (X). When examining my data, I realized ...
3
votes
3answers
1k views

Ratio of explanatory variables in multiple regression

I wonder if anyone has any links or advice on specifying a ratio of two explanatory variables in a linear regression? That is, specifying the two independent variables plus their ratio. We have data ...
3
votes
1answer
298 views

Regression when response variable is a function

I have a set of data $(X_i,Y_i)$, $i=1,\ldots,n$ where $X$ and $Y$ are supposed to satisfy the following equation $$ y = \beta_0(1+x^2)^{\beta_1},\quad x>0, \quad\quad (1) $$ I am interested in ...
3
votes
1answer
463 views

Box Cox Transformation makes Out of sample Forecast Error worse?

I am doing a regression on time series data. I have 60 lagged predictors which I will call x to predict a continuous variable y. I used the BoxCox function from the forecast package to transform y and ...
3
votes
2answers
20k views

Error in boxcox.default(y ~ x) : response variable must be positive

Error in boxcox.default(y ~ x) : response variable must be positive I am getting this error in R when I am performing a Box-...
3
votes
0answers
861 views

Box-Cox transformation vs. predictor transformation in multiple linear regression

I first tried to transform my predictors one by one, but I couldn't find a satisfactory transformation. Then I tried Box-Cox which solved a lot of linearity and constant variance issues, but I still ...
3
votes
1answer
85 views

Ways to modify data minimally while the variables to follow the desired covariances

Let $\bf X$ be the p-variate dataset. I want to modify the data to p-variate data $\bf Y$ so that its variables satisfy ...

1 2
3
4 5
7