# Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

1,990 questions
Filter by
Sorted by
Tagged with
5 views

### Resample data from a histogram with higher resolution - assuming it follows a normal distribution

I have a problem which has me stumped on what to try next. I have some data from a farm, related to yield over a period of days, for a defined area. I have daily-resolution data, where the day's ...
7 views

### WOE of raw variable vs WOE of log of same variable | What can be the impact? [closed]

Should I expect different results if I transform "var 1" to WOE compared to "log(var 1)" to WOE?
14 views

### Modelling outcome variable with lognormal distribution

I have an outcome variable with a distribution as follows, and I need to do some regression modelling. I know that often such variables are transformed to get a normal distribution. However, it would ...
10 views

### How does mean of data change in log transformation? [closed]

How does variance and mean of Y = log(X+1) change given we know the mean and variance of X which is also a lognormal distribution (Poisson Distribution). I figured that variance is smaller for Y ...
11 views

### Whitening MNIST in R

I am analyzing the famous MNIST data set with R. I have never been analyzing handwritten digit data before, so I wonder wether it makes sense to whiten the 756-dimensional data in the sense that from ...
24 views

### Yeo-Johnson did not improve model. Can that really be? [closed]

I have a dataset which has several continuous columns including my y-variable. Most of these columns are non-normally distributed and some of them have also negative values. For that reason, I tried a ...
14 views

### Regression : Variable transformation necessary? If yes, why?

I am working on a real estate project and have historical rental prices and vacancy data. Interested in exploring the relationship between vacancy rates and change in rental prices. The unit of ...
22 views

### Boxcox Transformation Error - “contains observations that are not strictly positive”

I am trying to do a Box-Cox transformation for a data set in Stata, using the following code: boxcox Production_qty price technology gender, model(lambda) notrans(gender) It gave me an error saying &...
35 views

### Choosing a statistical method for customer preferences

I have received a data set about the ingredients of sweets of different brands, as well as information about prices in percent, sugar and profit in percent. The information on the ingredients are ...
9 views

### Quantify data leakage train test set for feature scaling

Topic: First appling feature scaling (e.g. standardisation) to a data set and then splitting the data set into train and test set, can introduce data leakage. Question: Although this is clear to me ...
30 views

### Log-transforming values on different scales

I'm trying to train a neural network to classify time series that represent the times between the sales of consumer products. I am training a single model across multiple products (spanning across ...
14 views

### How to Construct an Index

I'm sorry in advance that this question is a little bit broad. I'm trying to create on variable to measure the overall prestige of a university. I'm using data on federal obligation, tuition, as well ...
16 views

### power transformation with negative values?

Is there a name for this type of transformation: \begin{equation} sgn(x) * |x|^p \end{equation} where $p$ is an arbitrary number (e.g., 1/2 and 1/3 for square and cube roots, respectively) and $sgn$ ...
21 views

### When transforming data, why use resampling to estimate transformed values?

I am working through a machine learning book and am using the caret package in R. There is a function, preProcess, that uses ...
91 views

### Generate null distribution from pvalues

I have a set of experiments on which I apply the Fisher's exact test to statistically infer changes in cellular populations. Some of the data are dummy experiments that model our control experiments ...
14 views

### Document AI : Using FUNSD dataset to train a GNN to classify 'Linked' entities

I have been using the FUNSD dataset to predict sequence labeling in unstructured documents per this paper: LayoutLM: Pre-training of Text and Layout for Document Image Understanding . The data after ...
114 views

### Matrix and vectors, why different notation for dimensions?

If we collect data and put it into a matrix of size (100,3), we tend to say we have three-dimensional data. We think of each column as a dimension. On the other side, if we have a vector of size (100,...
18 views

### How should one transform a variable in which the further away from 0, the more significant it is?

Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
87 views

### Modelling the logarithm of a response

My response variable is positive and I decided to model the logarithm of the response. Some of the values are zero. For this reason I modelled $Z = \log(Y + 0.1)$. When I transform back, some of my ...
23 views

### Stationary Time series is showing better results when predicting(ARIMA) after differencing

I have a time series of daily maximum temperature of a city for 2 years 3 months. I removed the seasonality from the data by subtracting present values with the past year values(seasonal differencing)....
15 views

### Coefficient of Variation on Log tranformed data, Canchola formula vs. raw formula

I am new to this community and I am currently confused. My intended use for coefficients of variation (CV) is to assess precision between repeated measurements of a clinical tool. We wish to assess ...
23 views

### Can we normalize both continuous and discrete numerical values

I have a sensor dataset with 16 features as numerical values (12 are continuous and 4 are discrete). I am using LSTM model to fit the data and do some classification. As both continuous and discrete ...
17 views

### transforming regression variable

I am would like to create a regression model with different variables however before using these variables in my regression model I would like to transform the variable in order to make it more ...
28 views

### Box-Cox formation with model selection, regularization, etc

As my data is not normally distributed, I performed the Box-Cox Transformation on the response. ...
39 views

### How to generate 90% of the data in Exponential distribution above some threshold value in R

I want to generate a dataset from the Exponential distribution, where 90% of the observations are above a threshold value 0.15. ...
45 views

### How to make a johnson unbounded transformation to make my data more gaussian like? in python

I am a novice in stats and I would like to transform my data (house prices) using a johnson unbounded distribution to look more gaussian. I looked at pandas transform() but I can't really understand ...
29 views

### Is probit transformation the same as probability integral transform?

The image shows the original marginal data $u$ and $v$ on the left, which has a bounded support, and their probit transformations $r$ and $s$ on the right, which has an unbounded support. The ...
18 views

### Box Cox Transformation in auto.arima

I'm working with ARIMA models and was wondering about the necessity of BoxCox Transformation. When applying BoxCox on my training-set BoxCox.lambda(train) it ...
53 views

### Can this be written as a linear model?

$$y_i=\frac{(x_{0i}\beta_0+x_{1i})+\log(\beta_1^2 x_{2i})}{x_{3i}}+e_i\quad,\,i=1,\ldots,n,\,x_{pj}>0$$ I am wondering if this can be written as a linear model, I didn't think so because of the ...
66 views

### Up to what number of distinct values should I transform a categorical variable in a dummy variable?

When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
25 views

### why to take logs and remove mean?

I am replicating a paper (Gil Kim and David Vera 2017). They transform the price of oil by doing the following; "for the real price of oil, we first deflate it, transform it into natural log and ...
15 views

### Seasonal ARIMA lag differencing p-value not significant

I am using the below data to forecast using seasonal ARIMA model. I see at d=1 the p-value is not significant. But still ...
63 views

### Normality and homoscedasticity are lacking: Is transformation necessary?

I'm a student and I'm very new at this so I wanted to ask what to do. I have a data set and one of the groups didn't pass Shapiro-Wilk normality test (p value = 0.01) but testing with model residuals ...
16 views

### Constant-information scale transformation

I was recently introduced to the concept of constant information scale transformations in the book Generalized Linear Model with Examples in R, by Dunn and Smyth. With that, they mention in the book ...
59 views

### Data transformations and standard errors

I have reaction times (as a dependent variable), which I log-translate using the natural logarithm, to get it normally distributed. Then I run a regression on the log-translated data. I translated the ...
22 views

### If in the ARIMA model the residuals is not normally distributed then what should be the way to make it normally distributed?

If in the ARIMA model the data is not normally distributed then what should be the way to make it normally distributed?
44 views

### Alternating Conditional Expectations: Multiple regression transform

Alternating Conditional Expectations (ACE) is a non-parametric algorithm for multiple regression transform selection. It finds a set of transformed response variables that maximizes $R^2$ using ...
10 views

### Impact of having negative Box-Cox lambdas on a Tau correlation matrix

I'm trying to make the marginal distributions of my data easier to fit by first applying a Box-Cox transformation and then standardizing the transformed data. My concern is, as this is a multivariate ...
24 views

### Should highly skewed bivariate data be log transformed before doing copula analysis?

I have $(X, Y)$ data sets of two random variables. I want to apply copulas to obtain their joint probability distribution function. My data are highly skewed. In general, is it advisable to apply log ...
9 views

### Joining semi-related tables and predicting counts based on generalised proportions

I am working with UK census data and have found it to be quite restrictive when it comes to combining multiple variables. I have managed to get 3 separate tables that are relevant for a project of ...
2k views

### Why are log probabilities useful?

Probabilities of a random variable's observations are in the range $[0,1]$, whereas log probabilities transform them to the log scale. What then is the corresponding range of log probabilities, i.e. ...
64 views

### How can I calculate the mean and variance of a linearly transformed random variable?

Say I have a random variable $x$, with mean $\mu_x=35$ and standard deviation $\sigma_x=10$. I want to linearly transform $x$ to $y$ according to the formula $y=a+bx$ so that $\mu_y=100$ and \$\sigma_y=...
11 views

### What can go wrong if we don't scale and center data in a predictive model? [duplicate]

I have started using R recently, along with the caret package. I note that there is an option to preprocess data so that it is scaled and centered. I am interested in multiple linear regression. What ...
5 views

### Voters-party congruence on environmental protection: How I can moderate the relation using issue salience?

Right now I am using two very well-known datasets for political scientists. The first one is CHES data, an experts-survey that defines european political parties' positions on various policy issues. ...
276 views

### Providing latitude and longitude to a house price model

I'm new to machine learning, and I'm trying to get a sense of how you optimize data for a model. I'm following this official Kaggle tutorial, which teaches the basics of machine learning through house ...
12 views

### Transforming data on a per-instance basis

I am currently facing the following situation (in my feature engineering process): I am inspecting a step function for a set of instances (step function per instance), i.e., in each iteration "i&...
46 views

### lmer after Box-Cox transformation

Are the residuals close enough to normality after Box Cox transformation using the MASS package? ...
68 views

### Is there any possible statistical test for this data?

I have a data set of the daily dosage of a drug that participants take against some signs and symptoms. The data are highly skewed to the left because there are a lot of patients with 0mg per day of ...