Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Resample data from a histogram with higher resolution - assuming it follows a normal distribution

I have a problem which has me stumped on what to try next. I have some data from a farm, related to yield over a period of days, for a defined area. I have daily-resolution data, where the day's ...
0
votes
0answers
8 views

WOE of raw variable vs WOE of log of same variable | What can be the impact? [closed]

Should I expect different results if I transform "var 1" to WOE compared to "log(var 1)" to WOE?
1
vote
0answers
14 views

Modelling outcome variable with lognormal distribution

I have an outcome variable with a distribution as follows, and I need to do some regression modelling. I know that often such variables are transformed to get a normal distribution. However, it would ...
0
votes
0answers
10 views

How does mean of data change in log transformation? [closed]

How does variance and mean of Y = log(X+1) change given we know the mean and variance of X which is also a lognormal distribution (Poisson Distribution). I figured that variance is smaller for Y ...
0
votes
0answers
11 views

Whitening MNIST in R

I am analyzing the famous MNIST data set with R. I have never been analyzing handwritten digit data before, so I wonder wether it makes sense to whiten the 756-dimensional data in the sense that from ...
0
votes
0answers
24 views

Yeo-Johnson did not improve model. Can that really be? [closed]

I have a dataset which has several continuous columns including my y-variable. Most of these columns are non-normally distributed and some of them have also negative values. For that reason, I tried a ...
0
votes
0answers
14 views

Regression : Variable transformation necessary? If yes, why?

I am working on a real estate project and have historical rental prices and vacancy data. Interested in exploring the relationship between vacancy rates and change in rental prices. The unit of ...
0
votes
0answers
22 views

Boxcox Transformation Error - “contains observations that are not strictly positive”

I am trying to do a Box-Cox transformation for a data set in Stata, using the following code: boxcox Production_qty price technology gender, model(lambda) notrans(gender) It gave me an error saying &...
0
votes
0answers
35 views

Choosing a statistical method for customer preferences

I have received a data set about the ingredients of sweets of different brands, as well as information about prices in percent, sugar and profit in percent. The information on the ingredients are ...
0
votes
0answers
9 views

Quantify data leakage train test set for feature scaling

Topic: First appling feature scaling (e.g. standardisation) to a data set and then splitting the data set into train and test set, can introduce data leakage. Question: Although this is clear to me ...
0
votes
0answers
30 views

Log-transforming values on different scales

I'm trying to train a neural network to classify time series that represent the times between the sales of consumer products. I am training a single model across multiple products (spanning across ...
0
votes
0answers
14 views

How to Construct an Index

I'm sorry in advance that this question is a little bit broad. I'm trying to create on variable to measure the overall prestige of a university. I'm using data on federal obligation, tuition, as well ...
0
votes
0answers
16 views

power transformation with negative values?

Is there a name for this type of transformation: \begin{equation} sgn(x) * |x|^p \end{equation} where $p$ is an arbitrary number (e.g., 1/2 and 1/3 for square and cube roots, respectively) and $sgn$ ...
0
votes
0answers
21 views

When transforming data, why use resampling to estimate transformed values?

I am working through a machine learning book and am using the caret package in R. There is a function, preProcess, that uses ...
1
vote
1answer
91 views

Generate null distribution from pvalues

I have a set of experiments on which I apply the Fisher's exact test to statistically infer changes in cellular populations. Some of the data are dummy experiments that model our control experiments ...
1
vote
0answers
14 views

Document AI : Using FUNSD dataset to train a GNN to classify 'Linked' entities

I have been using the FUNSD dataset to predict sequence labeling in unstructured documents per this paper: LayoutLM: Pre-training of Text and Layout for Document Image Understanding . The data after ...
2
votes
2answers
114 views

Matrix and vectors, why different notation for dimensions?

If we collect data and put it into a matrix of size (100,3), we tend to say we have three-dimensional data. We think of each column as a dimension. On the other side, if we have a vector of size (100,...
0
votes
0answers
18 views

How should one transform a variable in which the further away from 0, the more significant it is?

Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
0
votes
3answers
87 views

Modelling the logarithm of a response

My response variable is positive and I decided to model the logarithm of the response. Some of the values are zero. For this reason I modelled $Z = \log(Y + 0.1)$. When I transform back, some of my ...
0
votes
0answers
23 views

Stationary Time series is showing better results when predicting(ARIMA) after differencing

I have a time series of daily maximum temperature of a city for 2 years 3 months. I removed the seasonality from the data by subtracting present values with the past year values(seasonal differencing)....
0
votes
0answers
15 views

Coefficient of Variation on Log tranformed data, Canchola formula vs. raw formula

I am new to this community and I am currently confused. My intended use for coefficients of variation (CV) is to assess precision between repeated measurements of a clinical tool. We wish to assess ...
1
vote
1answer
23 views

Can we normalize both continuous and discrete numerical values

I have a sensor dataset with 16 features as numerical values (12 are continuous and 4 are discrete). I am using LSTM model to fit the data and do some classification. As both continuous and discrete ...
0
votes
0answers
17 views

transforming regression variable

I am would like to create a regression model with different variables however before using these variables in my regression model I would like to transform the variable in order to make it more ...
0
votes
0answers
28 views

Box-Cox formation with model selection, regularization, etc

As my data is not normally distributed, I performed the Box-Cox Transformation on the response. ...
2
votes
2answers
39 views

How to generate 90% of the data in Exponential distribution above some threshold value in R

I want to generate a dataset from the Exponential distribution, where 90% of the observations are above a threshold value 0.15. ...
1
vote
1answer
45 views

How to make a johnson unbounded transformation to make my data more gaussian like? in python

I am a novice in stats and I would like to transform my data (house prices) using a johnson unbounded distribution to look more gaussian. I looked at pandas transform() but I can't really understand ...
0
votes
0answers
29 views

Is probit transformation the same as probability integral transform?

The image shows the original marginal data $u$ and $v$ on the left, which has a bounded support, and their probit transformations $r$ and $s$ on the right, which has an unbounded support. The ...
0
votes
0answers
19 views

Box Cox Transformation in auto.arima

I'm working with ARIMA models and was wondering about the necessity of BoxCox Transformation. When applying BoxCox on my training-set BoxCox.lambda(train) it ...
2
votes
1answer
53 views

Can this be written as a linear model?

$$y_i=\frac{(x_{0i}\beta_0+x_{1i})+\log(\beta_1^2 x_{2i})}{x_{3i}}+e_i\quad,\,i=1,\ldots,n,\,x_{pj}>0$$ I am wondering if this can be written as a linear model, I didn't think so because of the ...
3
votes
1answer
66 views

Up to what number of distinct values should I transform a categorical variable in a dummy variable?

When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
0
votes
0answers
25 views

why to take logs and remove mean?

I am replicating a paper (Gil Kim and David Vera 2017). They transform the price of oil by doing the following; "for the real price of oil, we first deflate it, transform it into natural log and ...
1
vote
0answers
15 views

Seasonal ARIMA lag differencing p-value not significant

I am using the below data to forecast using seasonal ARIMA model. I see at d=1 the p-value is not significant. But still ...
2
votes
1answer
63 views

Normality and homoscedasticity are lacking: Is transformation necessary?

I'm a student and I'm very new at this so I wanted to ask what to do. I have a data set and one of the groups didn't pass Shapiro-Wilk normality test (p value = 0.01) but testing with model residuals ...
1
vote
1answer
16 views

Constant-information scale transformation

I was recently introduced to the concept of constant information scale transformations in the book Generalized Linear Model with Examples in R, by Dunn and Smyth. With that, they mention in the book ...
0
votes
0answers
59 views

Data transformations and standard errors

I have reaction times (as a dependent variable), which I log-translate using the natural logarithm, to get it normally distributed. Then I run a regression on the log-translated data. I translated the ...
0
votes
0answers
22 views

If in the ARIMA model the residuals is not normally distributed then what should be the way to make it normally distributed?

If in the ARIMA model the data is not normally distributed then what should be the way to make it normally distributed?
1
vote
0answers
44 views

Alternating Conditional Expectations: Multiple regression transform

Alternating Conditional Expectations (ACE) is a non-parametric algorithm for multiple regression transform selection. It finds a set of transformed response variables that maximizes $R^2$ using ...
0
votes
0answers
10 views

Impact of having negative Box-Cox lambdas on a Tau correlation matrix

I'm trying to make the marginal distributions of my data easier to fit by first applying a Box-Cox transformation and then standardizing the transformed data. My concern is, as this is a multivariate ...
0
votes
0answers
24 views

Should highly skewed bivariate data be log transformed before doing copula analysis?

I have $(X, Y)$ data sets of two random variables. I want to apply copulas to obtain their joint probability distribution function. My data are highly skewed. In general, is it advisable to apply log ...
0
votes
0answers
9 views

Joining semi-related tables and predicting counts based on generalised proportions

I am working with UK census data and have found it to be quite restrictive when it comes to combining multiple variables. I have managed to get 3 separate tables that are relevant for a project of ...
24
votes
4answers
2k views

Why are log probabilities useful?

Probabilities of a random variable's observations are in the range $[0,1]$, whereas log probabilities transform them to the log scale. What then is the corresponding range of log probabilities, i.e. ...
0
votes
1answer
64 views

How can I calculate the mean and variance of a linearly transformed random variable?

Say I have a random variable $x$, with mean $\mu_x=35$ and standard deviation $\sigma_x=10$. I want to linearly transform $x$ to $y$ according to the formula $y=a+bx$ so that $\mu_y=100$ and $\sigma_y=...
0
votes
0answers
11 views

What can go wrong if we don't scale and center data in a predictive model? [duplicate]

I have started using R recently, along with the caret package. I note that there is an option to preprocess data so that it is scaled and centered. I am interested in multiple linear regression. What ...
0
votes
0answers
5 views

Voters-party congruence on environmental protection: How I can moderate the relation using issue salience?

Right now I am using two very well-known datasets for political scientists. The first one is CHES data, an experts-survey that defines european political parties' positions on various policy issues. ...
4
votes
1answer
276 views

Providing latitude and longitude to a house price model

I'm new to machine learning, and I'm trying to get a sense of how you optimize data for a model. I'm following this official Kaggle tutorial, which teaches the basics of machine learning through house ...
0
votes
0answers
12 views

Transforming data on a per-instance basis

I am currently facing the following situation (in my feature engineering process): I am inspecting a step function for a set of instances (step function per instance), i.e., in each iteration "i&...
2
votes
0answers
46 views

lmer after Box-Cox transformation

Are the residuals close enough to normality after Box Cox transformation using the MASS package? ...
1
vote
1answer
68 views

Is there any possible statistical test for this data?

I have a data set of the daily dosage of a drug that participants take against some signs and symptoms. The data are highly skewed to the left because there are a lot of patients with 0mg per day of ...
0
votes
0answers
2 views

Should transformed models be back-transfored before penalization? And should ensemble forecasts from these also be penalized? From the same data?

I have generated a number of regression models in an effort to predict a particular outcome. All the models are based on the same data over the same period. They include, e.g., a vector error ...
1
vote
1answer
31 views

Linear transformation *not* to use to scale mean and SD

I'm working through the Exercises in Regression and Other Stories. Exercise 3.6 on Linear transformations asks you to provide a formula to rescale a variable with mean = 35 and SD = 10 to have mean = ...

1
2 3 4 5
40