Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

When transforming data, why use resampling to estimate transformed values?

I am working through a machine learning book and am using the caret package in R. There is a function, preProcess, that uses ...
0
votes
1answer
319 views

Should the multivariate Box-Cox lambda value, of a variable against itself, be 1?

I have 10 variables, and am trying to determine which transformation between each variable provides the best linear relationship. To this end, I am using the Box-Cox method to determine a power ...
2
votes
2answers
115 views

Matrix and vectors, why different notation for dimensions?

If we collect data and put it into a matrix of size (100,3), we tend to say we have three-dimensional data. We think of each column as a dimension. On the other side, if we have a vector of size (100,...
0
votes
1answer
126 views

R auto.arima with transformed covariates

I have a non-stationary output time-series (oil prices) that is to be forecasted with 20 different input time series. The series are all non-stationary. I am considering two approaches. Approach 1) ...
1
vote
0answers
49 views

Comparing linear models between categories after different transformations

Suppose I have a predictor variable $X$ split into two categories $X_a$ and $X_b$, and a response variable $Y$. I wish to perform linear regression but need to transform both $X_a$ and $X_b$ as to be ...
3
votes
0answers
1k views

Box-Cox, log or arcsine transformation? [closed]

Box-Cox, log and arcsine transformations have the aim of make the data more Normal. My question is: how can I choose between each one of these transformations? Which assumptions do I need to have to ...
1
vote
0answers
138 views

How to interpretation of the results of PCA [closed]

There is a larger matrix (1500 rows x 40 columns), 1500 observations x 40 variables. then I follow the procedures of PCA(Principle components analysis), 1. find correlation 2. find eigenvalues 3. ...
0
votes
0answers
68 views

Can I perform Z-score values on percentages?

Can I perform Z-score values on percentages? It is correct? Or should I perform Z-values starting from frequencies? I want to perform Z-score on percentages to 'declosed' my data, as, If I understood ...
0
votes
0answers
9 views

Joining semi-related tables and predicting counts based on generalised proportions

I am working with UK census data and have found it to be quite restrictive when it comes to combining multiple variables. I have managed to get 3 separate tables that are relevant for a project of ...
2
votes
0answers
52 views

Curvature of a pencil's stroke

I'm trying to evaluate the curvature of an image of a pencil stroke based on the image's pixels and their shade of gray. I'm trying to get the curvature of the line at every point of the stroke. The ...
1
vote
0answers
47 views

which transformations of dependent variable are allowed?

I was wondering which transformations of dependent variable (y) are "allowed"? My problem is that I am trying to compare three groups (=categorical variable), basically to find out if the groups have ...
17
votes
3answers
5k views

Extract data points from moving average?

Is it possible to extract data points from moving average data? In other words, if a set of data only has simple moving averages of the previous 30 points, is it possible to extract the original data ...
0
votes
0answers
18 views

How should one transform a variable in which the further away from 0, the more significant it is?

Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
1
vote
1answer
117 views

How to interpret a specific data transformation?

I came across this specific data transformation in the context of a physics application, which by itself is rather complex and hence out of the scope of this question. However since this ...
3
votes
0answers
2k views

Is Freeman — Tukey's transformation the most powerful for percentages?

I've described a typical design for my experiments in this question. Well, 1-way RM ANOVA assumes a Gaussian distributed vector. I try $y=\arcsin{\sqrt{x}}$. But for some data it works, for some doesn'...
0
votes
0answers
155 views

Wiener transformation - K-Apriori Algorithm

Wiener transformation is used to transform a binary data to a real data? How it works? Is there a R package that implements this transformation? I have been searching about Market Basket Analysis and ...
0
votes
0answers
147 views

Transforming data with large negative and large positive values [duplicate]

The data I'm trying to analyze are the quadratic estimates from a quadratic fit to a curve. Most of the data vary between -.15 and .15. However, I have outliers in both directions up to things like -...
1
vote
0answers
47 views

transforming dependent bivariate data to independent data

Assume a bivariate data that has some sort of dependency between the two variables is generated by unknown distribution, but not bivariate normal. Is it possible to remove the dependency of the two ...
1
vote
0answers
31 views

Transforming a multivariate normal sample using the sinh-arcsinh transform

Let us say that we have sample from the multivariable normal distribution. I would like to understand how is possible to apply a transformation to this sample to produce sample that has the sinh-...
0
votes
0answers
17 views

Transformation of Time and Temperature into Aggregate Score

Is anyone familiar with a suitable transformation of time and temperature values into a single score? I am working with microbiological data and I have three continuous factors all measured at ...
0
votes
0answers
44 views

Find intercept of almost flat lines

I have a set of lines (image below) which should meet in a number of points. As you can see, now the angular coefficient doesn't vary noticeably, making intercepts hard to find. What transformation do ...
2
votes
1answer
1k views

Optimal values of lambda in box-cox transformations do not lead to the lowest SSE?

I am using R to find optimal values of Lambda in Box-Cox transformations. you can find the data I am using here: https://uwyo-files.instructure.com/courses/449832/files/36678098/course%20files/...
0
votes
0answers
24 views

Stationary Time series is showing better results when predicting(ARIMA) after differencing

I have a time series of daily maximum temperature of a city for 2 years 3 months. I removed the seasonality from the data by subtracting present values with the past year values(seasonal differencing)....
2
votes
2answers
72 views

Interpreting logistic regression coefficient of a ratio predictor

I'm fitting a logistic regression model in which my predictor of interest is a ratio of measurements in millimeters. Possible values for this ratio range from 0 to ~2.0, with typical values around 0.9-...
0
votes
1answer
206 views

Log transformation of TS-stationary time series

guys. I have another question about main econometric time series transformation. I usually see the $log$ transformation of prices: $$p_{new}\left(t\right) = ln\left(\frac{p_t}{p_{t-1}}\right), t \in [...
0
votes
0answers
18 views

Coefficient of Variation on Log tranformed data, Canchola formula vs. raw formula

I am new to this community and I am currently confused. My intended use for coefficients of variation (CV) is to assess precision between repeated measurements of a clinical tool. We wish to assess ...
1
vote
1answer
239 views

Multiple transformations of data to Gaussian distribution

I am working with a fairly small dataset (20 < N < 40) and I would like to compute a Reference Interval. As per this field's guidelines, the preferred method ...
0
votes
2answers
271 views

Non-linear transformation to increase separability between clusters

I want to do a classification on PC scores. I have a $400$ dimensional matrix, e.g. $2000\times 400$ ($2000$ number of samples and $400$ dimensions). I first apply PCA on it and take it to 3D, i.e. $...
1
vote
1answer
26 views

Can we normalize both continuous and discrete numerical values

I have a sensor dataset with 16 features as numerical values (12 are continuous and 4 are discrete). I am using LSTM model to fit the data and do some classification. As both continuous and discrete ...
4
votes
2answers
803 views

What methods can be used to transform data?

I am solving a binary classification problem with 4 predictor variables. The variables didn't seem to be linearly separable. I have used Neural Networks and Kernel SVM which work and give desired ...
0
votes
0answers
17 views

transforming regression variable

I am would like to create a regression model with different variables however before using these variables in my regression model I would like to transform the variable in order to make it more ...
0
votes
0answers
31 views

Box-Cox formation with model selection, regularization, etc

As my data is not normally distributed, I performed the Box-Cox Transformation on the response. ...
1
vote
1answer
126 views

Transformation of data with zero and R squared

I have a conceptual concern about data tranformation and R^2. Often we transform data to respect the assumption of the linear model. Therefore, we can use multiple type of transformation such as log ...
1
vote
1answer
49 views

How to make a johnson unbounded transformation to make my data more gaussian like? in python

I am a novice in stats and I would like to transform my data (house prices) using a johnson unbounded distribution to look more gaussian. I looked at pandas transform() but I can't really understand ...
0
votes
2answers
3k views

Regressions with long-tail variables (GDP, etc)

It seems common to apply standard linear regression to variables with long-tail distributions, like GDP, by first taking the log. What is the justification for doing that? Is it effectively assuming a ...
2
votes
2answers
39 views

How to generate 90% of the data in Exponential distribution above some threshold value in R

I want to generate a dataset from the Exponential distribution, where 90% of the observations are above a threshold value 0.15. ...
0
votes
0answers
30 views

Is probit transformation the same as probability integral transform?

The image shows the original marginal data $u$ and $v$ on the left, which has a bounded support, and their probit transformations $r$ and $s$ on the right, which has an unbounded support. The ...
0
votes
0answers
36 views

Box Cox Transformation in auto.arima

I'm working with ARIMA models and was wondering about the necessity of BoxCox Transformation. When applying BoxCox on my training-set BoxCox.lambda(train) it ...
2
votes
1answer
54 views

Can this be written as a linear model?

$$y_i=\frac{(x_{0i}\beta_0+x_{1i})+\log(\beta_1^2 x_{2i})}{x_{3i}}+e_i\quad,\,i=1,\ldots,n,\,x_{pj}>0$$ I am wondering if this can be written as a linear model, I didn't think so because of the ...
3
votes
1answer
80 views

Up to what number of distinct values should I transform a categorical variable in a dummy variable?

When working with categorical variables, it's common to do some sort of transformation. Usually people apply a one-hot encoding. Putting it simply, we transform a categorical into a dummy variable. ...
0
votes
0answers
25 views

why to take logs and remove mean?

I am replicating a paper (Gil Kim and David Vera 2017). They transform the price of oil by doing the following; "for the real price of oil, we first deflate it, transform it into natural log and ...
0
votes
1answer
225 views

When to use Normalized Root-Mean-Squared Error vs Spearman Correlation?

I am doing some Machine Learning experiments with Azure and the graphs that it gives me are measured in Spearman Correlation vs Iteration Number (part of the machine learning) However I was just in ...
4
votes
3answers
2k views

Transforming data with positive, negative, and zero values

I have a multiple linear regression model with several dependent variables that have positive, negative, and zero values, and are not normally distributed. I can't do a natural log transformation ...
0
votes
0answers
13 views

Does Mean and Variance Stationary need to be applied to all kinds of forecast models in time series data?

I am in need of Clarification about the Mean & Variance Stationary for time series data.. I was reading this discussion about the importance of Stationary Why use differencing and Box-Cox in time ...
0
votes
1answer
88 views

How can I calculate the mean and variance of a linearly transformed random variable?

Say I have a random variable $x$, with mean $\mu_x=35$ and standard deviation $\sigma_x=10$. I want to linearly transform $x$ to $y$ according to the formula $y=a+bx$ so that $\mu_y=100$ and $\sigma_y=...
1
vote
2answers
59 views

Final Step: Prediction new values using model

I'm about to finish up a machine learning challenge, but I'm suck on the final part. Before running my model I did a simple power transformation ($dependent ~variable^{1/4}$) on the dependent ...
1
vote
0answers
15 views

Seasonal ARIMA lag differencing p-value not significant

I am using the below data to forecast using seasonal ARIMA model. I see at d=1 the p-value is not significant. But still ...
2
votes
1answer
63 views

Normality and homoscedasticity are lacking: Is transformation necessary?

I'm a student and I'm very new at this so I wanted to ask what to do. I have a data set and one of the groups didn't pass Shapiro-Wilk normality test (p value = 0.01) but testing with model residuals ...
1
vote
1answer
16 views

Constant-information scale transformation

I was recently introduced to the concept of constant information scale transformations in the book Generalized Linear Model with Examples in R, by Dunn and Smyth. With that, they mention in the book ...
0
votes
0answers
22 views

If in the ARIMA model the residuals is not normally distributed then what should be the way to make it normally distributed?

If in the ARIMA model the data is not normally distributed then what should be the way to make it normally distributed?

1
2
3 4 5
41