Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

1,681 questions
14 views

Trade Log of Market Prices [duplicate]

I have a situation wherein I need to trade in "Log of price" and I am looking for work around for that. The detailed situation is as follows: Step 1: Find correlation between Market Price and LOG OF ...
33 views

Position in Log of price

I need to trade in Log of price and I am wondering how do I take that. The situation is as follows: Step 1: Find correlation between Market Price and LOG OF Hedge price ( instead of hedge price for ...
20 views

Is it ok to use polynomial model to fit logarithmically transformed data?

Can we use polynomial regression using logarithmically transformed data. I transformed both y and x1 and x2 variables by natural log. After this, I used polynomial regression to fit. when using poly(...
29 views

Understanding variance stabilization and its uses

I recently came across the variance stabilization method that tries to remove the dependency of variance from the mean(for example consider poisson distribution). ...
9 views

Questions on variables regarding multiple linear regression. Percentages and ratios + standardisation [on hold]

I am new to statistics and regression analysis. For our R class, we are required to perform linear regression. Dependent variable: P/E ratios of firms Independent variables: revenue growth (in %), ...
14 views

When to use Normalized Root-Mean-Squared Error vs Spearman Correlation?

I am doing some Machine Learning experiments with Azure and the graphs that it gives me are measured in Spearman Correlation vs Iteration Number (part of the machine learning) However I was just in ...
4 views

How to normalize data by mapping data points from one mixture of multivariate normal distributions to another mixture

How to normalize data by mapping data points from one mixture of multivariate normal distributions to another mixture Problem description I am trying to normalize multivariate time series data. The ...
10 views

Multiple & multi-step time series forecast training data with RNN

I have read a lot of discussion on how to do cross-validation on time series data (e.g. walk forward) but I failed to understand how to properly prepare the training data for multiple time series ...
26 views

Adjusting normally distributed values for another variable [closed]

Suppose I have this data set: ...
17 views

What transformations can I use for time series with negative values? [on hold]

Currently, I am working with different time series. Some of the series that I have can have all negative values or positive and negative values. When I have a time series with all positive values I ...
13 views

How to (and if) transform and standardize two types of data (count proportions & low-value inflated latencies) for a MRIM model

I am looking to analyse a variety of traits in a Multivariate random intercept model (MRIM) with the help of the MCMCglmm package in R. All traits are measured on different scales so I wish to ...
14 views

What would be the mirror equation of a Brier curve?

The Brier equation gives a right skewed humped curve (i.e. long left tail and right skew to the curve),given by the equation y = q*X (X - Xnot) * sqrt(Xmax - X). I ...
22 views

What is the largest n root transformation I should consider for making a time series stationary?

Currently, I am working with multiple time series and not all of them are stationary. In order to make them stationary I am considering different transformations and checking the augmented dickey ...
10 views

Data normalization

Hello, I want to eliminate the effect of a parameter over another parameter. Let's say, we are using a,b,c parameters to calculate x. and the trend of x is very similar to b. I want to nullify this ...
47 views

BOX-COX TRANSFORMATION always stabilize variance

I am aware that box-cox transformation may make data set significantly normal distributed with constant mean and variance. But sometimes fails to convert data into normal. My question is even though ...
47 views

Should one drop independent variables if they don't have linear relationship with the response variable?

I am building a linear regression model using Ridge regression. Some of the independent variables don't have linear relationships with the dependent variable. I've tried to do data transformations on ...
15 views

Manipulate Data Series to Produce Equal but Opposite Betas [duplicate]

Suppose I have two different series, $A$ and $B$, and perform a single-variable regression of $A$ and $B$ upon a third series, $C$, to get two betas: $Xa$ and $Xb$. Given these initial betas, I want ...
17 views

Eigenvectors - principle lines of force

In the article about PCA and coavariance matrix I've read the following: Finding the eigenvectors and eigenvalues of the covariance matrix is the equivalent of fitting those straight, principal-...
22 views

Difference between differencing data and removing trend line for stationarity

In reference to making data stationary for Arima: Is there a difference between subtracting a best fit line from data, and a first order difference? Or, subtracting an exponential fit from data, ...
26 views

Questions on making data stationary for ARIMA

When doing a time series analysis I have read these instructions : ...
15 views

Data transformation on grouped data

I am running an Ax(BxCxS) mixed design. During data screening, I assessed for normality by splitting my data based on the two levels of Factor A. Some variables for one level are nonnormal (and normal ...
403 views

Is the absolute value of a stationary series also stationary?

I know that linear transformations of time series arising from (weakly) stationary processes are also stationary. Is this true, however, for a transformation of a series via taking the absolute value ...
6 views

How can I convert rows to columns (with custom names) after grouping? [migrated]

I'm trying to get some row data as columns with pandas. My original dataframe is something like the following (with a lot more columns). Most data repeats for the same employee but some info changes, ...
88 views

PDF of log transformed variable

I want to know if I've understood log transformation correctly in terms of functions of the distributions. If $\log(X)$ is normally distributed, then $X$ is lognormally distributed. Let's say I have ...
38 views

Data visualization on original or transformed data?

I have a data set with two independent variables (species and life-stage) and one dependent variable (concentration of a protein). There are 5 replicates within each level (please see table below). I ...
37 views

Why Permuted MNIST is good for evaluating continual learning model?

while I was reading papers about continual learning, I found that many researchers use permutated MNIST to evaluate their approach. I understand what it is but it ...
8 views

What is the methodology behind Filter Based Feature Selection (i.e. Pearson correlation, etc.) on Azure Machine Learning Studio?

Filter Based Feature Selection on Azure Machine Learning Studio supports feature selection and ranking through Pearson Correlation, Kendall Correlation, Spearman Correlation, Mutual Information, Chi ...
23 views

How to treat ordinal average rating as a regression predictor?

I have several variables that are averages of responses to a 5-star scale to rate attributes of a product. I have spent a little while reading about the potential issue of averaging a Likert scale and ...
21 views

Non-normally distributed interest rates transformation

I am studying the effects of short- and long-term interest rates on bank risk-taking in the Euro zone countries. To analyse the effects, I will use, amongst other, an OLS regression. However I have ...
5 views

Should we take antilog of the target value before fitting data into model?

I' am a noob in Data Science and was working on House Price Prediction data set on Kaggle in which the target value i.e 'SalePrice' is left skewed. To make the distribution normal, I used log ...
13 views

Interpreting regression coefficients with cube root transformation of both dependent and independent variables

I have used the cube root $(1/3)$ transformation on both my dependent and one of my independent variables. I would have preferred a log-transformation but my data has both negative values and zeroes. ...
39 views

Is it possible that PCA works better without data scaling? [duplicate]

I am a beginner. I have a dataset of 1700 samples with 4 features and I have to perform Hierarchical Clustering (the agglomerative version) and I need to decide whether or not to scale the data and ...
16 views

How to transform a non-stationary time series with short spikes to stationary?

I used ADF test to test stationarity: ...
37 views

How can I interpret the coefficient of a logit transformed explanatory variable in linear regression?

I fit a linear regression model with continuous response. One of my predictor variables is given in percentage. So I transformed the predictor with logit transformation. My question is how can I ...
58 views

find distribution of the ratio of two normal r.v. with transformation U=X/Y V=Y

Statistical Inference by Casella and Berger (2nd ed) page 162 provides a way of finding the ratio of normal variables using U=X/Y and V=|Y|. I understand it. But what if I use V=Y? (I know it is ...
54 views

Differencing and trend in time series forecasting

I understand that a time series is differenced to remove trend. But if trend can be modeled for forecasting purposes then why difference a time series at all?
14 views

First or second difference or log for simulated real GDP data?

For a paper I need to use simulated real GDP data to regress this on average income mobility (how much more the next generation earns). As a hint the assignment indicates that the STATA code for a ...
43 views

What is point of transformation if the effects does not change?

with a question related to transformation. I have done a linear mixed model using the formula ...
30 views

Bias of a test statistic

I have a test statistic $T$ that has a Gamma distribution $\Gamma(k, \theta)$. The problem is when I do a Monte carlo simulation for large sample sizes, the parameters of the empirical distribution ...
890 views

If the square of a time series is stationary, is the original time series stationary?

I found a solution that stated that if the square of a time series is stationary, so is the original time series, and vice-versa. However I don't seem able to prove it, anyone has an idea if this is ...
26 views

Confidence interval before or after target variable transformation?

I created a machine learning model where I normalized the target variable in the training set before fitting the model. I then used a package which produced a standard deviation of eqch prediction. If ...
40 views

Assumption multiple regression: normality of residuals

I want to run a multiple regression analysis for a given dataset in SPSS. However, the dataset violates the assumption of normality of residuals, as depicted in the picture. The values for the ...
30 views

Is Hellinger transformation suitable for repeated (in time) site-species abundance data?

I have fish survey data from four different years and several locations, where I would like to study the difference in abundance and biomass between sites and check whether time dependence influences ...
15 views

In Regression Analysis, should variable transformations occur before or after subset selection?

I'm looking at fitting a model that has many parameters. In order to simplify the model and prevent overfitting, I am planning to use the best subset selection for variable selection. My question is, ...
26 views

Log transformation of TS-stationary time series

guys. I have another question about main econometric time series transformation. I usually see the $log$ transformation of prices: p_{new}\left(t\right) = ln\left(\frac{p_t}{p_{t-1}}\right), t \in [...
23 views

Interpreting the likelihood estimate of transformed data

There is data $X(t)=[325200,500000,240000,130000,1200,10,10]$ where $t$ is time, $t=[0, 1 ,2 ,3,4,5,6]$ and I am fitting to these data assuming $X(t)$~Poisson$(X_0e^{\theta t})$ to estimate the ...
12 views

Is there any method except Box-Cox transformations? [duplicate]

I always see that in order to reduce heteroscedasticity we can employ Box-cox transformations. But this totally useful if variance is a function of mean like $u_t^5$ or $u_t^2$....What should we do ...
24 views

How to simulate from LMS parameters

I am trying to simulate a distribution of weights from the LMS parameters given in the CDC growth chart AGEWT given here Let's ...
I came across the following in some old class notes of mine: if $\chi_{v_{1}}^{2}$ is independent of $\chi_{v_{2}}^{2}$ then \$\frac{\chi_{v_{1}}^{2}}{\chi_{v_{1}}^{2}+\chi_{v_{2}}^{2}}\backsim ...