Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

1
vote
1answer
19 views

What is the largest n root transformation I should consider for making a time series stationary?

Currently, I am working with multiple time series and not all of them are stationary. In order to make them stationary I am considering different transformations and checking the augmented dickey ...
0
votes
0answers
10 views

Data normalization

Hello, I want to eliminate the effect of a parameter over another parameter. Let's say, we are using a,b,c parameters to calculate x. and the trend of x is very similar to b. I want to nullify this ...
1
vote
1answer
45 views

BOX-COX TRANSFORMATION always stabilize variance

I am aware that box-cox transformation may make data set significantly normal distributed with constant mean and variance. But sometimes fails to convert data into normal. My question is even though ...
0
votes
3answers
45 views

Should one drop independent variables if they don't have linear relationship with the response variable?

I am building a linear regression model using Ridge regression. Some of the independent variables don't have linear relationships with the dependent variable. I've tried to do data transformations on ...
0
votes
0answers
15 views

Manipulate Data Series to Produce Equal but Opposite Betas [duplicate]

Suppose I have two different series, $A$ and $B$, and perform a single-variable regression of $A$ and $B$ upon a third series, $C$, to get two betas: $Xa$ and $Xb$. Given these initial betas, I want ...
0
votes
0answers
17 views

Eigenvectors - principle lines of force

In the article about PCA and coavariance matrix I've read the following: Finding the eigenvectors and eigenvalues of the covariance matrix is the equivalent of fitting those straight, principal-...
1
vote
1answer
20 views

Difference between differencing data and removing trend line for stationarity

In reference to making data stationary for Arima: Is there a difference between subtracting a best fit line from data, and a first order difference? Or, subtracting an exponential fit from data, ...
0
votes
1answer
24 views

Questions on making data stationary for ARIMA

When doing a time series analysis I have read these instructions : ...
0
votes
0answers
13 views

Data transformation on grouped data

I am running an Ax(BxCxS) mixed design. During data screening, I assessed for normality by splitting my data based on the two levels of Factor A. Some variables for one level are nonnormal (and normal ...
0
votes
0answers
30 views

Manipulating Series to Output Desired Betas [closed]

Suppose I have two different series, $A$ and $B$, and perform a single-variable regression of $A$ and $B$ upon a third series, $C$, to get two betas: $Xa$ and $Xb$. Given these initial betas, I want ...
8
votes
4answers
402 views

Is the absolute value of a stationary series also stationary?

I know that linear transformations of time series arising from (weakly) stationary processes are also stationary. Is this true, however, for a transformation of a series via taking the absolute value ...
0
votes
0answers
6 views

How can I convert rows to columns (with custom names) after grouping? [migrated]

I'm trying to get some row data as columns with pandas. My original dataframe is something like the following (with a lot more columns). Most data repeats for the same employee but some info changes, ...
1
vote
1answer
86 views

PDF of log transformed variable

I want to know if I've understood log transformation correctly in terms of functions of the distributions. If $\log(X)$ is normally distributed, then $X$ is lognormally distributed. Let's say I have ...
2
votes
1answer
38 views

Data visualization on original or transformed data?

I have a data set with two independent variables (species and life-stage) and one dependent variable (concentration of a protein). There are 5 replicates within each level (please see table below). I ...
1
vote
2answers
31 views

Why Permuted MNIST is good for evaluating continual learning model?

while I was reading papers about continual learning, I found that many researchers use permutated MNIST to evaluate their approach. I understand what it is but it ...
0
votes
0answers
8 views

What is the methodology behind Filter Based Feature Selection (i.e. Pearson correlation, etc.) on Azure Machine Learning Studio?

Filter Based Feature Selection on Azure Machine Learning Studio supports feature selection and ranking through Pearson Correlation, Kendall Correlation, Spearman Correlation, Mutual Information, Chi ...
0
votes
0answers
23 views

How to treat ordinal average rating as a regression predictor?

I have several variables that are averages of responses to a 5-star scale to rate attributes of a product. I have spent a little while reading about the potential issue of averaging a Likert scale and ...
0
votes
0answers
21 views

Non-normally distributed interest rates transformation

I am studying the effects of short- and long-term interest rates on bank risk-taking in the Euro zone countries. To analyse the effects, I will use, amongst other, an OLS regression. However I have ...
0
votes
0answers
5 views

Should we take antilog of the target value before fitting data into model?

I' am a noob in Data Science and was working on House Price Prediction data set on Kaggle in which the target value i.e 'SalePrice' is left skewed. To make the distribution normal, I used log ...
0
votes
0answers
12 views

Interpreting regression coefficients with cube root transformation of both dependent and independent variables

I have used the cube root $(1/3)$ transformation on both my dependent and one of my independent variables. I would have preferred a log-transformation but my data has both negative values and zeroes. ...
0
votes
0answers
38 views

Is it possible that PCA works better without data scaling? [duplicate]

I am a beginner. I have a dataset of 1700 samples with 4 features and I have to perform Hierarchical Clustering (the agglomerative version) and I need to decide whether or not to scale the data and ...
0
votes
0answers
16 views
1
vote
1answer
36 views

How can I interpret the coefficient of a logit transformed explanatory variable in linear regression?

I fit a linear regression model with continuous response. One of my predictor variables is given in percentage. So I transformed the predictor with logit transformation. My question is how can I ...
-1
votes
1answer
57 views

find distribution of the ratio of two normal r.v. with transformation U=X/Y V=Y

Statistical Inference by Casella and Berger (2nd ed) page 162 provides a way of finding the ratio of normal variables using U=X/Y and V=|Y|. I understand it. But what if I use V=Y? (I know it is ...
2
votes
3answers
51 views

Differencing and trend in time series forecasting

I understand that a time series is differenced to remove trend. But if trend can be modeled for forecasting purposes then why difference a time series at all?
1
vote
0answers
13 views

First or second difference or log for simulated real GDP data?

For a paper I need to use simulated real GDP data to regress this on average income mobility (how much more the next generation earns). As a hint the assignment indicates that the STATA code for a ...
0
votes
0answers
43 views

What is point of transformation if the effects does not change?

with a question related to transformation. I have done a linear mixed model using the formula ...
0
votes
0answers
30 views

Bias of a test statistic

I have a test statistic $T$ that has a Gamma distribution $\Gamma(k, \theta)$. The problem is when I do a Monte carlo simulation for large sample sizes, the parameters of the empirical distribution ...
9
votes
1answer
889 views

If the square of a time series is stationary, is the original time series stationary?

I found a solution that stated that if the square of a time series is stationary, so is the original time series, and vice-versa. However I don't seem able to prove it, anyone has an idea if this is ...
0
votes
1answer
26 views

Confidence interval before or after target variable transformation?

I created a machine learning model where I normalized the target variable in the training set before fitting the model. I then used a package which produced a standard deviation of eqch prediction. If ...
0
votes
1answer
40 views

Assumption multiple regression: normality of residuals

I want to run a multiple regression analysis for a given dataset in SPSS. However, the dataset violates the assumption of normality of residuals, as depicted in the picture. The values for the ...
1
vote
1answer
28 views

Is Hellinger transformation suitable for repeated (in time) site-species abundance data?

I have fish survey data from four different years and several locations, where I would like to study the difference in abundance and biomass between sites and check whether time dependence influences ...
1
vote
0answers
15 views

In Regression Analysis, should variable transformations occur before or after subset selection?

I'm looking at fitting a model that has many parameters. In order to simplify the model and prevent overfitting, I am planning to use the best subset selection for variable selection. My question is, ...
0
votes
1answer
25 views

Log transformation of TS-stationary time series

guys. I have another question about main econometric time series transformation. I usually see the $log$ transformation of prices: $$p_{new}\left(t\right) = ln\left(\frac{p_t}{p_{t-1}}\right), t \in [...
0
votes
0answers
23 views

Interpreting the likelihood estimate of transformed data

There is data $X(t)=[325200,500000,240000,130000,1200,10,10]$ where $t$ is time, $t=[0, 1 ,2 ,3,4,5,6]$ and I am fitting to these data assuming $X(t)$~Poisson$(X_0e^{\theta t})$ to estimate the ...
0
votes
0answers
12 views

Is there any method except Box-Cox transformations? [duplicate]

I always see that in order to reduce heteroscedasticity we can employ Box-cox transformations. But this totally useful if variance is a function of mean like $u_t^5$ or $u_t^2$....What should we do ...
0
votes
1answer
23 views

How to simulate from LMS parameters

I am trying to simulate a distribution of weights from the LMS parameters given in the CDC growth chart AGEWT given here Let's ...
4
votes
3answers
180 views

Proving transformations of two independent chi-squared random variables is equivalent to a Beta distribution

I came across the following in some old class notes of mine: if $\chi_{v_{1}}^{2}$ is independent of $\chi_{v_{2}}^{2}$ then $\frac{\chi_{v_{1}}^{2}}{\chi_{v_{1}}^{2}+\chi_{v_{2}}^{2}}\backsim ...
0
votes
0answers
43 views

Principal Component Analysis - Why Use Eigenvectors of the covariance matrix? [duplicate]

In PCA we start with a dataset and we reduce its dimensions by giving it new features that are each a linear combination of the original features of the dataset, and only keeping the ones with maximum ...
0
votes
0answers
44 views

How Differencing influences an ARIMA model

When finding a model for some data, I immediately look at the ACF and PACF to look to see if the data is stationary or if there is seasonality and so on. The moment I find out the data is not ...
1
vote
0answers
37 views

Do i need to log transform a variable if it displays a linear relationship?

I am writing a thesis on the relationship between gin consumption and health outcomes for my final year at university. After completing my research, I am wondering whether I should have logged some of ...
0
votes
0answers
25 views

Survival analysis: Individuals with event time = 0, exclude or not?

Data set: 50000 participants Assessments of various risk factors at baseline Dates when participants were included in the study Dates when participants died or were censored (after 2 years) ...
0
votes
0answers
41 views

Back transforming the intercept of double log normal distribution

I apologise if this is a duplicate, i couldn't find a thread that seem to be talking about the same thing. I have a dataset with a bunch of duration of varying lengths in seconds, i log-transformed ...
0
votes
1answer
41 views

Difference between Cross validation,GridSearchCV and does cross validation refer the train test split?

what is difference between Cross Validation and GridSearchCV and i know how train test split works but i heard many times cross validation, does it refer the same (train test split)? Can someone ...
0
votes
1answer
27 views

scaling in time series

I would like to know if its fine to use scaling(Time series * constant) before applying time series.Is it similar to data transformations( log , sqrt, box cox) etc or is there any implication on ...
4
votes
1answer
113 views

Interpret a multiple linear regression when Y is log transformed [duplicate]

I have the following multiple linear regression model: Log(y) = B0 + B1X1 + B2X2 + B3x3 + e. X1 is a dummy that can take 0 = male and 1 = female and X2 and X3 are continuous variables. I am not ...
1
vote
0answers
38 views

Can the first difference of a time series follow a random walk?

I have a time series in levels, say $X_t$ and a variance-ratio test suggests that it does not follow a random walk. Now I differenced the time series once and the variance-ratio test suggests that the ...
0
votes
0answers
40 views

Back-Transformation for Ln(X+1) of zero rich data

I have seen and read several similar questions, but mine pertains specifically to zero rich data. I will be back transforming my data based on a first order Taylor series approximation. As outlined ...
0
votes
0answers
10 views

What is the reasoning for using a log transformation for positively skewed data? [duplicate]

What is the (mathematical) reasoning for using a log transformation for positively skewed data?
0
votes
0answers
7 views

Two-way mixed ANOVA - do I need to transform pretest AND posttest

I am conducting a two-way mixed ANOVA on intervention data from a randomized controlled trial (i.e., my within-subject factor is time, and my between-subject factor is treatment condition). My ...