Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
4
votes
1answer
267 views

Interpretation difference between log link and log transformation

I have a question about the interpretation difference between log link of GLM and log transformation of LM. I know that log transformation is for target variable but log link is for mean .But related ...
1
vote
1answer
66 views

Using R auto.arima and arima.sim for stock prices

I simulate around 16000 stock prices with using auto.arima and arima.sim and I have two questions. Do I need to use plain ...
0
votes
0answers
13 views

Log transform all DVs for multiple regression?

Hello brainier folk than me, I'm looking to perform a multiple hierarchal regression and considering log transforming at least one IV which is the main one of interest to improve linearity. Should I ...
1
vote
0answers
18 views

Mixed model assumptions of linearity

I want to use a mixed model to analyse the association of distance covered per minute (TD_min) with ball possession (phase), playing position (role). Data have been recorded for the same subjects ...
0
votes
0answers
118 views

Regression in high dimensions ($p>n$) with nonnegative outcome

I need to perform regression in a setting where my outcome is positive ($Y > 0$) and there are more variables than rows ($p > n$). The goal of the analysis is to obtain predictions, which, of ...
0
votes
1answer
113 views

Ordered quantile normalization properties explained

My question is about the properties and proper use of "Ordered quantile normalization" from the paper titled “Ordered quantile normalization: a semiparametric transformation built for the ...
0
votes
1answer
36 views

modeling an outcome that has a range from [-0.158 ; 1 ]

I am trying to model a quality of life score that has a range [-0,158 ; 1], where values equal to one indicate the patient is fully healthy, values equal to 0 indicate the patient is dead, values ...
0
votes
0answers
66 views

Box-Cox, Exogenous Variables and Time Series Models

I am building time series models using SARIMAX from Statsmodels (Python). The independent variables in my models include 3 to 5 exogenous variables that are other than the target variable I am trying ...
0
votes
0answers
13 views

Creating ordinal variable based on percentages

I have a database with the results of a test applied to students. Each item had 4 to 5 answer options, only one of which was correct. I also have another dataset with the percentages of response to ...
2
votes
1answer
39 views

Data Leakage Concerns

I've come across the concept of data leakage in which optimistically biased generalisation errors occur due to test data in some sense 'seeing' the training data. For instance, normalisation on an ...
0
votes
0answers
64 views

Hellinger or log transformation for community data?

I have a dataset of abundances of phytoplankton species from many years and two locations and I want to study the communities and look for any differences. The abundances are, of course, very ...
0
votes
0answers
127 views

Negative binomial assumptions and errors

I'm conducting a negative binomial regression in SPSS using the GLM menu and I'm receiving the following error message: Warnings: The Hessian matrix is singular. Some convergence criteria are not ...
3
votes
1answer
148 views

Projecting a distribution onto the unit sphere in an arbitrary norm

So we represent the Dirichlet distribution as the projection of the $d$ independent gammas (on $R_+^d$ onto the unit simplex, and we arrive at that through the $L_1$ norm. That is, divide ${\bf x} \...
0
votes
1answer
28 views

Dealing with Skewed Normality for the Standardized Residuals to meet Normality Assumption (OLS)

I am required to build a OLS model. Currently, My model of log(response) against a number of predictors have fulfilled homogeneity assumption (constant variance) and low multicollinearity (based on ...
0
votes
0answers
13 views

Is there a clustering algorithm that can divide population for few classes and neglect some samples?

As far as I know all clustering algorithms assume that all delivered data points have to find its cluster. Is there an algorithm that could focus only on n clusters (number stated by user) and try to ...
0
votes
1answer
24 views

If data is square rooted to assume normality, should the data used to test with Levene's Test for Equality of Variance also be square rooted?

I am going to run a MANOVA, two of the assumptions are normally distributed data and equal variances. I had a data set that non-normally distributed. I square rooted it, and it is approximately ...
0
votes
0answers
59 views

Box-Cox back transformation with indicator in linear regression

I have a fitted linear regression of a Box-Cox transformed dependent variable, using an indicator variable as one of the two predictors : $$ g(Q, \lambda) = \hat{\beta_0} + \hat{\beta_1}P+ \hat{\...
2
votes
1answer
42 views

Why is a non-linear transformation a parameter?

This in reference to my answer at What model should I use to prove statistical significance?. I test the correlation between x and ...
0
votes
0answers
15 views

Calculating spread (standard deviation or CV) for ranges of values

Due to data suppression/confidentiality, I only have ranges of values, not individual data points. I have the sample sizes, the percentage of acres that receive ranges of fertilizer application rates, ...
0
votes
0answers
41 views

log transform a time series before fitting an ARIMA model?

That is my line graph using my data above in the snipboard link ^ Do I need to log-transform my data before doing ARIMA? I cannot see any variance increase in my opinion.
0
votes
0answers
7 views

How to quantify different sizes and colors of pie slices?

Suppose want to build a regression model that compares pies (say, "goodness scores"). One of the properties of these pies is that they are cut into differently sized slices across pies and ...
0
votes
0answers
10 views

Best Way To Encode Categorical Variable To Capture Impact of Each Unique Value For Tree Based Model When Data has Collinearity

I'm working on a project right now where I'm looking to use XGBoost to model a binary classification problem and use feature importances to look at the relative importances of group characteristics in ...
1
vote
0answers
94 views

Conducting a meta-analysis of proportions and one proportion is negative

I am conducting a meta-analysis of proportions and plan to first use the Freeman-Tukey double arcsine transformation before meta-analysing the data. However, one of the proportions is negative (-1.3%)....
3
votes
1answer
102 views

np.log() vs StandardScaler() in preprocessing of dataset variables

I'm anew to DS and now I'm passing this introductury Course on Kaggle. I'm trying to catch the logic behind this exercise, introduction. Particularly the part of data transformations is unclear: ...
2
votes
1answer
84 views

Do the statistical testing results of a transformed variable apply to the original variable?

Suppose a variable is transformed, and a statistical test is applied to the transformed variable. Do the results of the test (specifically, p-value) apply to the original variable? For example, ...
0
votes
0answers
17 views
0
votes
1answer
37 views

Is there a name for this generalisation of the exponential distribution

Is there a name for the following: $$ f(x) = \lambda(x) e^{\int_0^x -\lambda(t) dt} $$ which is similar to an exponential distribution. If $f(x)$ is a polynomial, would this be classed as a gamma ...
0
votes
0answers
46 views

Checking the distribution of input variables with missing data imputed in exploratory data analysis

Suppose there are missing data in input variables and the missing rates are relatively high, we use some certain value to impute the missing info. When we check the distribution for input variables in ...
0
votes
0answers
22 views

Can I transform percentage data using just the square root?

I have a data set with the number of larvae (out of 100) that have metamorphosed after 0,1,2, and 5 days. I want to perform a repetitive ANOVA on the results but they are not normally distributed. I'...
0
votes
0answers
12 views

Significant difference on boxcox transformed data where raw data means overlap

I boxcox transformed my data, created a linear model in r, then did anova(linear_model). I did this because my data was non-normal and was not compatible with any generalized linear model. My means (+/...
1
vote
1answer
38 views

Factor Analysis - multiple answers from same respondents

i am conducting an Exploratory Factor Analysis. I asked 100 subjects to rate the credibility of 4 marketing tactics. I examined 8 tactics in total and 4 tactics were allocated randomly to every ...
0
votes
0answers
475 views

How do I normalise severely right-skewed data?

I have a few continuous variables in my dataset that are severely right-skewed. I have tried several log transformations, incrementally increasing it to log(10^8). Nothing has worked. The log ...
0
votes
0answers
47 views

Interpreting multiple regression with transformed response variable

I have build regression model focused on association between physical activity and fat mass. The model is as follows: ...
0
votes
1answer
31 views

How to add both long-term and short-term interest rates as variables for a GARCH model?

I was facing some difficulties with a model of mine. I want to look up how the portfolio reacts to interest rate changes and I would like to use a GARCH model. However, both the short-term and long-...
0
votes
0answers
39 views

Box Cox transformation in multiple regression using car package in R [duplicate]

Could you please confirm that my aproache is right. In my regression model: ...
2
votes
2answers
188 views

Way to calculate the variance or SE of correlation from Fisher's z

Is there any way to get the sampling variance of correlation given Fisher's z and its variance? To make it more clear: suppose z = 0.5493 and ...
1
vote
0answers
157 views

A VECM in logs or a ARDL in the first difference of logs?

Suppose I have a number of time series that appear to have exponential growth at similar rates, with errors I believe to be generally proportional to the level of the variable. I believe that one of ...
0
votes
0answers
17 views

First difference in logs transformation produces biased results on back-transformation [duplicate]

I have a strongly trended series where the trend appears to be exponential and I believe the errors tend to be proportional to the current value. In order to convert it to a stationary series for ...
0
votes
1answer
42 views

Reversing Log-transformed target after training : r2 score interpretation

I have been running a log-transform on my target values because the distribution appears to be highly right skewed as you can see in the picture. After having called ...
0
votes
0answers
14 views

Interpretation of variables after Target Transformation

I create a linear regression where I used sklearn's Target Transformation to transform the target with a natural log. How would I interpret the coefficients? In the original unit or now as a percent ...
0
votes
0answers
18 views

Identify non-linear trend to know when to use log transformation in ARIMA?

There are basically three often used approaches to make time series stable based on three difference scenarios: 1) first difference for linear trend; 2) log for non-linear trend; 3) log seasonal ...
0
votes
0answers
17 views

Transformation for making variables not correlated

I once saw the following implementation that is claim to ensure the transferred variables are not correlated. I am not clear on the underlying mechanism for this. Any explanation is very appreciated. <...
1
vote
1answer
36 views

Why Do Distributional Forecasts Need to Produce Normally-Distributed Forecasts to be Ensembled/Combined?

I am forecasting a collection of different types of items, using many different forecasting techniques. Some of the techniques I use take the input data as is to produce a distributional forecast. ...
1
vote
0answers
27 views

Random variable with finite exponential first moment, infinite exponential variance [duplicate]

Could you provide an example of a random variable $X$ such that $\mathbb{E}(e^X)<\infty$ but $\text{Var}(e^X)=\infty$? Related: "Random variable with finite logarithmic first moment, infinite ...
7
votes
2answers
469 views

Is (covariance) stationarity preserved under log or exponential transformation?

In this lecture note, it (proposition 2) says that strict stationarity is preserved under transformation. However, it doesn't give the proof of this statement. Second, what if the process is ...
0
votes
0answers
218 views

What are the limits of inverse hyperbolic sine for regression?

I know that if your data contains zeros, log transforming your variable can be problematic, and all the zeros become missing. It is often suggested to use the inverse hyperbolic sine transform, rather ...
1
vote
0answers
26 views

Would a log transformation on my features change the elastic net result

We built the elastic net model on a set of my features and control features. With that, we did various experiments to discuss the importance of the selected features. For example, we showed more of my ...
1
vote
0answers
36 views

How to interpret a transformed linear regression model

I'm playing with the trafo R-package and this small data. After using the assumptions function, I found the log shift opt transformation is the best for normalizing ...
2
votes
0answers
21 views

Survey Data: What is tidy?

My question is similar to this one, but framed in the context of survey data. I'd like to format data from a survey in a tidy manor where some questions are yes/no answers and others are numerical ...
1
vote
1answer
20 views

Does this shape one cluster? and why angles change every time i run the code?

I have data and tried to do clustering on it. every time I run the code with the below statements it changes the angle of the shape but still the same below shape ...

1 2 3
4
5
44