Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

Transformation of a fixed effect?

I'm working with a data set where I relate the response variable weight gain/loss (it goes from -110 g to 150 g) to multiple explanatory variables. It looks like this: lmer (weight.difference ~ A * B ...
1
vote
0answers
22 views

Converting data from exponential model to a linear model

This question came up recently and I am really struggling to understand how to even do this. I have just started learning statistics and would like is someone could show me how this could be done in R....
1
vote
1answer
47 views

Conversion of probabilities to rate [closed]

I need to convert this matrix of weekly probabilities to annual probabilities. However, when converting it into annual rates and transforming them into probabilities does not give reasonable values ​​(...
1
vote
0answers
57 views

Standardize first four moments: match sample moments with population moments

Let $X$ be a sample from $N(0,1)$ and $m$, $v$, $s$, $k$ denote sample mean, variance, skewness and kurtosis of $X$. I want to transform the sample $X$ such that the sample moments equal the true ...
0
votes
0answers
39 views

Applying standard normal CDF to normal random vector

Let $\mathbf{X} = (X_1, ..., X_k)^T \sim \mathcal{N}(0, \mathbf{\Sigma})$. Transform it as follows: $$\mathbf{Y} = (\Phi(X_1), ..., \Phi(X_k)),$$ where $\Phi$ is the standard normal CDF. I need to ...
0
votes
1answer
81 views

Yeo-Johnson does not increase normality

I have used Box-Cox Yeo-Johnson transformation to make my skewed data columns less skewed and more normal so that I can remove outliers. e.g. originally most of my columns have a 'skewness' of 400! ...
1
vote
1answer
31 views

How to apply a statistical test on either a bi-modal distribution OR how to transform it to parametric?

I have lifetimevalue (LTV) data for 3 groups in my set. For each group, their respective LTV looks bi-modal. I need to test if there is a statistical significance between those groups with respect to ...
0
votes
0answers
12 views

Regression using trig-computed terms for non-time series data sets

Follwing up from my recent post: How to construct this "prediction heatmap" assuming OLS (worked out example) , I want to build my intuition around model specification for the classic ...
0
votes
0answers
53 views

Box-Cox power transformation, adding or subtracting constants, and interval scales

I am trying to understand how the Box-Cox power transformation works. So I took one of my datasets and ran the powerTransform function of the "car" package (via R Commander) after having computed a ...
0
votes
0answers
29 views

How does polynomial trend get 'detrended' by the d-differencing?

Suppose $X_t$ is an ARIMA(p, d, q) process, then so is $X_t + m(t)$ where $$ m(t) =a_0 + a_1t +...+ a_{d-1}t^{d-1} $$ is some polynomial of degree $d-1$. How does such a polynomial trend get '...
3
votes
3answers
372 views

Why prices are usually not stationary, but returns are more likely to be stationary?

I read in a course material for time-series that Daily stock prices $X_t$ are in general not stationary but the daily returns defined by $Y_t := \frac{X_t - X_{t-1}}{X_{t-1}}$ may be stationary. ...
2
votes
2answers
168 views

Non linear data, need a transformation method to make the data linear

I have FX data for USD/SEK and I am trying to use the OLS to build a predictive model to predict the closing price. The closing price is the response variable. The USD/SEK opening price, low price, ...
1
vote
0answers
28 views

In what order should I perform Fisher's r-to-z transformation and correction for range restriction and measurement error on correlation coefficients?

I have a two part question; both parts relate to correcting/transforming raw correlation coefficients for the purpose of a meta-analysis. Confirming my understanding of ...
0
votes
0answers
12 views

Representing a time-series smoothed curve as a sinsoidal?

So attaches is an example of the kind of time series data I am working with. So far I have used Gaussian filters with sigma=3 and 6 to smooth the data, which has worked very well (especially sigma=3). ...
0
votes
0answers
12 views

Removing group effects from data

I have a set of data which consists of measurements from sensors for a machine. I know that depending on what the machine is doing at the time, that several different ranges of measurement data will ...
3
votes
2answers
189 views

Planar Flow in Normalizing Flows

While I've read "Variational Inference with Normalizing Flows" (abstract), I don't understand about an intuition of Planar Flow. The author defined Planar Flow as below Let $\boldsymbol{w} \in \...
0
votes
0answers
15 views

Estimating a logit directly with ordinary least squares

I've read that the coefficients on the logit regressions are equivalent to that of estimating the model: $log(p/(1-p))$ = $\beta$$_o$+$\beta$$_1$X+$\eta$ If i wanted to transform my data to run ...
0
votes
1answer
44 views

Variance stabilizing transformation for time series

I differenced the time series and got this plot. I think I'm supposed to use a variance stabilizing transform because variance is increasing over time but I'm a bit confused as to which one I'm ...
1
vote
0answers
35 views

Lapply does not work as expected [closed]

I want to use map or lapply to iterate over a list of dimension variables (dim_list below) using my custom function (transf_fun()). My overall goal is to generate a set of tables and later to pass ...
1
vote
0answers
10 views

can I compare orignal arima with sqrted arima model?

I use arima to fit my data. I use data and it's root sqrted respectively. Can I compare two model by comparing their AIC, loglikelihood etc.? Thanks.
0
votes
0answers
25 views

Is the KL-Divergence invariant to strictly monotonic transformations of the random variable?

Let $p$ and $q$ be two distributions on a variable $X$. Let $\widetilde{p}$ and $\widetilde{q}$ be the corresponding distributions on $f(X)$, where $f$ is a strictly monotonic function (e.g. $f(x)=e^x$...
0
votes
0answers
37 views

Is it reasonable to do log transformation on both input and output variables in multioutput regression problem?

I have been working on a small machine learning project, and I decided to use regression algorithms to solve the problem, however, I have encountered some problems in the project. Let me show some ...
0
votes
0answers
19 views

Transformation vs Scaling

I'm not sure I understand the different uses between the 2 methods: Scaling - scale features to same scale (Normalization or Standardization) Transformation - makes the data normally distributed ...
0
votes
0answers
6 views

what are the implications of standard normalizing non gaussian distributed data?

Many Bayesian regression neural network models and experiments in the corresponding papers (example: https://arxiv.org/abs/1705.07832) use standard normalized data $\frac{x - \mu}{\sigma}$, but the ...
0
votes
1answer
21 views

Transform linear, equally spaced points to a gaussian over the same interval?

Suppose I have linearly and equally spaced data points over an interval, e.g. 500 points over the interval 1.5 to 2.3, how would I transform these points so that they are normally distributed over the ...
1
vote
1answer
66 views

Any other way to calculate range of transformed variable?

I have been doing questions related to transformations of 2d random variables. In a question I have to find a range for $$u = (x-y)/2 $$ and $$v = y $$ where $x,y > 0$ So is it necessary that ...
0
votes
0answers
12 views

what distributions are limited with power transformation?

if I have a rough bell shaped distribution but with a heavy skewness, I could "correct" that by transforming that distribution similar to a normal distribution via power transformation Can power ...
1
vote
0answers
27 views

Transforming a multivariate normal sample using the sinh-arcsinh transform

Let us say that we have sample from the multivariable normal distribution. I would like to understand how is possible to apply a transformation to this sample to produce sample that has the sinh-...
0
votes
1answer
72 views

What does it mean when the Box-Cox pre-calculated lambda is negative?

I used guerrero to calculate a lambda from a sample of aus_retail dataset but it sometimes has given me a negative value. From what I understand, the lambda should be between 0 and 1. I am not sure ...
2
votes
0answers
33 views

Based on this graph, should I use a Box-Cox transformation?

Graph of difference data and residuals are below d data The graph displays monthly US oil production.
2
votes
1answer
159 views

Can AIC/BIC be used to compare models that differ by transformation of response variable? [duplicate]

I fit two models: one with a log-transformed response variable, and one without. All else is the same between them. Can I use AIC (or BIC or other, similar criteria) to choose which approach best fits ...
3
votes
1answer
48 views

How do I get all the betas from a specified production function? [closed]

$$Q_i = a L_i^{\beta_1} K_i^{\beta_2}\exp\left(\beta_3 L_i + \beta_4 K_i\right)$$ The above is the given production function and I'm supposed to find the coefficient values in Stata through ...
0
votes
0answers
22 views

Separating hardly distinguishable classes for easier classification

I apologize for this seemingly basic question, but I was just thrust into this data science role and would like to get some advice from some experts. I have a dataset with a small number of features ...
0
votes
0answers
27 views

Negative binomial modelling versus transformed linear modeling

My dataset (Data) contains data on distance traveled by an organism (Distance) based on their genotype (the two genotypes are A and B). For each genotype, there are 3 replicate genetic lines (A1,A2,A3,...
0
votes
2answers
60 views

What is the best way to combine data from two sessions for a variable for two independent groups before performing statistical tests between groups?

We have data on a physiological variable of interest (Metabolic Cost of Walking) from 2 groups of subjects (10 young adults and 10 old adults). We measured each one of them twice, once in the morning ...
1
vote
0answers
38 views

Anomaly detection using Clustering techniques on data with skewed features

I am working on finding anomalies in data. My dataset consists of 2 variables - Price of item and Volume of item purchased in a single checkout. I do not have information on the item other than its ...
0
votes
0answers
22 views

Building an analytical base table from a relational database

I need to train a (supervised) machine learning model. However, up to now I have only made model for data that was given to me as 1 single data frame. Now I need to create a model for data that is ...
0
votes
0answers
11 views

How do I transform and/or analyse data with left-skewed response variable that will not transform by most common methods?

I am aiming to analyse a response variable (numpick) which has a left-skewed distribution (very low frequency at low values, increasing frequency with response value, peaking at highest value). I was ...
1
vote
0answers
36 views

Data transformation : bimodal feature

I have a data feature that follows closely a bimodal distribution (mixture of two separate normal distributions with different mean, standard deviation and weights). Is it meaningful to transform that ...
0
votes
1answer
26 views

calculate a quarterly average, a monthly data [closed]

I'm new to R, maybe this is very simple but I don't know how to do it I have the following data ...
0
votes
0answers
25 views

When, during the model building process, should I split dataset into training ang test one?

There are many steps throughout the process of building a model, like data cleaning, feature extraction, etc. My question is, when exactly should I split data set into two separatable datasets? In the ...
0
votes
0answers
14 views

Are the orthogonal distances between data and the principal component line invariant to transformations?

Suppose a translation or rotation is applied to data and the approximating line, be this the regression line or the principal component line. The regression line minimises the vertical distances ...
0
votes
0answers
34 views

Making a series stationary, 1st and 12th order differencing

I know how to make a series stationary if it has a seasonal trend or a linear trend, I would do 1st order differencing. But the bit I don't understand is how make a series stationery if it has both a ...
0
votes
0answers
16 views

Power Transformations of a Single Variable for Multivariate Data Sets [duplicate]

Let's say I have a dataset with 5 variables and 100 observations. After checking for normality, all but one is normal, let's say the 2nd variable. After using a Box-Cox method I find a suitable lambda ...
3
votes
0answers
20 views

How to deal with data from different centers?

I want to examine the psychometric properties (e.g. internal consistency, factorial validity) of a scale. The data (N=700) comes from 16 different locations. Central means and variances did not differ ...
1
vote
0answers
38 views

How to convert log distribution to Normal Distribution

I have tried Box-Cox , exp, log etc but some features are not converting into normal distributions Please suggest me some alternative option .. As you can see in second and third graph it is not ...
1
vote
0answers
11 views

Scaling for mixed data and reverse transform

I have mixed type data (numerical, one hot encoded, ordinal). My data also has outliers. I am trying to scale them using a StandardScaler from sklearn, but I also need to revese transform the test ...
1
vote
1answer
52 views

PDF of cosine of a uniform random variable with additional shift

I need to calculate the PDF of a random variable, which is quite similar to what was asked here. However, I have to deal with a shifted cosine function. Thus, my random variable is defined as $$Y:=cos(...
1
vote
0answers
10 views

Mutual Funds Analysis

I am working on a dataset with aim to predict the MF ratings. There are cols like, 10 yr, 7 yr, 5 yr etc returns. I also have commencement date of MFs, the question is there are MFs with commencements ...
1
vote
0answers
39 views

When and why to (log) transform dependent or independent variables in machine learning models?

I know that linear regression (and any other machine learning model) doesn't assume normality in both independent and dependent variables, but assumes normality of the residuals (in case of linear ...

1 2
3
4 5
40