Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

How does using log transformation for a confidence interval of the survival stabilizes variance?

I only found that it eliminates the estimator of survival from the variance formula, but could anyone show some references or write a few formulas to show how exactly the Greenwood becomes more stable ...
0 votes
0 answers
29 views

What exactly does the Box-Cox transformation do to a time series?

If I were to try and rephrase the argument in the original Box-Cox paper in my own words, I would say something like the following: given a model $$ y = x \beta , $$ if the residuals do not appear to ...
  • 314
1 vote
1 answer
25 views

Derivation of Box-Cox and Yeo-Johnson Log-Likelihood Functions

The scipy documention lists expressions for the Log-likelihood functions for the Box-Cox and Yeo-Johnson transformations here and here. I'm looking for a source ...
  • 314
0 votes
0 answers
13 views

STATA: convert from wide to long format [closed]

I tried to convert according to the letter "y", thats why i added the letter to the variable names. The structure of the data is pictured below. Im trying to group the "kommune" ...
1 vote
0 answers
28 views
+50

How to learn steep functions using neural network?

I am trying to use a neural network to learn the below function. In total, I have 25 features and 19 outputs. The above image shows the distribution of two features with respect to one of the outputs....
  • 165
0 votes
0 answers
7 views

Which is the condition for using log transform for data before taking cointegration test?

I saw some discussions on whether to use log transformation on data before taking the cointegration test, many people recommend taking log transformation before the test. I want to ask if there are ...
0 votes
0 answers
5 views

Log input data multi target regression [duplicate]

I'm building a multi target regression model with random forest regressor, in python. My database has 20 variables, from which 13 are x values and 7 are y values (targets). I've treated 3 of these 13 ...
0 votes
1 answer
24 views

Reverting differencing for Time Series VAR model component wise?

I have data with seasonality so I used seasonal and then first order differencing to make the series stationary in order to fit VAR model. After the model is fitted on the differenced level I can ...
  • 3
2 votes
0 answers
35 views

Derive the pdf/cdf of a variable given as a formula of two random variables [closed]

Let's assume I have two random variables X and Y with x>0 and y>0, respectively. Let's also assume that their marginals are known as well as the joint cdf is known as the product of the two ...
  • 21
1 vote
0 answers
13 views

Is there a difference between kernel PCA with a non-linear kernel vs PCA with a non-linear change of variables?

I see that kernel PCA with a linear kernel is the same as PCA. On Wikipedia's introduction of the kernel to PCA they suggest that there exists a non-trivial arbitrary choice of map $\Phi$ that is ...
  • 4,560
0 votes
2 answers
53 views

modelling proportion data - transforming 0 and 1s to fit beta regression

I have proportion data which includes 0s and 1s (the ri column below, calculated from the ud/days_lib)). I have found a few threads related to modelling this issue. A few of which (e.g., this thread) ...
0 votes
0 answers
11 views

Instrumental Variable Transformation Question on Fixed Effects model

My question is regarding something my professor did, but I didn't understand the purpose of it. I have panel data for 1052 municipalities from the year 2003 to 2019. I have a variable that tells me ...
  • 1
1 vote
1 answer
64 views

Does log-transforming a slightly-skewed y variable make any sense?

I'm separately analyzing 5 subsets of my data and running multiple linear regressions with the same outcome variable in each. (I am also running a single regression with all the subsets and using the ...
2 votes
1 answer
25 views

Jacobian of function returning $m$ evenly-spaced order statistics of an $n$-dimensional vector

Let $y\in\mathbb{R}^n$, and let $f:\mathbb{R}^n\to\mathbb{R}^m$ be the transformation that outputs $m$ evenly-spaced order statistics (including the extremes) of $y$. What is the Jacobian of this ...
  • 1,560
0 votes
0 answers
7 views

Johnston transformation of censored data for tobit regression, to estimate the proportion pitted beyond the specification depth

I have a sample of 35 components from a population of machines. The depth of pitting on all these components has been measured. The resulting dataset looks to be left censored, a large number of no ...
  • 11
0 votes
0 answers
19 views

Multiple comparisons of Box-Cox transformed data

I'm working on a dataset of highly-skewed data that I have transformed using Box-Cox. I have 2 groups (healthy controls and diseased participants) and I need to perform multiple comparisons (to ...
  • 1
2 votes
2 answers
98 views

Interpret bell curve, calculate 95th percentile value

I have data showing the distribution of a given metric across a population. The data is broken up into increments, with values for the 90th 97.5th and 99th percentiles. How can I interpret this data ...
  • 23
1 vote
0 answers
23 views

Comparison of different IRFs from the VAR model (meta-analysis)

In my meta-analytic research, I have collected IRFs from papers where two variables x and y are both in log-levels (these variables were entered into some VAR model in this way). In addition, I also ...
  • 11
2 votes
3 answers
491 views

What's the right way to rescaling (min-max normalization)?

Say I have a scale 2-12 and want to transform it into the range between 0-1. I would use the following formula: $$ y={\frac {x-{\text{min}}(x)}{{\text{max}}(x)-{\text{min}}(x)}}$$ Sometimes I just saw ...
  • 31
0 votes
0 answers
14 views

What is the right order in dealing with outliers, missing values and log transformation?

I am currently working on a project involving banking stock price data. I have around 3000 observations, some columns have a lot of missing values (null value); they can account for 5 to 50% of the ...
5 votes
1 answer
313 views

Log of a log-transformed variable

I have been suggested to use the log of a log-transformed independent variable (i.e., log(log healthcare expenditure)). I am not sure how would this make sense. Is this a standard practice (in the ...
  • 61
0 votes
0 answers
43 views

How do you reverse log, square root, and Ln data transformations?

If you have transformed your data, is it true that when reporting results, such as descriptive statistics (e.g. mean, median, range, variance, standard deviation etc.), you need to revert the data ...
  • 1
0 votes
0 answers
22 views

Calculate $c$ so that $E[c|X_1 - X_2|]$ = 1

Given $X_i \sim N(1,1)$ for $i = 1,2$. Determine $c$ so that $E[c|X_1 - X_2|] = 1$ I am a bit confused whether $c$ should be $\sqrt\pi$ or $\frac{\sqrt\pi}{4} $. If we take $Y = X_1 -X_2$ then $Y \sim ...
0 votes
0 answers
19 views

What's a good rule of thumb for choosing a sufficient number of quantiles in quantile transformation?

I'm a currently developing a regression model based on the Choquet Integral. To tackle outliers I am using the Quantile Transformer, provided by scikit-learn. I was wondering how the quantile number ...
0 votes
0 answers
26 views

Transformation of discrete data to reduce Skew

In a dataset I have running through on kaggle's tabular playground, some of the measurement columns that are missing values follow a gaussian distribution. I want to fix these missing values by ...
0 votes
0 answers
17 views

Getting normal distribution from spike distribution through data transformation

I'm trying to preprocess my data before feeding it to neural network. The goal is to get a normal distribution. I have 3 different features distributed as follows: One can see bimodal, skewed and ...
0 votes
0 answers
77 views

What can OLS with a Box-Cox transformed dependent variable tell me?

Just to ellaborate: I’m doing an OLS-test to determine the following things: Do my independent variables have a significant effect on the dependent variable? What’s the direction of the effect of my ...
  • 1
2 votes
1 answer
23 views

Using several types of feature transformations on same dataset (i.e., log, square)

I have a question regarding feature transformation for when some your features are skewed. I've seen online that depending on the skew (right or left) and how great the skew is, it is preferable to ...
1 vote
0 answers
39 views

Train-test split within modeling function

I have this function I wrote: ...
0 votes
0 answers
25 views

Transform to Normal...or is there a better distribution for this case?

I'm not sure whether the right approach here is to transform to Normal or if there's another distribution that perfectly fits the data. What I'm trying to analyze is payroll data each pay period for a ...
1 vote
1 answer
29 views

How to generate variables with common latent factor?

I'm dealing with Kalnins (2018) work about multicollinearity. I can't comprehend one thing - how can I get variables that share a common latent factor? (not only correlate with each other). I can ...
1 vote
0 answers
124 views

GLMM with scaled variable: do I need to back-transform?

I am fitting a GLMM to powerline collision data for a bird species- using distance to seasonal water, habitat and the presence/absence of line markers as predictors. Incident is a binary response (50 ...
1 vote
0 answers
22 views

Finding a fit for left-skewed continuous data

I am looking at various plant community traits and how they relate to a number of environmental variables in a woodland. I am using mixed models as some samples come from the same woodlands so I'm ...
0 votes
1 answer
20 views

Transforming input features for ANN

I usually normalise my input variables for linear models (i.e. apply a log transformation, or rank-based inverse normal transformation). One of the reasons I like using tree based methods like Random ...
  • 115
2 votes
0 answers
21 views

How to apply the Jacobian correction to AIC for a transformed dependent variable when the transformation includes an independent variable?

I am comparing several OLS multivariate regression models of a dependent variable (we'll call it $Y$) using various transformations, some of which also involve one of the independent variables ($X_1$)....
3 votes
2 answers
88 views

Establish relationship between two variables with scatter plot like a trapezoid

What transformation can be used to establish relationship between two variables whose scatter plots looks like the plot below? I have applied log Y, log X, exp Y, exp . But a relationship could not be ...
  • 219
0 votes
0 answers
9 views

Box-cox transformation question about the formula [duplicate]

Recently I am learning box-cox transformation. I know the transformation is to make data normal. And the formula for box-cox is: However, I am confused why when people get lambda, they will do the ...
0 votes
0 answers
8 views

Neural Network Feeding - Custom Information Extraction

I am trying to train a neural net. to extract specific information from text. I need to find entity information like attribute, dependency, etc. The text input will be like this: ...
3 votes
1 answer
183 views

Are coordinates (e.g. Point(123, -123)) considered interval data?

I understand that latitude and longitude are interval data, but are coordinates (e.g. Point(123, -123)) also considered interval data? If so, how can the standard deviation, mean etc. be calculated?
  • 31
0 votes
0 answers
24 views

negative values after log transformation

I need to find input demand for several food crops. Here I used profit function approach to find price elasticities. It is a time series data analysis. However, I have to normalize and find real ...
  • 1
1 vote
0 answers
11 views

How do I create a dataset for vacation home bookings? [closed]

My family is renting out a vacation home and I would like to create a dataset in R to calculate statistics on the bookings (for some helpful insights and also for me to practice R). I have a question ...
  • 11
2 votes
1 answer
42 views

Is true that the sampling distribution of $\ln \left(\chi^{2}\right)$ converges to normality much faster than the sampling distribution of $\chi^{2}$?

If true is the consequence true that $X \sim \chi^{2}(k)$ then $\sqrt{2 X}$ is approximately normally distributed with mean $\sqrt{2 k-1}$ and unit variance? Also true that If $X \sim \chi^{2}(k)$ ...
2 votes
2 answers
110 views

Fixing outliers and normalizing a vector using R

I am trying to do factor analysis on a few variables and one particular variable (given in the example below) is covering/ explaining all the variance due to some outliers. I am not sure what else I ...
  • 155
1 vote
0 answers
30 views

Decomposition analysis for data between zero and one

I want to analyze latent components of data that has values between zero and one (including zero and one). In detail, the data structure is n x m and I'm looking to find the r underlying components. ...
  • 31
0 votes
1 answer
26 views

Transformation of periodic data prior to PCA?

Basically I have periodic data (angles from -180 to 180) that I want perform a PCA on. However, since the data is periodic, a change in angle from say 170 to 10 will not be accurately reflected. I was ...
0 votes
0 answers
8 views

Interpreting regression output estimates with normal and square root transformed predictors and some log transformed response variables

I understand this topic is well covered here, but having read several threads, I can't find an explicit or clear answer to my question. I looked at the following threads and can't glean an answer from ...
0 votes
1 answer
31 views

What transformation can I apply to a random vector to make its cumulative sum strictly negative (or positive)?

$X$ is a random vector of real numbers. What is a good $f(X)$ such that $Y = f(X)$ satisfies $\displaystyle\sum_{i <k} Y_i < 0$ for all values of $Y$, where $Y$ has $k$ elements indexed with $i$?...
  • 12.4k
3 votes
2 answers
61 views

Standardize dataset with high outliers

Is there a better way to standardize a dataset with outliers than to normalized value (z-score) based on the mean and standard deviation? I am using the Excel STANDARDIZE function. I have two datasets ...
0 votes
1 answer
22 views

Establish a relationship between calls and sales where calls data is count independent variable

I am trying to model sales of 2020 on calls of 2020 where the values of calls are (0,1,1.5,2,2.5,3,3.5......,19,19.5,20). My lookout is to establish some sort of relationship between the two variables....
0 votes
1 answer
34 views

Incorporating seasonality in regression #gTrans #rms

I have been looking into incorporating seasonal effects using the gTrans function in the rms R package. However, I am having some difficulty interpreting the statistical explanation in the document. ...

1
2 3 4 5
47