# Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

1,936 questions
Filter by
Sorted by
Tagged with
10 views

### Transforming the independent variable for better prediction results with decision tree and random forest [duplicate]

I am trying to predict the amount a customer will pay. Removed zeros and standardized key columns. Data looks like this The distribution of a dependent variable is very skewed. I tried to apply ...
19 views

### Why is transforming data valid, and how do we know, in general, that any given transformation is valid to apply?

I'm aware that we often transform data to make it easier to analyze. For instance, we will sometimes transform data by applying the logarithm function. In the case of the logarithm function, I'm not ...
19 views

### GLM: effect of link function on choice of transformation of covariate

It struck me that if I have data of the form below, ...
25 views

### Correlation uses observed values, Mutual information uses probability values

I have a paired dataset whose observed values are: ...
20 views

### AIC and transformation of the independent variable

I set up different models: always same dependent variable and dataset, only the independent variable changes. All model assumptions are fullfilled. Now i do a model selection with the AIC. I look at ...
103 views

### Will non-linear data always become linear in high dimension?

I was reading the Hands on ML book and I'm on the SVM and Logistic Regression chapters. I started looking up more on these algorithms and apparently they are "linear" classifiers i.e the ...
8 views

### Interpretation of regression coefficients after power transform (possibily with polynomial transform and PCA) [duplicate]

In general I standardize my features before regression by subtracting the mean and dividing by unit variance: $$\hat{X} = \frac{X - \bar{X}}{Var(X)}$$ With this basic standardization, interpreting ...
14 views

### How to handle compositional explanatory features in redundancy analysis?

I have a matrix where rows are sites and columns are categories of land usage. The cell values add to one across each site row; that is, for each site I calculated land usage within a set radius. I ...
10 views

### How to simulate random variables that satisfy target values for information theory measures? [closed]

How can I simulate a univariate or multivariate dataset whose distribution or marginals meet pre-specified, target values for certain information theory measures? For example, I want to create a ...
17 views

### How to apply box muller transform to a time series?

I have a time series where each point is independent and uniformly distributed in $[0,1]$. I want to apply the box-muller transform to obtain a normally distributed time series. However I am unsure if ...
21 views

### How can I use transformation properties to obtain the distribution of $h(\mathbf{s})$?

Let that $\mathbf{s}=(s_1,s_2) \sim Unif(S)$, where $S$ is some spatial area. Suppose $y=h(\mathbf{s})=1-[exp(exp(\beta_0+\beta_1(\mathbf{s}-\mathbf{x})^T(\mathbf{s}-\mathbf{x})))]^{-1}$. We have that ...
19 views

### how to estimate the precipitation value from other relative data

I have climate data but the precipitation values are missing, I would like to know if there is any formula used to estimate the precipitation value from the other recorded climate data: temperature, ...
10 views

### Interpreting differences in log-transformed data when constant has been added first

I have performed an ANOVA on some log-transformed data. I had planned to transform the mean differences in the log-transformed data back to the original scale and interpret them as ratios of the ...
9 views

1k views

### Linear regression with “hour of the day”

I am trying to fit a linear model using "hour of the day" as parameter. What I'm struggling with, is, that I've found two possible solutions on how to handle this: Dummy encoding for every ...
38 views

### What are some “data transformation” methods? [closed]

There's a tag here called "data-transformation" described as Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a ...
6 views

### Can I average a Volume-Weighted Average for the same result?

I'm currently consuming metering data for a large utility company. I want to calculate the volume weighted average return temperature of cooling systems, but I have a lot of data points. To reduce the ...
32 views

### Is there a way to deal with spiking growth rates due to small sample numbers?

I'm looking at plotting county coronavirus case density growth rates (moving 7 day window) and am finding that when cases first appear the new case growth rate is very large due to the fact that there ...
34 views

### Question about delta method and variance-stabilization

The delta method or variance-stabilizing transformation can be applied to make the variance be "nearly constant" (https://en.wikipedia.org/wiki/Variance-stabilizing_transformation). They use ...
34 views

### Why skewness is not being corrected by BoxCox transformation

I have 2 variables which I want to put as predictor (independent) variables in logistic regression. However, both on them are highly skewed (one on left and other on right). Also, both variables are ...
14 views

### Whitening on projection matrices

The projection matrix $P = I -xx^T\in \mathbf{R}^{d \times d}$ has a zero eigenvalue and eigenvalues equal to one with multiplicity $d-1$. Is it possible to apply whitening transform on $P$ taking ...
48 views

### Interpreting Results: Using Regression Model for Prediction

Suppose I have a model like the following: $\hat{y} = 100 + 250 \log{x_{1}} + 75 x_{2} + 80 x_{3} + 105 \log{x_{4}}$ If an observation in my validation set had $x_{1} = 50, x_{2} = 40, x_{3} = 30,$ ...
17 views

### burstiness in time data

So i have time of arrival data like this: time of arrival = [1,1.5,1.6,1.7,1.8,3,4,6, 8, 9, 10] we can see that there is a burst between 1.5s and 1.8s. i have already tried to create a plot of ...
45 views

### The log of a predictor and polynomial regression

I’m working with primate brain data as a predictor in regression models. In the primate brain literature it is custom to log brain data, but it is unclear to me why. It has been argued that since one ...
14 views

### Trade-off between explaining variance and correcting overdispersion

I am fitting linear model. I happen to be working in R, and the specific model I'm fitting is a generalized additive model using the package mgcv, but I think all that is incidental to my question. ...
62 views

### Compare models with AIC when log transforming predictors but keeping the response variable exactly the same

Can I compare say two models where one have predictor P and the other have log(P) all else being equal? I have found posts on the topic but I can only decipher from those that you cannot transform the ...
15 views

### Multilevel model with subset of compositional data

I am working on an analysis where I am trying to predict BMI with a subset of microbiota data. Microbiota data is inherently compositional. I will be using a multilevel linear model. For the analysis, ...
10 views

### I have brain wave data for different subects aka voltages recorded every X sec. How do I deal with different wave frequency between subjects?

Brain wave data is really just voltages recorded every certain number of seconds; this becomes a wave with a certain frequency. I want to use the voltages recorded every, say, 10s in a model; this is ...
49 views

### Should I calculate the mean and standard deviation with raw or transformed data?

I'm an undergrad chemistry student, and in a recent laboratory session, we were given a set of observations for the volume of a solution in order to find an unknown concentration of a reactant $R$, ...
32 views

### Transformation of data for groups skewed in different directions

I am running some data in R that is intended for multiple regression. However my data is not normally distributed. Most of my independant variables are continuous, however one is a catagorical factor ...
66 views

### Why does the log transformation bring data closer to normal distribution? [duplicate]

Quite often in published research we see researchers apply log transformation to their data, and some claim that this makes the data closer to normal distribution. My questions are: Mathematically, ...
10 views

### Customer Retention Rate for a subscription business selling products with multiple contract durations

I have a question related to the standard metric used to evaluate subscription model businesses, aka user retention. We define user retention as the percentage of customers that keep their ...
9 views

### How to interpret a specific data transformation?

I came across this specific data transformation in the context of a physics application, which by itself is rather complex and hence out of the scope of this question. However since this ...
34 views

### log transform in linear regression

Assume we have a data set and the theory suggests to model $Y \sim X$. We apply a simple linear regression and get the following: Next, let us make a log transform of both $X$ and $Y$. The result is ...
20 views

### What is the back-transformation of Log((Max Y + 1) - Y)?

I have performed a linear mixed model regression with the transformation on the outcome variable, Y, Log((Max Y + 1) - Y). This gives normally distributed residuals when simpler transformations did ...
42 views

20 views

### Understanding the output of powertransform

Working in a multiple linear regression setting I am attempting to fit a general model to a data set that consists of 7 predictor. To do so I have run the powertransform() function and received the ...
If $Y = \ln(X)$ and $P(Y) = 0.55$ Can i say that $P(X) = e^{P(Y)} = e^{0.55}$ ? ($P(Y)$ is the probability of $Y$, and $P(X)$ is the probability of $X$)