Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Transforming the independent variable for better prediction results with decision tree and random forest [duplicate]

I am trying to predict the amount a customer will pay. Removed zeros and standardized key columns. Data looks like this The distribution of a dependent variable is very skewed. I tried to apply ...
0
votes
0answers
19 views

Why is transforming data valid, and how do we know, in general, that any given transformation is valid to apply?

I'm aware that we often transform data to make it easier to analyze. For instance, we will sometimes transform data by applying the logarithm function. In the case of the logarithm function, I'm not ...
1
vote
1answer
19 views

GLM: effect of link function on choice of transformation of covariate

It struck me that if I have data of the form below, ...
1
vote
1answer
25 views

Correlation uses observed values, Mutual information uses probability values

I have a paired dataset whose observed values are: ...
0
votes
0answers
20 views

AIC and transformation of the independent variable

I set up different models: always same dependent variable and dataset, only the independent variable changes. All model assumptions are fullfilled. Now i do a model selection with the AIC. I look at ...
1
vote
2answers
103 views

Will non-linear data always become linear in high dimension?

I was reading the Hands on ML book and I'm on the SVM and Logistic Regression chapters. I started looking up more on these algorithms and apparently they are "linear" classifiers i.e the ...
0
votes
0answers
8 views

Interpretation of regression coefficients after power transform (possibily with polynomial transform and PCA) [duplicate]

In general I standardize my features before regression by subtracting the mean and dividing by unit variance: $$ \hat{X} = \frac{X - \bar{X}}{Var(X)}$$ With this basic standardization, interpreting ...
0
votes
0answers
14 views

How to handle compositional explanatory features in redundancy analysis?

I have a matrix where rows are sites and columns are categories of land usage. The cell values add to one across each site row; that is, for each site I calculated land usage within a set radius. I ...
0
votes
0answers
10 views

How to simulate random variables that satisfy target values for information theory measures? [closed]

How can I simulate a univariate or multivariate dataset whose distribution or marginals meet pre-specified, target values for certain information theory measures? For example, I want to create a ...
1
vote
0answers
17 views

How to apply box muller transform to a time series?

I have a time series where each point is independent and uniformly distributed in $[0,1]$. I want to apply the box-muller transform to obtain a normally distributed time series. However I am unsure if ...
0
votes
0answers
21 views

How can I use transformation properties to obtain the distribution of $h(\mathbf{s})$?

Let that $\mathbf{s}=(s_1,s_2) \sim Unif(S)$, where $S$ is some spatial area. Suppose $y=h(\mathbf{s})=1-[exp(exp(\beta_0+\beta_1(\mathbf{s}-\mathbf{x})^T(\mathbf{s}-\mathbf{x})))]^{-1}$. We have that ...
0
votes
0answers
19 views

how to estimate the precipitation value from other relative data

I have climate data but the precipitation values are missing, I would like to know if there is any formula used to estimate the precipitation value from the other recorded climate data: temperature, ...
0
votes
0answers
10 views

Interpreting differences in log-transformed data when constant has been added first

I have performed an ANOVA on some log-transformed data. I had planned to transform the mean differences in the log-transformed data back to the original scale and interpret them as ratios of the ...
0
votes
0answers
9 views

How can I obtain a full-conditional distribution resulting from a transformation that has a dimension reduction?

Problem statement Suppose we have a function $h(\mathbf{s})=1-[\exp(e^{\beta_0+\beta_1(||\mathbf{s}-\mathbf{x}||)^2})]^{-1}=y$, where both s and x are $1\times2$ vectors, $y$ is a scalar, $\mathbf{s}=(...
2
votes
0answers
16 views

Normalizing zero inflated predictors for multiple regression

Hope I got it right, as this is my first active post :-) I was trying to find a solution the whole day for my problem. I am trying to predict a continuos variable based on 20 different predictors. The ...
1
vote
1answer
11 views

Probability density function after transformation

Let $X,Z$ be random variables with probability density functions $p_X,p_Z$. Suppose $Z=f(X)$, where $f$ is continuous and differentiable. How is $p_Z$ related to $p_X$? It's tempting to say $p_Z(z) ...
13
votes
3answers
1k views

Linear regression with “hour of the day”

I am trying to fit a linear model using "hour of the day" as parameter. What I'm struggling with, is, that I've found two possible solutions on how to handle this: Dummy encoding for every ...
1
vote
1answer
38 views

What are some “data transformation” methods? [closed]

There's a tag here called "data-transformation" described as Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a ...
0
votes
0answers
6 views

Can I average a Volume-Weighted Average for the same result?

I'm currently consuming metering data for a large utility company. I want to calculate the volume weighted average return temperature of cooling systems, but I have a lot of data points. To reduce the ...
0
votes
0answers
32 views

Is there a way to deal with spiking growth rates due to small sample numbers?

I'm looking at plotting county coronavirus case density growth rates (moving 7 day window) and am finding that when cases first appear the new case growth rate is very large due to the fact that there ...
0
votes
0answers
34 views

Question about delta method and variance-stabilization

The delta method or variance-stabilizing transformation can be applied to make the variance be "nearly constant" (https://en.wikipedia.org/wiki/Variance-stabilizing_transformation). They use ...
2
votes
1answer
34 views

Why skewness is not being corrected by BoxCox transformation

I have 2 variables which I want to put as predictor (independent) variables in logistic regression. However, both on them are highly skewed (one on left and other on right). Also, both variables are ...
0
votes
0answers
14 views

Whitening on projection matrices

The projection matrix $P = I -xx^T\in \mathbf{R}^{d \times d}$ has a zero eigenvalue and eigenvalues equal to one with multiplicity $d-1$. Is it possible to apply whitening transform on $P$ taking ...
2
votes
2answers
48 views

Interpreting Results: Using Regression Model for Prediction

Suppose I have a model like the following: $\hat{y} = 100 + 250 \log{x_{1}} + 75 x_{2} + 80 x_{3} + 105 \log{x_{4}}$ If an observation in my validation set had $x_{1} = 50, x_{2} = 40, x_{3} = 30,$ ...
0
votes
0answers
17 views

burstiness in time data

So i have time of arrival data like this: time of arrival = [1,1.5,1.6,1.7,1.8,3,4,6, 8, 9, 10] we can see that there is a burst between 1.5s and 1.8s. i have already tried to create a plot of ...
0
votes
2answers
45 views

The log of a predictor and polynomial regression

I’m working with primate brain data as a predictor in regression models. In the primate brain literature it is custom to log brain data, but it is unclear to me why. It has been argued that since one ...
1
vote
0answers
14 views

Trade-off between explaining variance and correcting overdispersion

I am fitting linear model. I happen to be working in R, and the specific model I'm fitting is a generalized additive model using the package mgcv, but I think all that is incidental to my question. ...
0
votes
1answer
62 views

Compare models with AIC when log transforming predictors but keeping the response variable exactly the same

Can I compare say two models where one have predictor P and the other have log(P) all else being equal? I have found posts on the topic but I can only decipher from those that you cannot transform the ...
0
votes
0answers
15 views

Multilevel model with subset of compositional data

I am working on an analysis where I am trying to predict BMI with a subset of microbiota data. Microbiota data is inherently compositional. I will be using a multilevel linear model. For the analysis, ...
0
votes
0answers
10 views

I have brain wave data for different subects aka voltages recorded every X sec. How do I deal with different wave frequency between subjects?

Brain wave data is really just voltages recorded every certain number of seconds; this becomes a wave with a certain frequency. I want to use the voltages recorded every, say, 10s in a model; this is ...
4
votes
2answers
49 views

Should I calculate the mean and standard deviation with raw or transformed data?

I'm an undergrad chemistry student, and in a recent laboratory session, we were given a set of observations for the volume of a solution in order to find an unknown concentration of a reactant $R$, ...
0
votes
1answer
32 views

Transformation of data for groups skewed in different directions

I am running some data in R that is intended for multiple regression. However my data is not normally distributed. Most of my independant variables are continuous, however one is a catagorical factor ...
2
votes
0answers
66 views

Why does the log transformation bring data closer to normal distribution? [duplicate]

Quite often in published research we see researchers apply log transformation to their data, and some claim that this makes the data closer to normal distribution. My questions are: Mathematically, ...
0
votes
0answers
10 views

Customer Retention Rate for a subscription business selling products with multiple contract durations

I have a question related to the standard metric used to evaluate subscription model businesses, aka user retention. We define user retention as the percentage of customers that keep their ...
0
votes
0answers
9 views

How to interpret a specific data transformation?

I came across this specific data transformation in the context of a physics application, which by itself is rather complex and hence out of the scope of this question. However since this ...
2
votes
0answers
34 views

log transform in linear regression

Assume we have a data set and the theory suggests to model $Y \sim X$. We apply a simple linear regression and get the following: Next, let us make a log transform of both $X$ and $Y$. The result is ...
0
votes
0answers
20 views

What is the back-transformation of Log((Max Y + 1) - Y)?

I have performed a linear mixed model regression with the transformation on the outcome variable, Y, Log((Max Y + 1) - Y). This gives normally distributed residuals when simpler transformations did ...
1
vote
1answer
42 views

3-part question on joint PDFs

a.) Let U, V be uniformly distributed over the set $\{(u,v): $$0<u<v<1$}. Let $X$ = $-$$log(U)$, $Y$ = $-$$log(V)$, $Z$ = $max$($X$,$Y$). a.) Draw the support of the joint distribution ($U$, $...
0
votes
0answers
15 views

Can I log-transform the sample variance estimated from an originally log-transformed data?

I have a dataset that is log-transformed to meet the assumptions of normality. I have estimated the mean and variance of each individual. I understand that the obtained mean is log-transformed. My ...
0
votes
0answers
12 views

Dealing with an imbalanced dataset in text mining

As an English major with no traditional training in statistics, I am having a very rough time with this, so any help would be greatly appreciated. My problem is that only 849 books out of my 6360 book ...
1
vote
0answers
36 views

$T^2$-statistic for transformed data

I was faced with such an exercise in the textbook: Consider the $T^2$-statistic $N(\bar {\mathbf x} -\mu_0)'S^{-1}_X(\bar {\mathbf x} -\mu_0)$ for $H_0: E[X]=\mu_0$ based on a sample of size N with ...
0
votes
0answers
15 views

Multivariate Box-Cox Transformed Normal Generation?

My data sets are decidedly non-normal (mine have fat tails and sometimes have convex tails and other times concave tails) but correlated. For that reason, I think a Box-Cox transformation on each data ...
2
votes
1answer
37 views

interpretation of $A^T A$ [duplicate]

Is there a statistical/information theoretic interpretation of this matrix in the context where A represents observations of data (and this matrix which shows up in e.g. solving linear systems and ...
1
vote
0answers
26 views

Is it legit to use a rank-based transform for two linear regression in series?

I have to perform two linear regressions in series, where the response of the first one is used as a predictor in the second. For the sake of introducing a common notation, let us say that the first ...
3
votes
2answers
395 views

Effect of nonlinear transformations on the mean

Suppose I have a continuous random variable $X$ and a random variable $Z = f(X)$, where $f$ is a nonlinear monotonic transformation. How can I prove the following relation between the mean and the ...
0
votes
0answers
13 views

What is the effect of Standardization on tree based models?

I have read that tree-based model such that Decision Tree, RsndomForests, Xgboost, etc do not require the standardization of data before feeding it to model but when I was working on working on ...
2
votes
1answer
46 views

Parameter estimation of a transformation

imagine you have a sample $X_1, X_2, \ldots, X_n$ from a random variable $X$, and another sample $Y_1, Y_2, \ldots, Y_m$ from a random variable $Y$. You know that $Y = \phi(X)$. For concreteness, say $...
1
vote
1answer
20 views

Understanding the output of powertransform

Working in a multiple linear regression setting I am attempting to fit a general model to a data set that consists of 7 predictor. To do so I have run the powertransform() function and received the ...
0
votes
1answer
86 views

ARIMA forecasts with autocorrelated residuals

I have a time series on consumer price index (CPI) and want to forecast inflation which is in my case the first difference of the log of CPI: ...
0
votes
1answer
30 views

propability when Y = ln(X) [closed]

If $Y = \ln(X)$ and $P(Y) = 0.55$ Can i say that $P(X) = e^{P(Y)} = e^{0.55}$ ? ($P(Y)$ is the probability of $Y$, and $P(X)$ is the probability of $X$)

1
2 3 4 5
39