Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Number of bins for discretization

How do I decide on the right number of bins to discretize my continuous data? Are there are tests/techniques to do the same? Could someone give me some idea into existing approaches?
0
votes
0answers
26 views

Square Root Transformation on Simple Linear Regression Model

I am having a bit of trouble with square root transformation on a simple linear equation. I was given the following equation: Y = β1X + ε, where ε ~ N(0, σ2X). I need to show that with the square root ...
1
vote
0answers
22 views

Not sure of model choice - significant p-values but very low R2 adj

I am writing a thesis in which I'm trying to see what the relationship is between how innovative a patent is (as determined by a series of measures based on NLP that) and the financial value of that ...
0
votes
0answers
14 views

(Multiple) Linear Regression (LM and GAM); Q: Transform just the response and scale the predictors?

I've been working through some regressions, and I was told by a credible source to not worry about transforming the predictor(s) but be sure that the univariate response values are distributed close ...
1
vote
1answer
18 views

Should data be anonymized irrespective of the domain? [closed]

I am using a dataset from Marketing and sales department. The dataset contains customer name (company name), company address, pincode, no of orders placed, revenue generated from that customer etc. My ...
0
votes
0answers
20 views

Can I log transform independent variables in binomial regression? [duplicate]

I am doing a binomial regression with many independent variables (some of them are country-level macro data, others are firm-level performance data). One variable is GDP and I figured it would make ...
1
vote
0answers
39 views

Legitimacy of transforming data before statistical tests

I have two groups of samples (N=4 for each) and found that there is one outlier for each group (both are higher than the rest of the respected samples within the groups). I have no resources to repeat ...
0
votes
0answers
8 views

Handling null continuous values in Logistic Regression with inverse values

I've got a data set of prisoners, where the measured variable is the probability of a recurring act of crime, which is counted as having occurred when a former convict is interrogated as a suspect in ...
11
votes
2answers
698 views

Distribution of the exponential of an exponentially distributed random variable?

Let $X$ be an exponentially distributed random variable, that is, with density function $f(x)=\lambda e^{-\lambda x}$ for $x\ge 0$ ($\lambda>0$), and cdf $F_X(x)=1 - e^{-\lambda x}$. What is the ...
0
votes
1answer
29 views

A schizophrenics experiment from Bayesian Data Analysis, Gelman et al textbook

There is a psychology experiment in the Gelman et al textbook on page 524 section 22.2 Example: reaction times and schizophrenia, I believe the original work is from Belin and Rubin (1990) where the ...
0
votes
0answers
17 views

How to give non-numeric input feature in Keras' Neural Network model [closed]

I am training my NN model with Keras in Python. I have cleaned, formatted the data as per my need. (see the image below). The time is in 10-min. intervals (data is randomly shuffled so ignore the non-...
0
votes
1answer
19 views

Application of transformations of lognormal distributed random variables

I'm answering questions from a book and I have to do a simple transformation Y=g(X) of a lognormal distribution: $X \sim \mathcal{LN}(\mu,\,\sigma^{2})$ with $Y=ln(X)$ Then $Y \sim \mathcal{N}(\mu,\,\...
0
votes
0answers
7 views

When do we need to log transform predictors for logistic regression model? [duplicate]

I am interested in knowing when we should log-transform the predictors for the logistic regression model? My predictors are highly skewed but I read about some materials online that we don't need to ...
0
votes
0answers
11 views

What type of Chart is best to display for which of the 2 Data effects the Spread more at a given point?

For 2 Variables of Data, i have taken a Spread between them. Spread = Data A - Data B. I want to display inside of a Chart, Which Data has more of an Effect when the Spread changes. For Example, if ...
0
votes
1answer
14 views

Examine price changes vs price returns

In empirical financial analyses, is there a difference when examining stock price changes and stock price returns? Would a model with a response variable price changes tell a different story than ...
0
votes
1answer
44 views

Forecasting with log transformed data and then taking the exponential of my predictions [duplicate]

I am aware a similar question was asked here Interpreting forecast predictions of log transformed data. But I would like a deeper comment/answer. I have a time series and want to make forecast. I ...
0
votes
0answers
23 views

Can log-transform to TS reduce forecast accuracy vs non-transformed TS

I have to apply a SARIMAX model to over 300 TS. Many of them have increasing volatility which can be addressed by aplying log to the TS. Problem is that also many TSs don't have this problem. Is it ok ...
0
votes
0answers
19 views

How do I construct confidence intervals involving out of bounds values from the prob scale to the logit scale and backtransform to the prob scale?

I am trying to figure out how to construct confidence intervals with observations' outcome values that were out of bounds on the probability scale (≤0 and ≥1). For individual observations, I get ...
0
votes
0answers
15 views

mean center interaction term

I'm using multiple linear regression to measure both the main effects of SES (income, household size, education level) on a kindness scale, adding an interaction between income (rating between 1-7 ...
0
votes
0answers
16 views

Opportunity Evaluation within excel

I'm a student of data analytics. I've the following raw data https://1drv.ms/x/s!Amd7BXzYs7AVhAqQGtBZLHzQ1xZ6 The main goal as for finished example :https://1drv.ms/x/s!Amd7BXzYs7AVhAsWa0e5yvqPhuRf?e=...
1
vote
1answer
28 views

Appropriate to transform a "normal" distribution?

I have a dataset showing normality based on the p-value in a linear regression model. However, the drop on the left side gives me pause that the data is acceptable to analyze without transformation, ...
0
votes
0answers
33 views

Transformed Data with Boxcox and negative Lambda

unfortunately I'm a statistics beginner. I am trying to transform my data. Here, according to Boxcox, a transformation with negative lambda has resulted ( L =-0.25). But If I now run a T-tests with ...
0
votes
0answers
54 views

How to compute a prediction interval on rescaled power transformed data?

Given a training dataset $\mathcal{D} = \{ (x_i, y_i) \in \mathbb{R}_+^2 | 1 \leq i \leq n\}$ one can fit a linear regression $y_i = a \cdot x_i + b + \varepsilon_i = \hat{y}_i + \varepsilon_i$. That ...
0
votes
0answers
18 views

How to perform inverse ratio transformation?

This is partly a "how to" question and partly "is this correct". I have a variable that is the ratio of two variables (ratio=a/b). Similar to as shown below in R: ...
5
votes
3answers
122 views

Variance of a function of a random variable as function of the original variable

Suppose $X$ a random variable and $Y=f(X)$ a function of this variable I know I can write $\mathbb V(Y)$ as $\mathbb E(f(X)^2)-\mathbb E(f(X))^2$ I would like to know if it is possible to write $\...
0
votes
1answer
17 views

Box-Cox (or Yeo-Johnson) for Two-Way ANOVA

I want to run a Two-Way ANOVA test on an experiment which has a two-by-three configuration of experimental conditions, and one hundred data for each experimental condition, hence, 600 total data ...
1
vote
1answer
47 views

Normal distribution vs Lognormal distribution for residuals in Bayesian linear models

When should I use normal distributions for residuals, and when should I use lognormal distribution in Bayesian linear regression? For example, if I want to create a regression model for chemical ...
1
vote
1answer
38 views

Transform variables with zero-inflated values and positive skewness

I have over 30 features: several have zero-inflated and highly positive skewed distribution. Those distributions are expected because they are semi-continuous monetary related features. For example: ...
2
votes
0answers
12 views

Forecasting by autoregressions of invertible linear transform of variables

Suppose I have two variables and two alternative variables, where the latter are an invertible linear transform of the former. Suppose further, that I estimate both the new and the original variables ...
2
votes
2answers
99 views

Find the variance of $Y = X_1 X_2 / X_3$ where $X_i \sim$ Uniform(1,2) for $i = 1,2,3$

The problem is very straightforward and it says: Let $X_1,X_2,X_3$ independent random variables with a common density $f_{x_i} \sim Uniform(1,2)$. Define $Y = X_1 X_2 / X_3$ Find the numeric value of ...
1
vote
0answers
37 views

Power transformation & pearson's correlation - how to interpret?

In the first phase of my analysis, I had transformed a skewed variable (normality violated, heterogeneity of variance violated) using a power transform (x^-2) to be able to conduct parametric tests. ...
0
votes
0answers
21 views

Using PCA to construct index from survey items

I have a data set consisting of responses to dozens of different survey items. I would like to create indices (such as for example political orientation) from some of these items representing closely ...
0
votes
0answers
25 views

Is it OK to transform non-normal proportional data

I have a dataset for 'time spent' (hours) carrying out 'behaviours' (walking, lying, standing, grazing), during day or night 'periods' of cattle. Time spent is my dependent variable, with behaviours ...
0
votes
0answers
22 views

Difference in using mean of log(X1),log(X2)...log(Xn) vs. geometric mean (X1,X2,Xn)?

I have positively skewed data that I had to transform to log(). The observations are distributed across 3 groups. For example, in group #1: I calculated the geometric mean of the raw data (GM = 224), ...
0
votes
0answers
7 views

question about data transformation for structural equation modeling

I am using structural equation modeling and need to transform the data before using maximum likelihood estimation. I am using the book Principles and Practice of Structural Equation Modeling by Kline ...
2
votes
1answer
22 views

Why does Tukey's "1 - slope rule" work for transformations in ANOVA?

I'm reading the book STAT2: Modeling with Regression and ANOVA. In Section 5.6, the authors consider what to do when you want to do a one-way ANOVA but the group standard deviations are not equal. ...
0
votes
0answers
4 views

Visreg Plotting log-log as log-level

I want to run a regression of money spent on links clicked using a data set where I notice link clicks level off after a certain amount of money spent. I want to use a log transformation to better fit ...
1
vote
0answers
13 views

What is the best practice for logistic regression data perspective?

I'm running into an interesting problem (for me at least) right now where I'm worried that my data setup is negatively influencing the accuracy of my model. Lets use historical chess matches as our ...
0
votes
0answers
17 views

How to get the support of the inverse BoxCox's distribution?

I will take only the BoxCox's transformation for $\lambda \neq 0$ so it's: $Y_i=\displaystyle\frac{(X_i^{\lambda}-1)}{\lambda}$ and its inverse: $X_i=e^{ \displaystyle ln(Y_i \lambda+1)/\lambda}$ ...
0
votes
0answers
24 views

Can I use log of two variables and also the ratio of the two variables in linear regression

I am working on a linear regression model to predict revenue (of a city) from a marketing campaign. The independent variables are spend, population of city etc. [Please note for the model, I only care ...
0
votes
0answers
15 views

Differencing data for square root data

Let $Y_t$ follows ARIMA(1,1,1) and $\Delta(Y_t)=Y_t - Y_{t-1}$. If $\Delta(\sqrt{Y_t})=\sqrt{Y_t}-\sqrt{Y_{t-1}}$ then $\sqrt{Y_t}$ follows ?
2
votes
1answer
22 views

Is sampling from a transformed pdf the same as transforming samples in the same way from the original pdf?

Suppose I have a pdf $f(\vec{v})$ over $\vec{v}$ and define $\vec{u}=T(\vec{v})$, i.e. a transformation of $\vec{v}$. Would sampling $f(\vec{u})$ be the same as taking samples $\vec{v}_i$ from $f(\vec{...
1
vote
1answer
31 views

data transformation for regression

I'm trying to perform LASSO regression on a dataset, and the following 2 photos show the histograms of 8 attributes. I'd like to transform them in some way to improve the model. Data transformation is ...
0
votes
0answers
13 views

How do you interpret ANOVA with transformed response?

Let's say I have a quantitative variable Y that ranges between 0 and 1. I have a categorical variable X that has 3 levels, A, B, C. Since Y ranges between 0 and 1, I transform all of Y using the ...
0
votes
2answers
89 views

Adding Noise to continuous and categorical features?

Assume we have a dataset of 10 features, (combination of continuous and categorical features). I wish to add noise to each features separately, can i use the mean and SD of that particular feature to ...
1
vote
1answer
56 views

interpretation of odds ratio when independent variable is log transformed

In logistic regression, we derive the odds ratio by exponentiating the coefficient (estimate). If one of my independent variables ($X_1$) was log transformed (natural log), how should I interpret the ...
1
vote
1answer
50 views

Can I use predictions from LOESS as a form of predictor transformation in logistic regression?

I am wondering if I can use the predictions from LOESS as a form of predictor transformation in logistic regression? For example, if one of the predictors is X, then can I use predict(loess(Y~X)) as a ...
1
vote
0answers
33 views

How should I interpret this regression coefficient

So I have the model: $$\ln(V_{it})=\beta_1\Delta\ln(A_{it})$$ $$\ln(V_{it})=\beta_1[\ln(A_{it})-\ln(A_{it-1})]$$ $$\ln(V_{it})=\beta_1\bigg[\ln\bigg(\frac{A_{it}}{A_{it-1}}\bigg)\bigg]$$ I am having ...
0
votes
0answers
23 views

Should I normalise the dependant variable for CART regression? [duplicate]

I am working through An Introduction to Statistical Learning (ISLR), pdf. At "8.1.1 Regression Trees" on physical page 328, it says: We first remove observations that are missing Salary ...

1
2 3 4 5
44