Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

learn more… | top users | synonyms (2)

0
votes
0answers
6 views

LDA: strange results from sklearn, my code

I ran an LDA (LDA Transform Class) using the eignevector method. Here is my schema: $$ Sw^{-1}S_{bc} = W $$ I then find the eigenvectors of W, sort, and, for the transform (assuming the ...
0
votes
0answers
17 views

Transformation of dependent variable for MARS algorithm?

I am just wondering if its necessary to transform a dependent variable as it is a large monetary value? I'm unsure if its necessary with a non-parametric methods such as MARS. When I do a log ...
0
votes
0answers
10 views

If you transform response time data, e.g. to generate a CI, do you transform the values back for interpretation? [duplicate]

I would like to create a CI or highest density interval for response time data. The distribution of the response times is quite skewed and I think about transforming them by LN(y). However, my ...
0
votes
1answer
20 views

Method For Calculating Performance from Average Scores

In a simplification of my problem, let's assume we have players who can get an integer value score $s$ considering $\{ s \in \mathbb{N} : 0 \leq s \leq 3 \}$. Same sample data for scores: Player A: ...
0
votes
0answers
36 views

In OLS, how would log transformation of variables affect the estimation of coefficients? [duplicate]

I am not sure if the log transformation of the dependent (or independent) variable affects only the interpretations of the estimated coefficients of the independent variable? Are there any other ...
1
vote
1answer
21 views

Normality test on logarithmic data

I had to take the log of my data, and now I want to test for normality of my data. Can I use the standard normality tests or do I have to use some special test? I used the ...
0
votes
0answers
25 views

Pre-processing (center + scale, box-cox transformation) inside cross-validation?

I have extracted features and I have now a matrix where the rows are the data points and the columns are the features. Of course, I have to center and scale (zero mean and unit variance) each feature ...
1
vote
1answer
49 views

Percentage interpretation of negative values when you can't use log transformation

I have a data set of 5 indicators of the stock market. 2 of the indicators have negative values: e.g. they range from say -50 to 100. After running a regression I would like to be able to compare the ...
1
vote
0answers
62 views

How to calculate $E(x)$ and $V(x)$ when $g(x)\sim f(g(x))$

Let assume that we are interested in a variable $x$. We know that e.g. $g(x)=x^2$, $g(x) \sim Uniform(a,b)$ or any other distribution. From that I can calculate $E(g(x)) = (a+b)/2$ and ...
0
votes
1answer
24 views

Why do we transform proportions and what are the methods to do that?

When proportions are condensed below 1%. It become asymmetric on the distribution curve, so we get the log() of the proportions to make it symmetric around zero. Is that correct what else can the ...
2
votes
0answers
33 views

GLMs with transformed response variable

I wonder if use of generalized linear models (GLMs) with transformed response variables is correct. My particular case: I compared goodness of fit of several GLMs with response variable transformed ...
1
vote
0answers
16 views

How should I transform a featureset (15000 variables) that is mostly presence/absence, but present values are lognormal distributed?

I am trying to learn machine learning and have a nice featureset with a binary classification. The dataset is 15000 variables and 2500 data rows. For every data row, almost all variables are 0, and ...
1
vote
2answers
94 views

R-squared and fit of the model

This post is new version of my last question. It was deleted due to my error in data. Currently data updated and verified. I have 2 models. Model 1 gives me: ...
6
votes
1answer
150 views

Box-Cox transformation for repeated measures ANOVA (rANOVA) in R

My proposition of the solution Perform Box-Cox-transformation despite of knowledge that obeservations are nont independent and then use rANOVA to estiamte coefficients using the knowledge that the ...
0
votes
0answers
13 views

Generalized Linear Model on SPSS with the 'error': “set to zero because this parameter is redundant”

For my dissertation I have a lot of data and many nominal variables. None of my data is parametric. I tried transforming some of the percentage data with arcsine because it was in proportions and the ...
1
vote
0answers
29 views

How to interpretation of the results of PCA

There is a larger matrix (1500 rows x 40 columns), 1500 observations x 40 variables. then I follow the procedures of PCA(Principle components analysis), 1. find correlation 2. find eigenvalues 3. ...
2
votes
2answers
128 views

When does it make sense to log-transform input variables in multi-variable logistic regression? [duplicate]

When does it make sense to log-transform input variables in multi-variable logistic regression? The transformation improves model metrics a little bit, but I'm not sure how to justify it and whether ...
1
vote
1answer
38 views

Box Cox Transformation makes Out of sample Forecast Error worse?

I am doing a regression on time series data. I have 60 lagged predictors which I will call x to predict a continuous variable y. I used the BoxCox function from the forecast package to transform y and ...
10
votes
4answers
824 views

Is it a good practice to always scale/normalize data for machine learning?

My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect ...
0
votes
0answers
21 views

AIC model selection in R. Transform and back-transform parameter estimates (slopes)

I have run an AIC model selection in R, (following Grueber etal 2011 J.Evo.Bio), and standardised my global model to a mean of 0 and SD of 2, using the "arm" package. AICc selection identified 2 ...
1
vote
0answers
50 views

Alternatives to Box-Tidwell transformation for ridge regression?

I would like to fit the following model by ridge regression (the xs correlate strongly with one another) $y = \beta_1 {x_1}^{\lambda_1} + \beta_2 {x_2}^{\lambda_2} + \beta_3 {x_3}^{\lambda_3} + ...
1
vote
0answers
22 views

Structuring Data for Logistic Regression - Probability of Attrition Model

I am struggling with the best method for setting up my data for building a probability of attrition model and was hoping I could get some help/ideas here. Basically, I have a very large dataset ...
0
votes
0answers
21 views

The scaling problem of ridge regression

I have been confused with the scaling of ridge regression input for a long time. There are several sources about how to do the scaling: Just do the centering to input(From "The Elements of ...
0
votes
1answer
32 views

Is it necessary to transform data before k nearest neighbors?

I am trying to follow a method of filling in null values proposed by AirBnB. They have you transform the data before computing distances for KNN. Why is it necessary to transform data? Is its ...
0
votes
0answers
9 views

Given some files and their parsed content how can I reverse engineer the file format?

I have access to a few thousand binary files and I don't have the program that generated them. The files have been parsed and I also have access to the database where the parsed fields are stored. How ...
2
votes
1answer
69 views

Does a linear recombination of features affect random forest?

I'm aware that linear transformations of individual features do not affect Random Forest. But what if features were linearly recombined to construct a new feature? I do the following: I take each ...
0
votes
1answer
26 views

Create categories of individuals from data about them

I have a big amount of individuals and few variables defining them (between 5 and 10). I'd like to be able to categorize the individuals in few categories, regarding the values of the variables. I ...
4
votes
0answers
44 views

Why does Box-Cox transformation fail in following situation?

In the book, Linear Models with R by Julian Faraway, it says that if max(y)/min(y) is small, then Box Cox won't do anything because power transformations are well approximated by linear ...
0
votes
0answers
11 views

Transforming bi-variate uniform random to bi-variate normal [duplicate]

Hi please help me solve this question; I am given two $iid$ random variables $X_1$ and $X_2$ with $Uniform(0,1)$. Then we transform the variables as follows; $$ Y_1 = \sqrt{-2\ln(X_1)}\cos(2\pi ...
0
votes
1answer
31 views

Log-transforming time series data before cointegration testing

I am testing the cointegration between these variables: Gold Price (Ringgit), Exchange Rate - MYR to USD (Ringgit), Real Effective FX Rate Based on CPI, T-Bill 10 Years Rate, Consumer Price Index. ...
0
votes
1answer
29 views

Help understanding DCT compression for a vector

I have multiple 60-dimensional vectors on which I need to apply DCT and reduce to various dimensions. I'm trying to understand how this happens: ...
2
votes
1answer
60 views

How to deal with non-normal variable distributions in R glmnet

I am regressing actual counts of traffic against predictions using ridge regression (cv.glmnet in R). The data (both predicted and actual) has a roughly ...
3
votes
1answer
48 views

log-log transformation

I am making a linear regression model for house prices. My data set includes price per square foot of a property and floorspace so I have multiplied them to get the total price of each property. My ...
1
vote
1answer
36 views

Proof that square of a standard normal r.v. has Chi-Square Distribution using MGF's

Supposes $Z \sim N(0,1)$. We know that $Z^{\top}\!Z\sim\text{Chi-Square}(1)$. Does the proof for this concept require the use of moment generating functions/method of moments per say?
3
votes
1answer
58 views

Sort $X$, then scale the first differences by $\bar X$: what, if anything, is this used for?

Sort data vector $X$, take first differences of the sorted data, and divide by $\bar X$. I came across this transformation in someone's notes, without any citation. It would be applied to ...
0
votes
0answers
14 views

Advice about modeling visit data retrospectively

I have retrospective patient level data on number of emergency department visits in a given year for a large urban ED - mostly 1's and 2's but range up to over 20 (data were collected at an ED visit ...
0
votes
0answers
32 views

Difference of a mean and a median after logarithmic transformation

I have a rather skewed sample, its skewness is about $3$. Then I use the logarithmic transformation, the skewness of the transformed sample becomes about $0.4$; the histogram also seems to become more ...
0
votes
0answers
18 views

Transformation of Adoption Timing / Post-Adoption Usage

I'm currently working on a project where I intend to simultaneously model adoption timing (a dependent variable which takes on a value of 0 ~ 5 operationalised as ...
2
votes
0answers
35 views

Comparison of models with transformed dependent variable

I want to check if transforming the dependent variable positively influences the model performance. For example, I have built two models using the caret package. ...
5
votes
0answers
53 views

Find the rotation between set of points

I have two sets (sourc and target) of points (x,y) that I would like to align. What I did so ...
1
vote
0answers
24 views

Alternative scaling methods to log

I have data (as below). When I plot, the details of 0-10 are lost. So I scale it using a log function and then it seems that detail is lost throughout. Is there another scaling method that would work ...
1
vote
0answers
52 views

GLMM or LMM with transformed data

I have a response variable that is the absolute value of a difference between two proportions. The distribution looks like this: I also have two values for each of 20 individuals, so I need to use ...
7
votes
2answers
119 views

Transforming Order Statistics

Assume random variables $X_1, ... , X_n$ and $Y_1, ..., Y_n$ are independent and $U(0,a)$-distributed. Show that $Z_n= n\log\frac{\max(Y_{(n)},X_{(n)})}{\min(Y_{(n)},X_{(n)})}$ has an $\text{Exp}(1)$ ...
6
votes
2answers
176 views

How to determine how many variables and what kind of variables a table of data has?

Here's an example regarding what I'm lost on. In the past I've just found myself counting columns to determine how many variables a table has, but I've realized that's totally insufficient. For ...
1
vote
0answers
13 views

infer (approximate) ranks or scale values from partial paired comparisons with only one rater

I have a set of items $10$ $(A, B, .., J)$ and paired comparisons between a subset of those items, with a value of $1$ indicating that the column item is ranked higher than the row item and a value of ...
0
votes
0answers
9 views

Negative Cronbach's alpha after reversing results of a survey

As part of my master thesis, I am conducting a PCA analysis. The analysis is aiming to find commonalities between 8 survey questions that were answered by 219 respondents. The survey was answered on a ...
1
vote
0answers
14 views

Estimation of arithmetic Brownian motion volatility with transformed data

I want to estimate the volatility $\sigma$ of a process $(X_t)$ following an arithmetic Brownian motion, that is, for a constant time step $\Delta$, $X_{t+\Delta} = X_t + \sigma B_{\Delta}$ , where ...
0
votes
0answers
23 views

Transformation to be used for continuous variable

I have a data set where I am doing a binary classification. I have close to 500 features and 200K observations. Now I also have few continuous variables as features. I don't think just using these ...
0
votes
0answers
30 views

How to calculate marginal effects, average partial effects in multinomial logit model of the form Y ~ 0 | X1 + X2 + X3

I am a doing research with a dataset of students to develop a model that predicts a course that is available for them. the dataset ...
1
vote
0answers
16 views

Should I re-transform country dummy coefficients to be analyzed in a second stage?

I am working on panel (country-year) macroeconomic data which suffers from missing data in some of the independent variables. My sample is wide and short, around 120 countries over 10 years. In past ...