Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

learn more… | top users | synonyms (2)

0
votes
0answers
7 views

Recursive function that operates on its own preceding output

I have the price for a particular baseline year (in this case for 1993), and the multiplication factor for all the years. Using these known multiplication factor, I want to compute (project) price for ...
0
votes
0answers
5 views

Model name for regression with square root transformation on the response variable

Consider different transformations on the response variable. If log transformation is used: $$\log{Y} = \alpha + \beta X + \epsilon$$ The model is called log-linear model or semi-log model. My ...
0
votes
0answers
15 views

skewness and hypothesis testing (t-test and anova)

Some of my variables are heavily positive skewed (left skewed). With log transformation, some are closer to normal distribution, but some are still positively skewed, though not that bad before log ...
2
votes
1answer
53 views

How does the support of a continuous random variable change under transformations?

Let $X_1$, $X_2$ have a joint pdf and support set $S$. Suppose random variables $Y_1$, $Y_2$ are given by $Y_1=u_1(X_1,X_2)$, $Y_2=u_2(X_1,X_2)$ where the functions define one to one transformation ...
0
votes
0answers
5 views

R cut zero-length interval [migrated]

I have a column that has numeric values in the 1--7 range. I would like to use the cut function to split these values into the following intervals: 1 -> 1, ...
-1
votes
0answers
11 views

SPSS statistics data file - how to deaggregate dechotomous variables for logististic regression [on hold]

I want do to a logistics regression with data in a SPSS statistics data file. The independent variable is a set of sites, and the independent variable is success. The dependent variable information ...
10
votes
0answers
1k views

Why is a square root not a linear transformation? [migrated]

The question says: Prove that the function $f(x)=\sqrt{x}$ is not a linear transformation (particularly $\sqrt{1+x^2}≠1+x$) I think that this is because the exponent of $\sqrt{x}$ is $1/2$, ...
1
vote
1answer
27 views

Confused about PCA transformation vectors

I'm trying to get the intuition of how PCA works. So far I understood that: I start from the input matrix $X = [X_{1},...,X_{p}]$ where each $X_{i}$ is composed by $n$ elements that are the $n$ ...
0
votes
0answers
11 views

BoxTidwell test for logistic regression

I'm curious as to how BoxTidwell works in R. The page for the package itself seems to lack descriptions. I have a logistic regression with many numerical and categorical predictors. Every time I use ...
1
vote
2answers
52 views

Best way to optimize MAPE

The MAPE is a metric that can be used for regression problems : $$\mbox{MAPE} = \frac{1}{n}\sum_{t=1}^n \left|\frac{A_t-F_t}{A_t}\right|$$ Where $A$ represents the actual value and $F$ the the ...
0
votes
1answer
30 views

Can I iteratively transform a variable with log10 until it fits a linear model?

I have a response variable, $Z$, for which I'm trying to make a linear model. Here are some of the fit diagnostics plots: From the fan-like shape of the residual-vs-predicted value plots, I ...
1
vote
1answer
18 views

what's the difference between weighting and transforming?

In regression analysis, what's the difference between weighting and transforming when it comes to spreading residuals? For example, we need to weight the model $y=ax+b$ by $1/x$, isn't it just ...
0
votes
2answers
16 views

Non-linear transformation to increase separability between clusters

I want to do a classification on PC scores. I have a 400 dimensional matrix, e.g. 2000*400 (2000 number of samples and 400 dimensions). I fist apply PCA on it and take it to 3D, i.e. 2000*3. There are ...
1
vote
2answers
15 views

Understanding output of powerTransform

In the car package, we have the function powerTransform which transforms variables in a regression equation to make the ...
3
votes
1answer
110 views

Why do deep learning practitioners forego PCA for ZCA?

I have an understanding of PCA and ZCA, read a similar question on the subject which, unfortunately, does not have the specific answer to my question. I understand the benefits of data whitening: ...
3
votes
2answers
69 views

What to do with data that are bimodal at two tails of the distribution?

I am in a weird position where I prespecified a plan to use linear regression to analyze my data, and stated I would use transformations to address any assumption violations. I'm pretty certain my ...
0
votes
1answer
21 views

Is Woe Transformation is required for all variables in a logistic model

i am trying to build a logistic regression model. I have a doubt regarding using woe (weight of evidence) transformed variables. I wanted to know if it is ok to use a few woe transformed variables and ...
1
vote
1answer
26 views

In a class with multiple teachers, how can I transform student scores based on their teacher's average compared to the population average?

It has been ages since I've taken any statistics courses, and I have found myself in the following situation: I am in charge of a university course with about 400 students and 10 assessors. There ...
1
vote
1answer
20 views

transformation for non-constant variance?

This is from my textbook I don't understand what does the content in red mean, for example, what does $y^2_i \infty \sigma_i$ mean? How can we tell the relationship between $y^2_i$ and $\sigma_i$ ...
4
votes
2answers
175 views

R: Box-plot on log scale vs. log-transforming *then* creating box-plot: Don't get same result

In the boxplot() function in R, there exists the log = argument for specifying whether or not an axis should be on the log ...
0
votes
0answers
14 views

How to run 2 Way ANOVA on SPSS with data that is not normal distributed?

I am about to run 2x2 ANOVA on my data, but then I realized that my data is not normal. I have tried to do data transformation like Log10 and Ln, but the data is still not normal. The data has ...
1
vote
0answers
31 views

Is/are there any threshold value(s) to determine to see if PCA is useful at all, specially for high dimensional data?

Apologies if this is a naïve question, but it's not so naïve to me! Let's first assume we have 2D data which are perfectly linear but not along the x- or y-axis. PCA will rotate it so that it becomes ...
0
votes
0answers
18 views

Google correlate - data transformation / switching positive / negative correlation

Google correlate Takes your data and then searches for positive correlations in search terms. From my understanding it returns search terms that positively correlate to your data. For example if ...
1
vote
0answers
86 views

Data structure for rare event predictions in temporal domains

I am a beginner in rare event modeling. I am working on predicting modem failures within a network where failures occur approximately 3% of the time. Currently my data is structured as follows: ...
0
votes
1answer
23 views

Transformation from skewed to symmetric distribution

Let us consider a positive valued random variable $X$ which is following a positively skewed probability distribution. Is it possible to a get a function $f$ (one-to-one) for which $f(X)$ follow a ...
1
vote
0answers
14 views

Normalizing skewness with the Power or Box-Cox Transformation

Suppose I have a random sample drawn from an arbitrary strictly positive continuous distribution. Suppose moreover that I want to use the Box-Cox transform to zero out the skewness. Is there an ...
1
vote
0answers
43 views

Variable transformation: from angle to coordinates

I have the following problem: Let $\theta \in [0,2\pi)$ be an angle and let $f(\theta)$ be a function such that $\int_{0}^{2\pi}f(\theta)d \theta=1$. Now let consider the following transformations: ...
0
votes
0answers
6 views

How to use time dependent covariates with cox regression in R

I don't know how to generate time dependent covariates in R for use cox regression. I know you need to reorganize your dataset into intervals between event times. This I believe I can do with the ...
5
votes
1answer
34 views

Are parametric tests on rank transformed data equivalent to non-parametric test on raw data?

Many non-parametric tests are identical to their parametric equivalent on ranked data. At least, that's what I learned from this blog post on Friedman's test and skimming this 1981 article.. This ...
1
vote
0answers
20 views

How to perform quantile transformation with missing values?

Given are an input vector $I$ with missing values and a target/reference distribution ${T}$. For example: $I$: 0.215 NA 0.103 0.649 0.057 0.292 NA 0.433 0.521 NA $T$: -0.996 -0.606 -0.394 -0.090 ...
3
votes
0answers
23 views

Linear regression with trimmed data

I would like to know how experts deal with real data. Even if statistical text books uses real data I'm always surprised how good the real data are and at the end of the exercises the residuals are ...
5
votes
1answer
90 views

Should the correlation PCA projection be computed on original or normalized samples?

Suppose we compute the correlation PCA of a dataset $X$ (with $m$ variables and $n$ observations) by first normalizing the input variables. That is: mean -> 0 and standard deviation -> 1. Let us ...
0
votes
0answers
27 views

How to predict when using normalized data?

So, I am taking this course on machine learning by Andrew Ng. Wanted to write my own linear regression program. Everything is fine. I mean normalize data, and run linear regression. Now I'm left with ...
0
votes
1answer
19 views

Include both linear and non-linear dependency of the same variable in a multi-variate analysis

I am implementing a multi-variate analysis using 5 covariates. My model looks like this: lm1<-lm(Y ~ (T(A) +A + B + C + D + E)^2, data=data) where T(A) is a ...
2
votes
1answer
45 views

Variable standardization / scaling for PCA when all dimensions already have same scale [duplicate]

Often when PCA is performed on exam results where all variables (dimensions) have the same $0$ to $100$ scale, scaling is none the less applied. For different scales I can see the purpose of it, but ...
0
votes
1answer
21 views

How to interpret a log transformed (x+c)? [duplicate]

I need to use log-log regression and because I have lots of zero values I tried to add a very small constant c=8E-12 to x and it works pretty good. Xs are very small probabilities. lnY= a + b ln ...
0
votes
0answers
41 views

More effective seasonal adjustment to time series data?

I am trying to predict surface temperature using solar energy. I have 3650 daily averages for both variables. The plots of both are below: I attempt to seasonally adjust with a periodic ...
1
vote
2answers
56 views

What transformation should I use for a bimodal distribution?

I have some bimodal data like the one generated down (R language), and I don't know how to transform it to have a normal distribution or homoscedasticity. I'm running a linear discriminant analysis ...
3
votes
1answer
74 views

Does the IQR or standard deviation change when scaling or shifting non-normal data?

I'm aware that scaling or shifting data that is normal or almost perfectly normal will not significantly change the standard deviation. Through practice I've that this is not the case with non-normal ...
0
votes
0answers
27 views

Data does not have a normal distribution but has homogeneity of variance

I'm trying undertake some statistics for my masters thesis but I'm having some problem with my data not being normally distributed. I've essentially got 3 factors, one with 2 levels, one with 3 levels ...
2
votes
1answer
53 views

Does using difference transformation lead to bias? (Levels vs differences regression)

Consider the model estimated in levels (also assume this is the true population model): $$y_t = x_t\beta + e_t$$ As usual we have the dependent variable $y$, independent $x$, the error term $e$, and ...
1
vote
1answer
41 views

How can I combine nominal with ordinal data to build a unique variable?

I performed an Interview with 44 questions Protocol. The structure of questions is based on 18 variables. Major variables are coming from theory. Every major variable consists of 3,4 or more question ...
0
votes
1answer
31 views

Select and aggregate time series based on selection information of a second dataset

General problem: I have two datasets in r and I do not know how I can calculate information across groups of time series in one dataset based on selection-information of another dataset. The details: ...
1
vote
0answers
19 views

Which model(s) are appropriate for this kind of data

So, I tried to implement a model on some data. The dependent variable is a ratio that can get higher than 1, is lower bounded by zero and, seeing figure 1, is left skewed.Thus, a logit regression is ...
1
vote
0answers
9 views

Compare between median/IQR and reference mean/SD

I have a small set of non-normally distributed measurement data (Kolmogorov-Smirnov rejected similarity to a normal distribution) and a reference value from a large population (n=120) of healthy ...
1
vote
2answers
40 views

Can trees or random forests learn ratios

This is a question about feature engineering for decision trees/random forests. Given two continuous variables X1 and X2, is it ...
0
votes
0answers
32 views

Can I use the Xie-Beni index to validate data transformation parameters in fuzzy c-means clustering?

I am using fuzzy c-means algorithm to cluster my data in various feature spaces and the results differ depending on what kind of transformation I perform on my raw data. I want to know if using the ...
1
vote
1answer
25 views

Transforming Power and Exponential Functions

Suppose two variables x and y have no linear correlation. If we transform the data by replacing each y value with its base-10 logarithm, then will x and log $y$ also have zero correlation? In ...
0
votes
0answers
10 views

Log transformation in SPSS for percentage dependent var

indep_avg dep_% Frequency 3.4 5.6 67 5.6 2.5 96 3.2 6.3 23 I have some aggregated data. The Independent ...
0
votes
0answers
35 views

How to transform feature with peak at zero to normal distribution?

I have a feature in my dataset which has lot of zero values, i.e. a big peak at zero (the zeroes are valid and valuable information). The histogram is the following: I want to transform all my ...