Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

learn more… | top users | synonyms (2)

1
vote
0answers
26 views

Is/are there any threshold value(s) to determine to see if PCA is useful at all, specially for high dimensional data?

Apologies if this is a naïve question, but it's not so naïve to me! Let's first assume we have 2D data which are perfectly linear but not along the x- or y-axis. PCA will rotate it so that it becomes ...
0
votes
0answers
15 views

Google correlate - data transformation / switching positive / negative correlation

Google correlate Takes your data and then searches for positive correlations in search terms. From my understanding it returns search terms that positively correlate to your data. For example if ...
1
vote
0answers
24 views

Data structure for rare event predictions in temporal domains

I am a beginner in rare event modeling. I am working on predicting modem failures within a network where failures occur approximately 3% of the time. Currently my data is structured as follows: ...
0
votes
1answer
22 views

Transformation from skewed to symmetric distribution

Let us consider a positive valued random variable $X$ which is following a positively skewed probability distribution. Is it possible to a get a function $f$ (one-to-one) for which $f(X)$ follow a ...
1
vote
0answers
10 views

Normalizing skewness with the Power or Box-Cox Transformation

Suppose I have a random sample drawn from an arbitrary strictly positive continuous distribution. Suppose moreover that I want to use the Box-Cox transform to zero out the skewness. Is there an ...
1
vote
0answers
32 views

Variable transformation: from angle to coordinates

I have the following problem: Let $\theta \in [0,2\pi)$ be an angle and let $f(\theta)$ be a function such that $\int_{0}^{2\pi}f(\theta)d \theta=1$. Now let consider the following transformations: ...
0
votes
0answers
4 views

How to use time dependent covariates with cox regression in R

I don't know how to generate time dependent covariates in R for use cox regression. I know you need to reorganize your dataset into intervals between event times. This I believe I can do with the ...
5
votes
1answer
27 views

Are parametric tests on rank transformed data equivalent to non-parametric test on raw data?

Many non-parametric tests are identical to their parametric equivalent on ranked data. At least, that's what I learned from this blog post on Friedman's test and skimming this 1981 article.. This ...
1
vote
0answers
17 views

How to perform quantile transformation with missing values?

Given are an input vector $I$ with missing values and a target/reference distribution ${T}$. For example: $I$: 0.215 NA 0.103 0.649 0.057 0.292 NA 0.433 0.521 NA $T$: -0.996 -0.606 -0.394 -0.090 ...
3
votes
0answers
19 views

Linear regression with trimmed data

I would like to know how experts deal with real data. Even if statistical text books uses real data I'm always surprised how good the real data are and at the end of the exercises the residuals are ...
5
votes
1answer
78 views

Should the correlation PCA projection be computed on original or normalized samples?

Suppose we compute the correlation PCA of a dataset $X$ (with $m$ variables and $n$ observations) by first normalizing the input variables. That is: mean -> 0 and standard deviation -> 1. Let us ...
0
votes
0answers
20 views

I have 5 models and each one consists of two 95 % confidence intervals, one for each axes, how can I plot these in R (ggplot2)? [closed]

I have 5 models and each one consists of two 95 % confidence intervals, one for each axe, how can I find and plot these in R (ggplot2)? I tried to put the data in an ascending order and find them ...
0
votes
0answers
26 views

How to predict when using normalized data?

So, I am taking this course on machine learning by Andrew Ng. Wanted to write my own linear regression program. Everything is fine. I mean normalize data, and run linear regression. Now I'm left with ...
0
votes
1answer
18 views

Include both linear and non-linear dependency of the same variable in a multi-variate analysis

I am implementing a multi-variate analysis using 5 covariates. My model looks like this: lm1<-lm(Y ~ (T(A) +A + B + C + D + E)^2, data=data) where T(A) is a ...
2
votes
1answer
42 views

Variable standardization / scaling for PCA when all dimensions already have same scale [duplicate]

Often when PCA is performed on exam results where all variables (dimensions) have the same $0$ to $100$ scale, scaling is none the less applied. For different scales I can see the purpose of it, but ...
0
votes
1answer
20 views

How to interpret a log transformed (x+c)? [duplicate]

I need to use log-log regression and because I have lots of zero values I tried to add a very small constant c=8E-12 to x and it works pretty good. Xs are very small probabilities. lnY= a + b ln ...
0
votes
0answers
35 views

More effective seasonal adjustment to time series data?

I am trying to predict surface temperature using solar energy. I have 3650 daily averages for both variables. The plots of both are below: I attempt to seasonally adjust with a periodic ...
1
vote
2answers
45 views

What transformation should I use for a bimodal distribution?

I have some bimodal data like the one generated down (R language), and I don't know how to transform it to have a normal distribution or homoscedasticity. I'm running a linear discriminant analysis ...
3
votes
1answer
63 views

Does the IQR or standard deviation change when scaling or shifting non-normal data?

I'm aware that scaling or shifting data that is normal or almost perfectly normal will not significantly change the standard deviation. Through practice I've that this is not the case with non-normal ...
0
votes
0answers
19 views

Data does not have a normal distribution but has homogeneity of variance

I'm trying undertake some statistics for my masters thesis but I'm having some problem with my data not being normally distributed. I've essentially got 3 factors, one with 2 levels, one with 3 levels ...
2
votes
1answer
51 views

Does using difference transformation lead to bias? (Levels vs differences regression)

Consider the model estimated in levels (also assume this is the true population model): $$y_t = x_t\beta + e_t$$ As usual we have the dependent variable $y$, independent $x$, the error term $e$, and ...
1
vote
1answer
35 views

How can I combine nominal with ordinal data to build a unique variable?

I performed an Interview with 44 questions Protocol. The structure of questions is based on 18 variables. Major variables are coming from theory. Every major variable consists of 3,4 or more question ...
0
votes
1answer
29 views

Select and aggregate time series based on selection information of a second dataset

General problem: I have two datasets in r and I do not know how I can calculate information across groups of time series in one dataset based on selection-information of another dataset. The details: ...
1
vote
0answers
19 views

Which model(s) are appropriate for this kind of data

So, I tried to implement a model on some data. The dependent variable is a ratio that can get higher than 1, is lower bounded by zero and, seeing figure 1, is left skewed.Thus, a logit regression is ...
0
votes
0answers
7 views

Transform Variable R [migrated]

I have a data frame as shown below: ...
1
vote
0answers
9 views

Compare between median/IQR and reference mean/SD

I have a small set of non-normally distributed measurement data (Kolmogorov-Smirnov rejected similarity to a normal distribution) and a reference value from a large population (n=120) of healthy ...
1
vote
2answers
35 views

Can trees or random forests learn ratios

This is a question about feature engineering for decision trees/random forests. Given two continuous variables X1 and X2, is it ...
0
votes
0answers
30 views

Can I use the Xie-Beni index to validate data transformation parameters in fuzzy c-means clustering?

I am using fuzzy c-means algorithm to cluster my data in various feature spaces and the results differ depending on what kind of transformation I perform on my raw data. I want to know if using the ...
1
vote
1answer
21 views

Transforming Power and Exponential Functions

Suppose two variables x and y have no linear correlation. If we transform the data by replacing each y value with its base-10 logarithm, then will x and log $y$ also have zero correlation? In ...
0
votes
0answers
10 views

Log transformation in SPSS for percentage dependent var

indep_avg dep_% Frequency 3.4 5.6 67 5.6 2.5 96 3.2 6.3 23 I have some aggregated data. The Independent ...
0
votes
0answers
31 views

How to transform feature with peak at zero to normal distribution?

I have a feature in my dataset which has lot of zero values, i.e. a big peak at zero (the zeroes are valid and valuable information). The histogram is the following: I want to transform all my ...
0
votes
1answer
12 views

Correlation of levels vs. differences vs. percents

Sometimes, I have seen people using correlation of levels, correlation of differences and also correlation of percent changes. I understand these answer different questions. For example, for "what is ...
0
votes
1answer
41 views

Transformation of a Skewed Composite Outcome made up of 2 Z-scores?

I am running a repeated measures mixed model. For my outcome variable, I would like to sum 2 continuous variables, which consequently are both Z standardized in order to do so. However, my outcome ...
1
vote
2answers
49 views

Time Series Seasonality

how to identify whether seasonality is additive or multiplicative in a time series? Using Plots or any statistical tests?
1
vote
0answers
16 views

Relationship between weighted $r^2$, and $r^2$ of transformed data

When regressing heteroscedastic data, recommended practice is to either transform the data to remove heteroscedasticity weight the data to compensate So let's say we have some data $ y = x + ...
2
votes
0answers
28 views

how to model longitudinal big data?

Traditionally we use mixed model to model longitudinal data, i.e. data like: ...
0
votes
1answer
35 views

Comparing linear regression results with different log transformations

I am trying to compare the results of three studies. All use linear regression with continuous predictors/independent variables and outcomes/dependent variables. The predictors (level of a chemical in ...
0
votes
0answers
20 views

Analyzing time-use data + questionnaire

I'm doing research among students at a major institution. My initial hypothesis is that students are reading less, especially books and print, while their time is increasingly spent interacting with ...
0
votes
0answers
18 views

What's the motivation of data transformation (like Box-Cox) to skew data?

What's the motivation of data transformation (like Box-Cox) to skew data? Suppose I am only interested in predictive power and non-linear model. Does it help to do data transformation (like Box-Cox) ...
0
votes
0answers
17 views

Transformation and standardization for discrete features?

I have a dataset consisting of continuous and discrete features (predictors). For the discrete features, I have integer values (e.g. 0, 5, 20, 30 etc.). Of course, for the continuous features I could ...
0
votes
0answers
38 views

Data Transformations

I have a study utilizing criterion variables, against which my dependent variable are being compared to. It appears that some are positively skewed and some are negatively skewed... this is ...
0
votes
0answers
24 views

Trasformation to Positive or Negative Quadrant Dependent Random Variables

Can we transform two statistically dependent random variables $X_1,X_2$ such that the covariance $Cov(X_1,X_2) \geq 0$? Can we transform $X_1,X_2$ such that $Cov(f(X_1), g(X_2))\ge0$ for all real ...
1
vote
1answer
52 views

Skewed Distributions: Transforming or Non-parametric?

Say that I have two distributions. Both are very skewed distributions that don't seem to fit any distribution I know well. Should I turn to a non-parametric (distributionless) test or transform the ...
0
votes
0answers
14 views

Standardization of all variables and weighting of some variables for clustering

I am trying to segment a database based on certain variables. I understand that before i do start clustering, i should standardize all the variables. This can be done by Z score or other methods. ...
1
vote
0answers
11 views

Variability of the reference data and its effect on the transformed values

I conducted a study to compare two different tests at two different ages (children). Based on the previous literature we had hoped that the scores should be better (low score) with age but this was ...
1
vote
3answers
47 views

Transformation of specific data

I need help with data transformation. In the picture below the upper left picture shows the histogram of the variable V6. Because it is so right-skewed I tried 3 forms of transformation but none of ...
0
votes
0answers
39 views

Data transformation with chemical data

I'm relatively new to R and I have a question on some data transformation. The actual question is I want to test the data for normality because I'm supposed to conduct a PCA on the data set. Now I ...
2
votes
1answer
64 views

How is the box cox transformation valid?

The box cox transformation transforms our data into a normal distribution. How is that even a proper technique? What if our data didn't come from a normal distribution? How could someone just blindly ...
0
votes
1answer
21 views

Are we using transformed or original series to fit a time series model?

I would like to know whether one can use the transformed series to fit a model or not. For example, fitsereies=auto.arima(differenced_series) or ...
1
vote
0answers
15 views

How do I back transform predicted values when multiple transformations are present in multiple regression?

Suppose two predictors are desired for a multiple linear regression. Upon reviewing scatter plots, the first requires a power transformation, log(y)= b0 + b1log(x), and thus the back transformation of ...