# Tagged Questions

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

26 views

### Is/are there any threshold value(s) to determine to see if PCA is useful at all, specially for high dimensional data?

Apologies if this is a naïve question, but it's not so naïve to me! Let's first assume we have 2D data which are perfectly linear but not along the x- or y-axis. PCA will rotate it so that it becomes ...
15 views

### Google correlate - data transformation / switching positive / negative correlation

Google correlate Takes your data and then searches for positive correlations in search terms. From my understanding it returns search terms that positively correlate to your data. For example if ...
24 views

### Data structure for rare event predictions in temporal domains

I am a beginner in rare event modeling. I am working on predicting modem failures within a network where failures occur approximately 3% of the time. Currently my data is structured as follows: ...
22 views

### Transformation from skewed to symmetric distribution

Let us consider a positive valued random variable $X$ which is following a positively skewed probability distribution. Is it possible to a get a function $f$ (one-to-one) for which $f(X)$ follow a ...
10 views

### Normalizing skewness with the Power or Box-Cox Transformation

Suppose I have a random sample drawn from an arbitrary strictly positive continuous distribution. Suppose moreover that I want to use the Box-Cox transform to zero out the skewness. Is there an ...
32 views

### Variable transformation: from angle to coordinates

I have the following problem: Let $\theta \in [0,2\pi)$ be an angle and let $f(\theta)$ be a function such that $\int_{0}^{2\pi}f(\theta)d \theta=1$. Now let consider the following transformations: ...
4 views

### How to use time dependent covariates with cox regression in R

I don't know how to generate time dependent covariates in R for use cox regression. I know you need to reorganize your dataset into intervals between event times. This I believe I can do with the ...
27 views

### Are parametric tests on rank transformed data equivalent to non-parametric test on raw data?

Many non-parametric tests are identical to their parametric equivalent on ranked data. At least, that's what I learned from this blog post on Friedman's test and skimming this 1981 article.. This ...
17 views

### How to perform quantile transformation with missing values?

Given are an input vector $I$ with missing values and a target/reference distribution ${T}$. For example: $I$: 0.215 NA 0.103 0.649 0.057 0.292 NA 0.433 0.521 NA $T$: -0.996 -0.606 -0.394 -0.090 ...
19 views

### Linear regression with trimmed data

I would like to know how experts deal with real data. Even if statistical text books uses real data I'm always surprised how good the real data are and at the end of the exercises the residuals are ...
78 views

### Should the correlation PCA projection be computed on original or normalized samples?

Suppose we compute the correlation PCA of a dataset $X$ (with $m$ variables and $n$ observations) by first normalizing the input variables. That is: mean -> 0 and standard deviation -> 1. Let us ...
20 views

### I have 5 models and each one consists of two 95 % confidence intervals, one for each axes, how can I plot these in R (ggplot2)? [closed]

I have 5 models and each one consists of two 95 % confidence intervals, one for each axe, how can I find and plot these in R (ggplot2)? I tried to put the data in an ascending order and find them ...
26 views

### How to predict when using normalized data?

So, I am taking this course on machine learning by Andrew Ng. Wanted to write my own linear regression program. Everything is fine. I mean normalize data, and run linear regression. Now I'm left with ...
18 views

### Include both linear and non-linear dependency of the same variable in a multi-variate analysis

I am implementing a multi-variate analysis using 5 covariates. My model looks like this: lm1<-lm(Y ~ (T(A) +A + B + C + D + E)^2, data=data) where T(A) is a ...
42 views

### Variable standardization / scaling for PCA when all dimensions already have same scale [duplicate]

Often when PCA is performed on exam results where all variables (dimensions) have the same $0$ to $100$ scale, scaling is none the less applied. For different scales I can see the purpose of it, but ...
20 views

### How to interpret a log transformed (x+c)? [duplicate]

I need to use log-log regression and because I have lots of zero values I tried to add a very small constant c=8E-12 to x and it works pretty good. Xs are very small probabilities. lnY= a + b ln ...
35 views

### More effective seasonal adjustment to time series data?

I am trying to predict surface temperature using solar energy. I have 3650 daily averages for both variables. The plots of both are below: I attempt to seasonally adjust with a periodic ...
45 views

### What transformation should I use for a bimodal distribution?

I have some bimodal data like the one generated down (R language), and I don't know how to transform it to have a normal distribution or homoscedasticity. I'm running a linear discriminant analysis ...
63 views

### Does the IQR or standard deviation change when scaling or shifting non-normal data?

I'm aware that scaling or shifting data that is normal or almost perfectly normal will not significantly change the standard deviation. Through practice I've that this is not the case with non-normal ...
19 views

### Data does not have a normal distribution but has homogeneity of variance

I'm trying undertake some statistics for my masters thesis but I'm having some problem with my data not being normally distributed. I've essentially got 3 factors, one with 2 levels, one with 3 levels ...
51 views

### Does using difference transformation lead to bias? (Levels vs differences regression)

Consider the model estimated in levels (also assume this is the true population model): $$y_t = x_t\beta + e_t$$ As usual we have the dependent variable $y$, independent $x$, the error term $e$, and ...
35 views

### How can I combine nominal with ordinal data to build a unique variable?

I performed an Interview with 44 questions Protocol. The structure of questions is based on 18 variables. Major variables are coming from theory. Every major variable consists of 3,4 or more question ...
29 views

### Select and aggregate time series based on selection information of a second dataset

General problem: I have two datasets in r and I do not know how I can calculate information across groups of time series in one dataset based on selection-information of another dataset. The details: ...
19 views

### Which model(s) are appropriate for this kind of data

So, I tried to implement a model on some data. The dependent variable is a ratio that can get higher than 1, is lower bounded by zero and, seeing figure 1, is left skewed.Thus, a logit regression is ...
7 views

### Transform Variable R [migrated]

I have a data frame as shown below: ...
9 views

### Compare between median/IQR and reference mean/SD

I have a small set of non-normally distributed measurement data (Kolmogorov-Smirnov rejected similarity to a normal distribution) and a reference value from a large population (n=120) of healthy ...
35 views

### Can trees or random forests learn ratios

This is a question about feature engineering for decision trees/random forests. Given two continuous variables X1 and X2, is it ...
30 views

### Can I use the Xie-Beni index to validate data transformation parameters in fuzzy c-means clustering?

I am using fuzzy c-means algorithm to cluster my data in various feature spaces and the results differ depending on what kind of transformation I perform on my raw data. I want to know if using the ...
21 views

### Transforming Power and Exponential Functions

Suppose two variables x and y have no linear correlation. If we transform the data by replacing each y value with its base-10 logarithm, then will x and log $y$ also have zero correlation? In ...
10 views

### Log transformation in SPSS for percentage dependent var

indep_avg dep_% Frequency 3.4 5.6 67 5.6 2.5 96 3.2 6.3 23 I have some aggregated data. The Independent ...
31 views

### How to transform feature with peak at zero to normal distribution?

I have a feature in my dataset which has lot of zero values, i.e. a big peak at zero (the zeroes are valid and valuable information). The histogram is the following: I want to transform all my ...
12 views

### Correlation of levels vs. differences vs. percents

Sometimes, I have seen people using correlation of levels, correlation of differences and also correlation of percent changes. I understand these answer different questions. For example, for "what is ...
41 views

### Transformation of a Skewed Composite Outcome made up of 2 Z-scores?

I am running a repeated measures mixed model. For my outcome variable, I would like to sum 2 continuous variables, which consequently are both Z standardized in order to do so. However, my outcome ...
49 views

### Time Series Seasonality

how to identify whether seasonality is additive or multiplicative in a time series? Using Plots or any statistical tests?
16 views

52 views

### Skewed Distributions: Transforming or Non-parametric?

Say that I have two distributions. Both are very skewed distributions that don't seem to fit any distribution I know well. Should I turn to a non-parametric (distributionless) test or transform the ...
14 views

### Standardization of all variables and weighting of some variables for clustering

I am trying to segment a database based on certain variables. I understand that before i do start clustering, i should standardize all the variables. This can be done by Z score or other methods. ...
11 views

### Variability of the reference data and its effect on the transformed values

I conducted a study to compare two different tests at two different ages (children). Based on the previous literature we had hoped that the scores should be better (low score) with age but this was ...
47 views

### Transformation of specific data

I need help with data transformation. In the picture below the upper left picture shows the histogram of the variable V6. Because it is so right-skewed I tried 3 forms of transformation but none of ...
39 views

### Data transformation with chemical data

I'm relatively new to R and I have a question on some data transformation. The actual question is I want to test the data for normality because I'm supposed to conduct a PCA on the data set. Now I ...
64 views

### How is the box cox transformation valid?

The box cox transformation transforms our data into a normal distribution. How is that even a proper technique? What if our data didn't come from a normal distribution? How could someone just blindly ...