Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

learn more… | top users | synonyms (2)

2
votes
0answers
11 views

Are log difference time series models better than growth rates?

Often I see authors estimate a "log difference" model, e.g. $\log (y_t)-\log(y_{t-1}) = \log(y_t/y_{t-1}) = \alpha + \beta x_t$ I agree this is appropriate to relate $x_t$ to a percentage change in $...
0
votes
0answers
13 views

Classification modeling on data with several levels of grouping [on hold]

I need to create a classification model on data with several levels of grouping. The data looks like the below graphic. I have TV data where in for every house I have several TV devices, each TV ...
0
votes
0answers
24 views

drawbacks of woe (weight-of-evidence) transformation

I have read a few posts about woe transformation. It seems that most are about the pros. That makes me think: if woe is that effective in logistic regression, then there is no reason for not doing it. ...
0
votes
0answers
24 views

transform on a distribution

I have a pseudo-normal distribution with mean 0 and sd 0.03. Is there a way to transform this distribution such as values above or under +/- one standard deviation are pushed towards 1/-1 while values ...
3
votes
1answer
30 views

How to make sense of non-linear data transformations? What conclusions drawn can you apply to original data?

In stats class, the professor talked about the interest of transforming skewed data sets to make them more "normal". From what I've understood so far, the idea is that the normal curve has nice ...
6
votes
3answers
183 views
+150

If my goal is to test the absolute change of the ratios, can I compare the ratios directly without log transformation?

Ratios (e.g. $Z$=$Y$/$X$) are frequently used (e.g. fold-changes in mRNA or protein expression, body mass index [BMI], etc.). Many people advise that variables coded as ratios (e.g. fold-change) ...
0
votes
0answers
12 views

Question regarding variables transformation for GLMM's

I have a response variable (binary:1/0) and a set of explanatory variables, with different units: some have values in %, others in meters (distances, altitude and differences between elevation pixels),...
1
vote
0answers
18 views

How to separate two classes when the features values predicting them are so similar ?

What should be my approach. I got 13 principal components from 21 numerical features. The 13 features have a gaussian distribution. The plot below is between the top two components. Should I clean the ...
0
votes
2answers
74 views

Is log transforming square root transformed data a legitimate data transformation?

Is it legitimate to do a "double transformation" on data? Specifically, log transforming data which has already been square root transformed, or conversely, square root transforming log transformed ...
0
votes
0answers
47 views

What is the most effective way to get a baseline rating prediction from netflix rating data?

I am doing an assignment where I am working with some of the Netflix Challenge data. I have a database that maps movie ids to customer ratings of that movie and a database of customers mapped to ...
3
votes
1answer
245 views

Regression when response variable is a function

I have a set of data $(X_i,Y_i)$, $i=1,\ldots,n$ where $X$ and $Y$ are supposed to satisfy the following equation $$ y = \beta_0(1+x^2)^{\beta_1},\quad x>0, \quad\quad (1) $$ I am interested in ...
0
votes
1answer
57 views

Streaming data input in artificial neural network

Suppose we have continuous stream of data which length we cannot predict and discretize. Is there a type of neural network that can hold this stream and makes output based on the information stored in ...
3
votes
0answers
23 views

Sensitivity analysis on constants: logit transformation for ANOVA

I have categorical looking-time data (looks to visually presented items A, B, C and D over a number of seconds), containing 0s and 1s. I want to compare groups (adults, children; n=~30 for each group) ...
1
vote
1answer
38 views

Transformation of independent variables with negative values

I have a bunch of independent variables which are skewed and have negative and zero values. I am seeing a lot of suggestions of using cube root as a transformation. I want to ask what is the harm in ...
1
vote
1answer
48 views

How to deal with skewness in IV

I'm Building a logistic regression model and one of my independent variables is very skewed at zero. How do you suggest that i deal with this situation?
1
vote
0answers
18 views

Interpretation of LOGIT transformed predictor when outcome variable is LOG transform

I have a linear mixed effect model in which the dependent variable is a log-transformed frequency of livestock predation predation, and there are three predictor ...
1
vote
1answer
48 views

power regression when the power is a variable

I have this function : $y = x^\alpha$ using log: $\ln(y) = \alpha\,\ln(x)$ Now, $\alpha$ itself can be decomposed and considered as a function of two variables $w_1, w_2$. We have: $\alpha = \...
1
vote
0answers
19 views

Robust estimation of multivariate reference bands

I have a subjectivly healthy population of approx 1200 individuals with three measurements on the continous scale, we can call them y1, y2 and y3. All of these are strongly related to age in a non-...
0
votes
0answers
40 views

time stamp as input variable for regression (feature extraction)

I am working on web logs and have a time-stamp variable in the format dd-mm-yyyy hh-mm-ss. I have earlier worked on date variable and found that best way to extract feature from date is to create ...
0
votes
0answers
11 views

What are some of the more popular variable transformations and why/when are they used (to handle what types of distribution problems)?

Like the title states, I'm interested in learning about the more popular data transformation techniques. I know the internet is abound in this information, but I'd like to hear from those working ...
4
votes
0answers
47 views

Categorical PCA: Merge categories based on Transformation Plots?

A tutorial on categorical pca (CATPCA) (Linting et al. 2012) explains that a decision to merge categories of an ordinal variable can be made based on the category quantification ("none of the ...
0
votes
0answers
6 views

How to segment hours of audio for speech recognition?

I have 36 hours of speech data along with transcription. I'm planning to have 7 second audio segments, because I don't know any better. Suggestions are welcome. These segments will be passed through ...
1
vote
1answer
23 views

Interpreting the effect of predictor variables on outcome variable when the latter is logit transformed

I apologize if this question is very simple but I have found a lot of information on how to interpret log transformed variables (http://www.ats.ucla.edu/stat/mult_pkg/faq/general/...
0
votes
0answers
13 views

Imputing skewed variable?

I have a data set with missing values in the IVs. I intend to use MI and in particular PMM for the numerical variables. One of them is very skewed and has many 0s so I can't log tranform it. My ...
0
votes
0answers
13 views

Shouldn't A/B be correlated with B/A and A*B?

In data mining problems it is common to do variable transformation, sometimes doing pairwise combinations like A/(B+1), B/(A+1) or A*B. Now, let's say after performing transformations the two best ...
1
vote
1answer
64 views

log transformation logistic regression

I have a logistic regression in which i transformed geographical distance measured in km using a natural log. I've have run the regression, and now i am having trouble how to interpret the findings. ...
1
vote
0answers
14 views

Whitening Transformation with Autoregressive Model

I am new to the topic of whitening transformation. In financial time series studies on long memory in data, I have seen that researchers apply an AR(p) model to detrended return series in order to ...
2
votes
1answer
91 views

Impossible to bring to normality

I have given a data and I have to check the data if it's normally distributed and if not I have to transform the data into normality. I had done shapiro-wilk normality test and p-value is clearly ...
0
votes
0answers
7 views

Model name for regression with square root transformation on the response variable

Consider different transformations on the response variable. If log transformation is used: $$\log{Y} = \alpha + \beta X + \epsilon$$ The model is called log-linear model or semi-log model. My ...
0
votes
0answers
26 views

skewness and hypothesis testing (t-test and anova)

Some of my variables are heavily positive skewed (left skewed). With log transformation, some are closer to normal distribution, but some are still positively skewed, though not that bad before log ...
4
votes
1answer
64 views

How does the support of a continuous random variable change under transformations?

Let $X_1$, $X_2$ have a joint pdf and support set $S$. Suppose random variables $Y_1$, $Y_2$ are given by $Y_1=u_1(X_1,X_2)$, $Y_2=u_2(X_1,X_2)$ where the functions define one to one transformation ...
1
vote
1answer
30 views

Confused about PCA transformation vectors

I'm trying to get the intuition of how PCA works. So far I understood that: I start from the input matrix $X = [X_{1},...,X_{p}]$ where each $X_{i}$ is composed by $n$ elements that are the $n$ ...
0
votes
0answers
17 views

BoxTidwell test for logistic regression

I'm curious as to how BoxTidwell works in R. The page for the package itself seems to lack descriptions. I have a logistic regression with many numerical and categorical predictors. Every time I use ...
1
vote
2answers
104 views

Best way to optimize MAPE

The MAPE is a metric that can be used for regression problems : $$\mbox{MAPE} = \frac{1}{n}\sum_{t=1}^n \left|\frac{A_t-F_t}{A_t}\right|$$ Where $A$ represents the actual value and $F$ the the ...
0
votes
1answer
30 views

Can I iteratively transform a variable with log10 until it fits a linear model?

I have a response variable, $Z$, for which I'm trying to make a linear model. Here are some of the fit diagnostics plots: From the fan-like shape of the residual-vs-predicted value plots, I ...
1
vote
1answer
20 views

what's the difference between weighting and transforming?

In regression analysis, what's the difference between weighting and transforming when it comes to spreading residuals? For example, we need to weight the model $y=ax+b$ by $1/x$, isn't it just ...
0
votes
2answers
20 views

Non-linear transformation to increase separability between clusters

I want to do a classification on PC scores. I have a 400 dimensional matrix, e.g. 2000*400 (2000 number of samples and 400 dimensions). I fist apply PCA on it and take it to 3D, i.e. 2000*3. There are ...
1
vote
1answer
23 views

Understanding output of powerTransform

In the car package, we have the function powerTransform which transforms variables in a regression equation to make the ...
3
votes
1answer
132 views

Why do deep learning practitioners forego PCA for ZCA?

I have an understanding of PCA and ZCA, read a similar question on the subject which, unfortunately, does not have the specific answer to my question. I understand the benefits of data whitening: ...
3
votes
2answers
77 views

What to do with data that are bimodal at two tails of the distribution?

I am in a weird position where I prespecified a plan to use linear regression to analyze my data, and stated I would use transformations to address any assumption violations. I'm pretty certain my ...
0
votes
1answer
22 views

Is Woe Transformation is required for all variables in a logistic model

i am trying to build a logistic regression model. I have a doubt regarding using woe (weight of evidence) transformed variables. I wanted to know if it is ok to use a few woe transformed variables and ...
1
vote
1answer
28 views

In a class with multiple teachers, how can I transform student scores based on their teacher's average compared to the population average?

It has been ages since I've taken any statistics courses, and I have found myself in the following situation: I am in charge of a university course with about 400 students and 10 assessors. There ...
1
vote
1answer
34 views

transformation for non-constant variance?

This is from my textbook I don't understand what does the content in red mean, for example, what does $y^2_i \infty \sigma_i$ mean? How can we tell the relationship between $y^2_i$ and $\sigma_i$ ...
4
votes
2answers
205 views

R: Box-plot on log scale vs. log-transforming *then* creating box-plot: Don't get same result

In the boxplot() function in R, there exists the log = argument for specifying whether or not an axis should be on the log scale....
0
votes
0answers
16 views

How to run 2 Way ANOVA on SPSS with data that is not normal distributed?

I am about to run 2x2 ANOVA on my data, but then I realized that my data is not normal. I have tried to do data transformation like Log10 and Ln, but the data is still not normal. The data has ...
2
votes
0answers
37 views

Is/are there any threshold value(s) to determine to see if PCA is useful at all, specially for high dimensional data?

Apologies if this is a naïve question, but it's not so naïve to me! Let's first assume we have 2D data which are perfectly linear but not along the x- or y-axis. PCA will rotate it so that it becomes ...
0
votes
0answers
26 views

Google correlate - data transformation / switching positive / negative correlation

Google correlate Takes your data and then searches for positive correlations in search terms. From my understanding it returns search terms that positively correlate to your data. For example if ...
1
vote
0answers
99 views

Data structure for rare event predictions in temporal domains

I am a beginner in rare event modeling. I am working on predicting modem failures within a network where failures occur approximately 3% of the time. Currently my data is structured as follows: ...
0
votes
1answer
27 views

Transformation from skewed to symmetric distribution

Let us consider a positive valued random variable $X$ which is following a positively skewed probability distribution. Is it possible to a get a function $f$ (one-to-one) for which $f(X)$ follow a ...
1
vote
0answers
18 views

Normalizing skewness with the Power or Box-Cox Transformation

Suppose I have a random sample drawn from an arbitrary strictly positive continuous distribution. Suppose moreover that I want to use the Box-Cox transform to zero out the skewness. Is there an ...