Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

learn more… | top users | synonyms (2)

2
votes
1answer
27 views

What's the optimal way to encode a 'month' feature?

What's the optimal way to encode a 'month' feature? A single integer value or 12 binary values don't quite grasp the concept of modulo distance... Say I want to train an SVM for a certain task and ...
-1
votes
1answer
17 views

Box Cox transformed my data, now how to use it in my mixed model? [on hold]

Sorry for the (most likely) simple question, but I have Box Cox transformed my data in SAS, but I am unsure how to use the transformed data in my mixed model. Do I output it to a new file, and use ...
3
votes
1answer
43 views

Aspect data in linear regression

I have a dataset of various ecological variables on which I want to run linear regression. The variables are continuous, but also include aspect data (sun exposure), in grades. My problem is that the ...
1
vote
1answer
21 views

What methods can be used to transform data?

I am solving a binary classification problem with 4 predictor variables. The variables didn't seem to be linearly separable. I have used Neural Networks and Kernel SVM which work and give desired ...
0
votes
0answers
18 views

taking the log of (Only) some variables in a regression

My question is as follows: Ratio=Ratios + log (numbers) +dummy variables + volatility I have this type of regression in a paper published by the Federal Reserve Bank.Can someone tell me why we took ...
4
votes
3answers
76 views

What is the reason the $\log$ transformation is used with right-skewed distributions?

I once heard that log transformation is the most popular one for right-skewed distributions in linear regression or quantile regression I would like to know is there any reason underlying this ...
0
votes
1answer
41 views

Using F-tests for variance in non-normal populations

I'm fairly new to stats, so please excuse me if this problem is hopelessly elementary or misinformed. Basically, I'm wondering if you can help me understand whether I'm using the F-Test for variance ...
0
votes
0answers
11 views

Compute how much the one percent wealthier concentrates with negative data

I want to compute, using the Survey of Consumer Finances (SCF) database, how much the one percent wealthier concentrates. The variable I'm using to measure wealth is the "networth". The problem is ...
0
votes
1answer
30 views

Interpreting Log-Transformed Percentages in OLS

In a log-log model, such as $\log(y) = b_0 + b_1 \log(x)$, I know that with OLS the standard interpretation is a "1% increase in x is associated with a $b_1$% increase in y." I have three related ...
2
votes
3answers
96 views

Best way to turn a date into a numerical feature?

I have a fairly large dataset with a few fields containing time-related data. This data comes in various shapes and sizes, but most of it can be parsed and rephrased in more appropriate formats for ...
1
vote
1answer
52 views

Data transformation and confidence intervals for mean difference

I have a between-group independent variable with two level (A and B) and a dependent variable Y that I transformed in order to normalize the distribution of the residual. I used a Box Cox ...
1
vote
2answers
56 views

Appropriate way to treat [0,1]-distributed variables in HLM

Brief intro: I'm not really sure how to appropriately treat the dependent variables in a set of hierarchical linear models that I'm trying to run. In my models, Level 1 units are children and Level 2 ...
2
votes
2answers
73 views

Best transformation for sinuous data sets?

I am analysing annual behavioural patterns and am currently scratching my head over how to best transform data to test for correlation where my independent variable is the moon phase (given as ...
0
votes
0answers
14 views

Questions about $R^2$, VIFs and very non normal input variables

I have been working with a small part of my dataset trying to eliminate variables and do some micro models. When analysing my micro set I initially found a few high correlations with inputs (0.95 ...
1
vote
0answers
12 views

Combine variables that are extremely lightly populated?

A similar question to my other question about mixed distributions. Here i have quite a few variables that are populated to less than 5%, many are even populated to less than 1% this 1% would represent ...
0
votes
1answer
34 views

Collection of continuous variables with >70% more zeros

This is dataset that is going to be data mined for factors that affect an output that of interest A large Part of my dataset (150 of 300 potential inputs) has a heavy skew of Zero values. usually ...
0
votes
0answers
27 views

Transforming data with large negative and large positive values

The data I'm trying to analyze are the quadratic estimates from a quadratic fit to a curve. Most of the data vary between -.15 and .15. However, I have outliers in both directions up to things like ...
0
votes
0answers
16 views

What's the best way to transform this vector to be normal-like?

Assume I have an outcome like this: ...
2
votes
0answers
18 views

How to transform an outcome which censored from both sides?

I'm trying to fit a tobit model for a response outcome where both sides were censored and the distribution is heavily skewed, even for the none-censored ...
3
votes
1answer
31 views

Intercept from standardized coefficients in logistic regression

I have fit a logistic regression model with original y and standardized x variables. Slope coefficients can be easily converted back to their original scale by $\beta^*_j/\sigma_{x_j}$ where ...
1
vote
0answers
45 views

Do you need justification to transform non-normal data

First, a bit of background: I'm currently working on a project at work that produces variable width data. During a operation qualification I collected a bunch of data at low, nominal, and high ...
6
votes
2answers
92 views

Does Box-Cox parameter estimation count towards parameters for AIC?

Suppose I have a regression model with e.g. 2 parameters $y = ax + b$ But the data are non-normal so before regressing I transform both sides with Box-Cox estimation. Thus I get two Box-Cox ...
1
vote
3answers
57 views

Correcting data for heteroscedasticity in a regression model

I applied OLS on a regression model that looks as follows: $$ y = b_0 + b_1x_1 + b_2x_2 $$ and found that signs of heteroscedasticity. In an econometrics text book, I found that I can divide each ...
4
votes
0answers
38 views

Algorithms for data symmetrization

There are statistical methods (e.g. by Box-Cox or Yeo-Johnson, see references below) to automatically bring data vectors as close as possible to symmetry/normality using optimal power transformations. ...
9
votes
1answer
252 views

Are normally distributed X and Y more likely to result in normally distributed residuals?

Here the misinterpretation of the assumption of normality in linear regression is discussed (that the 'normality' refers the the X and/or Y rather than the residuals), and the poster asks if it is ...
0
votes
0answers
48 views

Dealing with linear dependent variables

I have a large dataset with many subject each with responses from a consecutive year going back 10 years (ie 100,000 persons per year (not necessarily 10 data points per person as they may not have ...
1
vote
1answer
45 views

Standardizing feature vectors for regression

Suppose I have a data set with the following structure: Each row of the data set indexes a town. The first column/feature variable is the total population while the other feature variables include ...
1
vote
1answer
50 views

Fusing Aggregate (Macro) and Micro Data?

I have a microdata recipient survey and a macro (aggregate) donor. How can I fuse the (binary categorical) data? The Statistical Matching techniques/software I'm familiar with are micro-to-micro. ...
1
vote
1answer
14 views

Transform a non-monotonic value before decision tree (concrete example)?

Newbie question here. I am building a toy decision tree to differentiate personal names from business, government, or organizational names, like: AAA ENTERPRISES LLC DBA AAA BBB SERVICE SMITH ...
0
votes
0answers
24 views

Using transformed variables in SEM (using Amos)

Typically when I'm using Amos graphics I access my SPSS datafile and use individual items from a survey (observed variables) to make up my constructs (latent variables). In my current study, the ...
10
votes
4answers
287 views

A regression model whose response variable is the day of year that an annual event (usually) occurs

In this particular case I'm referring to the day on which a lake freezes. This "ice-on" date only occurs once a year, but sometimes it doesn't occur at all (if the winter is warm). So on one year the ...
1
vote
1answer
46 views

Interpreting transformed dependent and independent variables [duplicate]

How would I interpret a transformed dependent variable (4th root) with some of its predictor variables transformed as well? In our study, we transformed our dependent variable to 4th root, ...
1
vote
1answer
61 views

Data tranformation in panel data

I am doing panel data analysis and some of my variables have high kurtosis. I am not sure whether I have to transform these variables. I have tried to delete outliers but one of the variables still ...
0
votes
1answer
31 views

Grouping cases on certain criterion

You experts have been helpful so many times that I read others' threads, but this time I have to ask a new question. Long story short I need to group all my cases into 4 categories. Could someone ...
0
votes
1answer
54 views

Transformation for negative skewness data

My analysis involved some behavioral data on swine. One measure we had was standing time (min) for pigs using accelerometers. Using SAS, I checked for normality, and results showed data to be ...
0
votes
1answer
36 views

Using inverse of cube In linear model

What's the formula for a equation that can produce the continuum from the red to green lines in this graph below? I can easily get anywhere from the green line to the blue with $$y = B_0 + B_1x + ...
0
votes
1answer
90 views

Regression and transforming variable with square root

I have a variable that can range from -100 to +100. When the number is -100, the output from this transformation should be something like 10. For +100 it should be around -10, and for 0 it should be ...
1
vote
1answer
49 views

Data Fusion/Statistical Matching: Match Keys & Algorithms?

I need to fuse multiple surveys using statistical matching, and would like to comply with any best practices. I have studied multiple documents but -- despite the plethora of information -- it's ...
1
vote
0answers
16 views

How to create a model with dummy coding to put into log regression

How do I use information from a chi square table to build a statistical model. To be specific, I am using Likert-style survey information that measures student engagement. I would like to see if ...
1
vote
1answer
97 views

Transform odds ration (OR) in the context of multivariate meta-analysis in R (metafor package)

I would like to conduct a multivariate meta-analysis (multiple treatment arm meta analysis) comparing the effect of different drugs. My outcome measure is discrete and describes the number of ...
0
votes
0answers
16 views

Replacing categorical variables with historic response rate

In Linoff and Berry's "Data Mining Techniques" they mention reducing the number of categorical variables in a classification model by replacing the variable with the historic response rate. "When ...
1
vote
1answer
64 views

What does (SAS) Enterprise Miner's “Bucket Transformation” do?

In SAS's Enterprise Miner there is a "Bucket Transformation" which from what I can tell, is a term that is unique to Enterprise Miner. It takes a continuous variable and groups it to become an ...
0
votes
0answers
37 views

Comparison of Box-Cox transformed data with different lambdas.

I'm trying to compare some data and see if there is a significant Pinteraction value between them. The data is highly skewed and thus I would like to use a transformation; a log transformation results ...
0
votes
0answers
24 views

Suitable data transformation in multiple GLS regression of raw variables against PCs

Im trying to make a multiple regresion in which the dependent variable and one of the factors were originally positive variables but were centered and scaled for easire interpretation of effect sizes. ...
0
votes
0answers
19 views

Subspace clustering with random transformation

One approach for clustering a high dimensional dataset is to use linear transformation, and the most common approaches are PCA and random projection (where random projection arises from the ...
1
vote
3answers
256 views

Feature Selection

I've a facebook users dataset in which each user has a "huge" set of attribute, i.e about 220 attributes like age, hometown, religion, and a set of facebook liked pages to store the users tastes. Now ...
1
vote
1answer
28 views

how to transform data of two experimental groups? one is positively skewed and one is negatively..

I have two experimental groups. Then I test their normality respectively. Result shows that one is positively skewed and the other is negatively skewed. In this case, how should I do the data ...
2
votes
0answers
48 views

Why linear transformation can improve classification accuracy when the dimensionality of data is high?

Let $X$ be an $m\times n$ ($m$: number of records, and $n$: number of attributes) dataset. When the number of attributes $n$ is large and the dataset $X$ is noisy, classification gets more ...
0
votes
0answers
15 views

Non-Normal residuals after Box-Cox

I applied Box-Cox on a regression model with heteroscedasticity. Although the transformed model looked better than the original, it still showed signs of heteroscedasticity. What else can I do after ...
2
votes
1answer
62 views

What exactly does the 'boxcox' function in R do?

I am familiar with the power transform family and I know how to estimate the MLE for $\lambda$ for given samples of a random variable. I have been using the 'boxcox' function in R for a sample of a ...