Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

learn more… | top users | synonyms (2)

0
votes
1answer
29 views

Transforming time series to compensate for change in variance

I have a time series (shown below) that comes from a sensor whose calibration was changed in the middle of last year. As part of this change, the sensor's reading of the variance (or volatility) of ...
0
votes
0answers
19 views

how to handle the imbalanced data in regression analysis

The problem here is very similar to the problem asked by someben in 2012 (link:Sampling for Imbalanced Data in Regression). It involves the linear regression analysis using an unbalanced dataset. Say, ...
0
votes
0answers
16 views

Untransforming unbiased estimates

Suppose I have some measured experimental data and I want to fit it to a power law of the form $y=ax^b$. Suppose I transform the data to log-log space and then I fit a straight line of the form ...
1
vote
0answers
17 views

How to eliminate dependent inputs?

There are a lot of statistical methods that rely on the assumption of input independence. For example, Naive Bayes text classifiers operate under the assumption that occurrences of different words are ...
1
vote
0answers
26 views

Order of preprocessing steps in a binary classification problem

I have these stages (ordered) for preprocessing in my binary classification problem. Dividing data based on criteria (class1 and class2 databases) Outlier ...
3
votes
1answer
33 views

Spread-Level Plot versus Power Transformation Functions in R

I'm having trouble interpreting the results from the Spread-Level Plot function in R (car package). The documentation says: PowerTransformation spread-stabilizing power transformation, ...
4
votes
1answer
92 views

Regression where the dependent variable is the difference between two correlated variables — bias and other issues to consider

I am interested in estimating a regression that looks like this: $(x_{1,i} - y_{i} )_{i} = x’_{i}*\beta + \epsilon_{i}$ (1) However, I am not sure if doing this—in this form—is appropriate. ...
1
vote
0answers
28 views

Plotting data and polynomial equation when x-values have been transformed

I hope I can make my question clear, if not I will be happy to clarify. I want to present a simple regression plot with the polynomial equation line linked to it. Those data have been tested in a ...
0
votes
0answers
24 views

Have Narrow Distribution - Need Standard Distribution

I am building a model in R using GLM based on this predictor variable. As you can see, the data is concentrated in the center of the distribution, and then falls off sharply. Is there a ...
2
votes
0answers
24 views

Derivation of normalizing transform for GLMs

How is the $A(\cdot) = \int\frac{du}{V^{1/3}(\mu)}$ normalizing transform for the exponential family derived? More specifically: I tried to follow the Taylor expansion sketch on page 3, slide 1 of ...
1
vote
0answers
17 views

Transformation for zeros

I have been working on seed germination experiments with some dormancy breaking treatments and only for some treatments only I got germination. I need to analyze the parameters like days for initial ...
0
votes
1answer
27 views

Multlinear regression: analysis of residual of transformed response and predictor variables

In the first step of modeling a regression equation I came up with the following model: $T_c = 26.73 + 0.042{\rm Sc} + 0.247{\rm Lc} - 14.709{\rm Lf} + 1.41{\rm Lu} - 0.214{\rm Fc} + 0.041{\rm Ad} - ...
0
votes
0answers
26 views

How can i make a fraud detection dataset (I have the data ready but unordered)

I'm a little confused with the creation of the dataset for a fraud detection predictive model. Here i put a link with a sample of the dataset that I made. (the real dataset have ~950.000 clients). ...
2
votes
2answers
34 views

transformation of percentages (and eventual display)

I have some percentage data derived from the analysis of grain presence and absence in ears of wheat that have undergone two treatments (control and heat stress). For example, data might be like 10%, ...
0
votes
0answers
11 views

inverse Gaussian data transformation in R? [migrated]

Anyone know of an R package that will allow me to do an inverse Gaussian/Wald distribution transform of my data (reaction times)? Thanks!
0
votes
0answers
14 views

How to interpret log of independent variable in Poisson regression? [duplicate]

How to interpret log of independent variable in Poisson regression? Can we compare two independent variable one with Log and one without log in the model equation? For e.g. Y = b1*X1+b2*log(X2) Once ...
0
votes
0answers
28 views

Is this poor transformation advice for predictive modeling?

I have gotten some advice from a PhD statistician on doing predictive modeling on large datasets (lots of variables AND lots of observations) that I should perform transformations to eliminate ...
2
votes
1answer
39 views

Linear Discriminant Analysis and non-normal distributed data

If I understand correctly, a Linear Discriminant Analysis (LDA) assumes normal distributed data, independent features, and identical covariances for every class for the optimality criterion. Since ...
1
vote
0answers
78 views

Data preparation for Poisson regression: use of individual data

Most texts I have read about Poisson-regression assumes that the data is available in an already grouped form, i.e. counts are given for each unique covariate combination. For instance, we have (in R) ...
4
votes
1answer
67 views

Working with residuals of regression

So the background is that the I collected yield data for past 5-6 decades and location from where I collected yield data had high yielding varieties introduced over time. I am looking at the ...
3
votes
1answer
61 views

Normal distribution to triangular distribution

I would like to know if it is possible to convert a normal distribution into a triangular distribution. If it is, how it can be done? I know the mean and the coefficient of variation of the normal ...
1
vote
0answers
53 views

nMDS in vegan for soil data

I am working with abiotic soil data such as bulk density, moisture levels and soil chemistry as response data (some quantitative some as percentages) and a mix of abiotic and biotic data as ...
0
votes
0answers
5 views

Design a feature with time and presence information

Context: I am working on a decision tree classifier, trying to classify businesses as to whether they are likely to have an event occur (default) in the next 90 days. One input I get is whether, and ...
0
votes
0answers
16 views

strange coeficient estimates in GLS with ranked variables

Somebody could explain me why the estimated coefficients of a multiple regression through GLS seem not to pass through the majority of observations? Here is a example: ...
4
votes
1answer
95 views

Model with non-linear transformation

I don't understand this concept well and need help. I was choosing whether to use a linear model or apply a non-linear transformation in my model formula. To do a diagnostic, I quickly plotted my ...
0
votes
0answers
12 views

Mixing Categorical and Continuous variables where cardinality of categorical can surpass data points

Suppose we have a dataset of people that can be described with a mix of some continuous variables (eg height, age) some ordinal (eg social status) and some categorical (eg city, car brand, favourite ...
1
vote
1answer
51 views

Empirical logit transformation on percentage data

I have already used the logit transform on my outcome variables (which are displayed in percentages). However, this obviously gives me -INF values and since my data includes a lot of zeros in some ...
1
vote
0answers
20 views

clustering for histogram shapes

I am trying to get a start on a clustering problem. The sample data is trade volume at a particular price. Some notes about the data: number of bins vary from sample to sample (larger price range ...
1
vote
1answer
24 views

Transforming data for canonical correlation analysis

Okay, I'm a stats newbie so I'll try to be as specific and clear as possible. I have a set of predictor variables (2 predictor variables) and a set of response variables (7 response variables). I am ...
2
votes
1answer
39 views

Choosing variable transformations in non-linear relationships

I am confused about how to apply a transformation to my predictor/response variables to test curvilinear relationships. I read about log transformations, polynomials, quadratic functions. But I am not ...
1
vote
1answer
16 views

Whether to transform non-normal pre-test when running linear regression on transformed post-test?

I'm running linear regression model on a post-intervention test score controlling for pre-intervention test score. I used Box-Cox transformation on the post-intervention test score to normalize it. ...
2
votes
1answer
46 views

What's the optimal way to encode a 'month' feature?

What's the optimal way to encode a 'month' feature? A single integer value or 12 binary values don't quite grasp the concept of modulo distance... Say I want to train an SVM for a certain task and ...
-1
votes
1answer
31 views

Box Cox transformed my data, now how to use it in my mixed model? [closed]

Sorry for the (most likely) simple question, but I have Box Cox transformed my data in SAS, but I am unsure how to use the transformed data in my mixed model. Do I output it to a new file, and use ...
3
votes
1answer
45 views

Aspect data in linear regression

I have a dataset of various ecological variables on which I want to run linear regression. The variables are continuous, but also include aspect data (sun exposure), in grades. My problem is that the ...
1
vote
1answer
24 views

What methods can be used to transform data?

I am solving a binary classification problem with 4 predictor variables. The variables didn't seem to be linearly separable. I have used Neural Networks and Kernel SVM which work and give desired ...
0
votes
0answers
18 views

taking the log of (Only) some variables in a regression

My question is as follows: Ratio=Ratios + log (numbers) +dummy variables + volatility I have this type of regression in a paper published by the Federal Reserve Bank.Can someone tell me why we took ...
4
votes
3answers
82 views

What is the reason the $\log$ transformation is used with right-skewed distributions?

I once heard that log transformation is the most popular one for right-skewed distributions in linear regression or quantile regression I would like to know is there any reason underlying this ...
0
votes
1answer
46 views

Using F-tests for variance in non-normal populations

I'm fairly new to stats, so please excuse me if this problem is hopelessly elementary or misinformed. Basically, I'm wondering if you can help me understand whether I'm using the F-Test for variance ...
0
votes
0answers
11 views

Compute how much the one percent wealthier concentrates with negative data

I want to compute, using the Survey of Consumer Finances (SCF) database, how much the one percent wealthier concentrates. The variable I'm using to measure wealth is the "networth". The problem is ...
0
votes
1answer
46 views

Interpreting Log-Transformed Percentages in OLS

In a log-log model, such as $\log(y) = b_0 + b_1 \log(x)$, I know that with OLS the standard interpretation is a "1% increase in x is associated with a $b_1$% increase in y." I have three related ...
2
votes
3answers
117 views

Best way to turn a date into a numerical feature?

I have a fairly large dataset with a few fields containing time-related data. This data comes in various shapes and sizes, but most of it can be parsed and rephrased in more appropriate formats for ...
1
vote
1answer
70 views

Data transformation and confidence intervals for mean difference

I have a between-group independent variable with two level (A and B) and a dependent variable Y that I transformed in order to normalize the distribution of the residual. I used a Box Cox ...
1
vote
2answers
63 views

Appropriate way to treat [0,1]-distributed variables in HLM

Brief intro: I'm not really sure how to appropriately treat the dependent variables in a set of hierarchical linear models that I'm trying to run. In my models, Level 1 units are children and Level 2 ...
2
votes
2answers
80 views

Best transformation for sinuous data sets?

I am analysing annual behavioural patterns and am currently scratching my head over how to best transform data to test for correlation where my independent variable is the moon phase (given as ...
0
votes
0answers
18 views

Questions about $R^2$, VIFs and very non normal input variables

I have been working with a small part of my dataset trying to eliminate variables and do some micro models. When analysing my micro set I initially found a few high correlations with inputs (0.95 ...
1
vote
0answers
14 views

Combine variables that are extremely lightly populated?

A similar question to my other question about mixed distributions. Here i have quite a few variables that are populated to less than 5%, many are even populated to less than 1% this 1% would represent ...
0
votes
1answer
36 views

Collection of continuous variables with >70% more zeros

This is dataset that is going to be data mined for factors that affect an output that of interest A large Part of my dataset (150 of 300 potential inputs) has a heavy skew of Zero values. usually ...
0
votes
0answers
34 views

Transforming data with large negative and large positive values

The data I'm trying to analyze are the quadratic estimates from a quadratic fit to a curve. Most of the data vary between -.15 and .15. However, I have outliers in both directions up to things like ...
0
votes
0answers
18 views

What's the best way to transform this vector to be normal-like?

Assume I have an outcome like this: ...
2
votes
0answers
19 views

How to transform an outcome which censored from both sides?

I'm trying to fit a tobit model for a response outcome where both sides were censored and the distribution is heavily skewed, even for the none-censored ...