Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
2
votes
2answers
229 views

Transformation by subtraction (for continuous data arbitrarily limited in the experiment)

Consider a study looking at the relationship of chili-pepper plant height to number of chilies produced. Data was collected on all plants taller than 10cm. Initial look at the distribution of ...
3
votes
1answer
1k views

Analysing log and square-root transformed variables

I'm just wondering if it's possible to analyse and interpret variables that have been transformed using different transformations (eg log and square-root) or if all variables in the analysis have to ...
10
votes
3answers
25k views

Linear model Heteroscedasticity

I have the following linear model: To address the residuals heteroscedasticity I have tried to apply a log transformation on the dependent variable as $\log(Y + 1)$ but I still see the same fan out ...
1
vote
0answers
73 views

Pre-process classification data according to ‘amount of evidence’

I have an area which is divided into polygons of different sizes. Each polygon has the same associated features/predictors and I know whether something occurs within the polygon or not. If something ...
1
vote
2answers
1k views

Is paired t-test valid for half-normal data?

ok i am not a statistician. I am doing paired comparison for several outcomes of cross validated interpolation results. I have to consider only the absolute value in my analysis and so although ...
1
vote
1answer
358 views

Is the true relation between independent and dependent variables assumed to be a function or a distribution?

In classification and regression tasks, we try to learn from a training data set a function mapping a independent variable $X$ to a dependent variable $Y$. When evaluating the error rate of a ...
8
votes
1answer
242 views

Why is my replication of Silver & Dunlap 1987 not working out?

I'm trying to replicate Silver & Dunlap (1987). I'm just comparing averaging correlations or averaging z transform correlations and back transforming. I seem to not be replicating the asymmetry ...
9
votes
1answer
736 views

How to fairly determine winners for a regional science fair?

I need help figuring out the correct way to calculate winners at our science fair. I don't want my ignorance of statistics & math to get in the way of a kid's chances of winning. (lots of ...
4
votes
0answers
806 views

Interpretation of a log likelihood function for PROC NLMIXED in SAS

I have a data set of skewed nutrient intake values, from around 7800 individuals, of whom around 3000 had two measures of daily nutrient intake (the others only had one measure), so this is a repeated ...
3
votes
1answer
506 views

Power transformation on interactions in linear regression

In a multiple linear regression I have one predictor showing a cubic relationship with the outcome. If the same predictor $A$ is also part of an interaction with a continuous variable $B$, do I have ...
5
votes
1answer
701 views

Power analysis for matched poisson variables

I have two matched groups consisting of 22 observations each for a Poisson distributed variable. I am interested in evaluating the power to detect a 20% decrease in the outcome for the second group. I ...
3
votes
1answer
1k views

Running transformed data with a Generalized Linear Model in SPSS

I am running a generalized linear model in SPSS on data highly skewed to the right with a bunch of zeroes (average # caterpillars per tree species branch sampled) so I decided to use the Tweedie ...
4
votes
1answer
486 views

Manually transforming two dependent variables according to their correlation matrix in R?

I have two continuous variables, X and Y, that are correlated - they are not independent. To correct for non-independence, I have a known correlation structure, a matrix S. If one calls ...
3
votes
1answer
3k views

Data preparation for logistic regression

This figure: shows my odds ratios. I believe they have to be linear for logistic regression to work and am wondering how to pre-process these data. Thanks.
4
votes
2answers
1k views

Adjusting for tilt of the earth

I have rewritten the old question (below) to hopefully make things a bit clearer. Basically I think that the temperature of the earth should be normally distributed but is not due to the ‘seasonal ...
0
votes
1answer
188 views

Formally speaking, is converting some variable into a per capita measure a transformation?

When I convert some data for a model (e.g., wealth, pollution) into per capita figures (e.g., wealth per inhabitant, tons of pollution per inhabitant) is this a transformation or re-expression of the ...
3
votes
3answers
781 views

Grouped income data in Stata or SAS

I have an income distribution dataset which I am attempting to manipulate in Stata (or SAS). I need to come up with a table listing the number of households earning between one income level and ...
11
votes
1answer
23k views

When to transform predictor variables when doing multiple regression?

I'm currently taking my first applied linear regression class at the graduate level, and am struggling with predictor variable transformations in multiple linear regression. The text I'm using, Kutner ...
7
votes
2answers
3k views

Transformation to fit gamma distribution for glm

The data simulated below has a maximum value of 4 and is interestingly skewed. The maximum of 4 is a limitation imposed by the instrument used and the data is semi-discrete, i.e., there are a ...
2
votes
0answers
364 views

Is it valid to model discrete numerical test scores as coming from a continuous random variable?

I'm working with a sample of test scores which range from 0 to 100. These scores are generated from a set of 100 binary responses (0 or 1), so the higher the resulting sum, the better the performance. ...
3
votes
1answer
1k views

What order preserving transformation makes data more evenly spread, decreasing the peak, and fattening the tails of the distribution?

How can I transform a variable (non linear transformation) such that its values are more evenly spread, that is reduce the peak in the middle of the histogram and move more into tails?
4
votes
3answers
1k views

Literature on generating “similar” synthetic time series from observed time series

The motivation for this question is from Finance. I have some market data (daily time series) for the price of some securities and I would like to generate synthetic versions of these which are ...
3
votes
1answer
174 views

How to analyse this data obtained from a simple physics experiment on attractive forces?

I did a simple physics experiment that measures the attractive force a plate experiences towards the other plate as a function of the applied voltage and distance between the plates. Now I have to ...
4
votes
1answer
2k views

Volatility of x and y variables in linear regression?

I have a simple regression with price returns: $r_{t+1} = \alpha + \beta r_t + \epsilon$ My question is: do I need to do anything if $r_{t+1}$ and $r_t$ are over different horizons? Suppose the x-...
0
votes
1answer
171 views

What assumptions do I need to check to combine levels in a categorical predictor for use in a GLM?

If I have a GLM with a number of explanatory variables, where one is a categorical variable with levels "no treatment" "treatment A" "treatment B" Assuming that it is reasonable to combine the 2 ...
5
votes
1answer
1k views

Poisson regression instead of log transformation of a continuous outcome variable? [duplicate]

Possible Duplicate: Poisson regression vs. log-count least-squares regression? This post got me to thinking about when it makes sense to use poisson regression instead of log transforming a ...
1
vote
1answer
214 views

How to compare data with difference in scale?

I am working with the data obtained from a travel survey from a town. I have the data from mean of trips made by different travel modes on the day of the survey. From a national suevey I have got data ...
1
vote
2answers
2k views

Normalisation for regression

First off, I know little about statistics, so some of this question may seem naive. I'm trying to perform linear regression to model the relationship between x and y where: -x is a company's daily ...
2
votes
4answers
13k views

Detrending method for nonstationary data

Could anybody help me with detrending data that is nonstationary? I have already made the mistake of trying to detrend it by plotting the residuals of a linear regression in excel but it was pointed ...
2
votes
1answer
9k views

Types of inverse transformations

In performing an inverse transformation to correct for skewness/kurtosis in SPSS, it asks me to choose what "type" of inverse transformation and I have no idea what the differences between these ...
2
votes
2answers
360 views

Exponentially decaying variable

I have a variable which decays very quickly (exponentially). I want to transform it in a way it maintains "exponentially decaying" property but which will allow me to control the "speed" of a decay. <...
1
vote
1answer
1k views

Detect multi-punch responses in SPSS syntax?

I have a dataset that includes an open-ended response that was coded into categories by a vendor. The vendor created 50 multi-punch categories stored as 50 true/false variables. By putting this in a ...
6
votes
0answers
393 views

What, if any, dissimilarity is preserved in partial least squares (PLS)?

When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
5
votes
2answers
7k views

Rescaling for desired standard deviation

Apologies for what is probably a very basic question. I have looked around both here and in the usual places and haven't had any luck. I have read that there are at least two methods for linearly ...
12
votes
3answers
4k views

A transform to change skew without affecting kurtosis?

I am curious if there is a transform which alters the skew of a random variable without affecting the kurtosis. This would be analogous to how an affine transform of a RV affects the mean and variance,...
5
votes
2answers
3k views

Expected value of a transformation

A simple question. If $Y=\frac{1}{X}$ and I know $f_X(x)$, is it true that $E(Y) = E(1/X) = \int_{-\infty}^\infty \frac{1}{x}f_X(x) dx$?
7
votes
1answer
430 views

Is it necessary to perform a transformation on proportion data if it's reasonably well behaved?

If you're modelling a proportion response against numerous predictors that are also proportions, is it necessary to transform the response if the standard OLS model is seemingly well behaved? By well ...
0
votes
1answer
94 views

Analyzing historical incident rates and rating future performance

I am analyzing a large of dataset (n>100) of incident rates, with the aim of forming a normal distribution. Then I will know if a future incident rate (x%) is either close to a historical mean or not, ...
3
votes
2answers
21k views

How to split a numeric variable into a binary low-high variable

I have measured frequency of a certain behavior on 15 individuals. I would like to create two groups based on the amount of this behaviour that was observed (i.e., a group exhibiting high levels of ...
4
votes
2answers
267 views

Categorical or continuous scale for area chart?

I would like to plot the attached income distribution dataset (rendered as an image) as an area chart. As you can see, personal income is divided into 26 intervals of varying width. I also have the ...
2
votes
1answer
418 views

Questions about number of groups and group size in planned comparisons in ANCOVA

I have three questions. I want to conduct a planned comparison in ANCOVA which compares three groups (n=2, n=7 and n=5) with two groups (n=11 and n=25). Is it ok to include the group with n=2 if I am ...
11
votes
3answers
3k views

Are these formulas for transforming P, LSD, MSD, HSD, CI, to SE as an exact or inflated/conservative estimate of $\hat{\sigma}$ correct?

Background I am conducting a meta-analysis that includes previously published data. Often, differences between treatments are reported with P-values, least significant differences (LSD), and other ...
3
votes
2answers
1k views

How to transform data to have a new mean and maximum value?

I have data relative to image intensity, which range from 0 to 255. For some models I need to run, I must transform this intensity data to a different scale, which must have a certain mean (for ...
8
votes
3answers
7k views

How to do ANOVA on data which is still not normal after transformations?

I'm looking at the effect defeat and entrapment inducing conditions have on subjective ratings of defeat and entrapment at three different time points (among other things). However the subjective ...
6
votes
1answer
140 views

Correlation between two nodes of a single layer MLP for joint-Gaussian input

Let's say you have a jointly gaussian vector random variable $\mathbf{x}$, with mean $\mathbf{M}$ and covariance $\mathbf{S}$. I now transform each scalar element of $\mathbf{x}$ , say $x_j$, with a ...
4
votes
2answers
1k views

Methodology for data transform that best fits normal

SAS EM has a node that generates the transformation for a dataset that makes it best fit a normal distribution. I can't imagine the people at SAS are the only ones to figure out how to do this. Is ...
4
votes
1answer
1k views

Noise in the Fourier transform

I have a smooth but rather complex curve, sampled with a good frequency. I apply the discrete Fourier transform to it using the fast Fourier transform (FFT) algorithm and get its Fourier image. I need ...
14
votes
5answers
5k views

SVD dimensionality reduction for time series of different length

I am using Singular Value Decomposition as a dimensionality reduction technique. Given N vectors of dimension D, the idea is to ...
4
votes
1answer
741 views

How do I transform the parameters of the F distribution?

Motivation I have a prior on a random variable $X\sim \beta^\prime(\alpha,\beta)$ that I would like to use in JAGS. JAGS does not support the $\beta^\prime$ distribution, but it does support the $F$ ...
9
votes
3answers
11k views

Linear regression effect sizes when using transformed variables

When performing linear regression, it is often useful to do a transformation such as log-transformation for the dependent variable to achieve better normal distribution conformation. Often it is also ...