# Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

2,007 questions
Filter by
Sorted by
Tagged with
229 views

### Transformation by subtraction (for continuous data arbitrarily limited in the experiment)

Consider a study looking at the relationship of chili-pepper plant height to number of chilies produced. Data was collected on all plants taller than 10cm. Initial look at the distribution of ...
1k views

### Analysing log and square-root transformed variables

I'm just wondering if it's possible to analyse and interpret variables that have been transformed using different transformations (eg log and square-root) or if all variables in the analysis have to ...
25k views

### Linear model Heteroscedasticity

I have the following linear model: To address the residuals heteroscedasticity I have tried to apply a log transformation on the dependent variable as $\log(Y + 1)$ but I still see the same fan out ...
73 views

### Pre-process classification data according to ‘amount of evidence’

I have an area which is divided into polygons of different sizes. Each polygon has the same associated features/predictors and I know whether something occurs within the polygon or not. If something ...
1k views

### Is paired t-test valid for half-normal data?

ok i am not a statistician. I am doing paired comparison for several outcomes of cross validated interpolation results. I have to consider only the absolute value in my analysis and so although ...
358 views

### Is the true relation between independent and dependent variables assumed to be a function or a distribution?

In classification and regression tasks, we try to learn from a training data set a function mapping a independent variable $X$ to a dependent variable $Y$. When evaluating the error rate of a ...
242 views

### Why is my replication of Silver & Dunlap 1987 not working out?

I'm trying to replicate Silver & Dunlap (1987). I'm just comparing averaging correlations or averaging z transform correlations and back transforming. I seem to not be replicating the asymmetry ...
736 views

### How to fairly determine winners for a regional science fair?

I need help figuring out the correct way to calculate winners at our science fair. I don't want my ignorance of statistics & math to get in the way of a kid's chances of winning. (lots of ...
806 views

### Interpretation of a log likelihood function for PROC NLMIXED in SAS

I have a data set of skewed nutrient intake values, from around 7800 individuals, of whom around 3000 had two measures of daily nutrient intake (the others only had one measure), so this is a repeated ...
506 views

### Power transformation on interactions in linear regression

In a multiple linear regression I have one predictor showing a cubic relationship with the outcome. If the same predictor $A$ is also part of an interaction with a continuous variable $B$, do I have ...
701 views

### Power analysis for matched poisson variables

I have two matched groups consisting of 22 observations each for a Poisson distributed variable. I am interested in evaluating the power to detect a 20% decrease in the outcome for the second group. I ...
1k views

### Running transformed data with a Generalized Linear Model in SPSS

I am running a generalized linear model in SPSS on data highly skewed to the right with a bunch of zeroes (average # caterpillars per tree species branch sampled) so I decided to use the Tweedie ...
486 views

### Manually transforming two dependent variables according to their correlation matrix in R?

I have two continuous variables, X and Y, that are correlated - they are not independent. To correct for non-independence, I have a known correlation structure, a matrix S. If one calls ...
3k views

### Data preparation for logistic regression

This figure: shows my odds ratios. I believe they have to be linear for logistic regression to work and am wondering how to pre-process these data. Thanks.
1k views

### Adjusting for tilt of the earth

I have rewritten the old question (below) to hopefully make things a bit clearer. Basically I think that the temperature of the earth should be normally distributed but is not due to the ‘seasonal ...
188 views

### Formally speaking, is converting some variable into a per capita measure a transformation?

When I convert some data for a model (e.g., wealth, pollution) into per capita figures (e.g., wealth per inhabitant, tons of pollution per inhabitant) is this a transformation or re-expression of the ...
781 views

### Grouped income data in Stata or SAS

I have an income distribution dataset which I am attempting to manipulate in Stata (or SAS). I need to come up with a table listing the number of households earning between one income level and ...
23k views

### When to transform predictor variables when doing multiple regression?

I'm currently taking my first applied linear regression class at the graduate level, and am struggling with predictor variable transformations in multiple linear regression. The text I'm using, Kutner ...
3k views

### Transformation to fit gamma distribution for glm

The data simulated below has a maximum value of 4 and is interestingly skewed. The maximum of 4 is a limitation imposed by the instrument used and the data is semi-discrete, i.e., there are a ...
364 views

### Is it valid to model discrete numerical test scores as coming from a continuous random variable?

I'm working with a sample of test scores which range from 0 to 100. These scores are generated from a set of 100 binary responses (0 or 1), so the higher the resulting sum, the better the performance. ...
1k views

### What order preserving transformation makes data more evenly spread, decreasing the peak, and fattening the tails of the distribution?

How can I transform a variable (non linear transformation) such that its values are more evenly spread, that is reduce the peak in the middle of the histogram and move more into tails?
1k views

### Literature on generating “similar” synthetic time series from observed time series

The motivation for this question is from Finance. I have some market data (daily time series) for the price of some securities and I would like to generate synthetic versions of these which are ...
174 views

### How to analyse this data obtained from a simple physics experiment on attractive forces?

I did a simple physics experiment that measures the attractive force a plate experiences towards the other plate as a function of the applied voltage and distance between the plates. Now I have to ...
2k views

### Volatility of x and y variables in linear regression?

I have a simple regression with price returns: $r_{t+1} = \alpha + \beta r_t + \epsilon$ My question is: do I need to do anything if $r_{t+1}$ and $r_t$ are over different horizons? Suppose the x-...
171 views

### What assumptions do I need to check to combine levels in a categorical predictor for use in a GLM?

If I have a GLM with a number of explanatory variables, where one is a categorical variable with levels "no treatment" "treatment A" "treatment B" Assuming that it is reasonable to combine the 2 ...
1k views

### Poisson regression instead of log transformation of a continuous outcome variable? [duplicate]

Possible Duplicate: Poisson regression vs. log-count least-squares regression? This post got me to thinking about when it makes sense to use poisson regression instead of log transforming a ...
214 views

### How to compare data with difference in scale?

I am working with the data obtained from a travel survey from a town. I have the data from mean of trips made by different travel modes on the day of the survey. From a national suevey I have got data ...
2k views

### Normalisation for regression

First off, I know little about statistics, so some of this question may seem naive. I'm trying to perform linear regression to model the relationship between x and y where: -x is a company's daily ...
13k views

### Detrending method for nonstationary data

Could anybody help me with detrending data that is nonstationary? I have already made the mistake of trying to detrend it by plotting the residuals of a linear regression in excel but it was pointed ...
9k views

### Types of inverse transformations

In performing an inverse transformation to correct for skewness/kurtosis in SPSS, it asks me to choose what "type" of inverse transformation and I have no idea what the differences between these ...
360 views

### Exponentially decaying variable

I have a variable which decays very quickly (exponentially). I want to transform it in a way it maintains "exponentially decaying" property but which will allow me to control the "speed" of a decay. <...
1k views

### Detect multi-punch responses in SPSS syntax?

I have a dataset that includes an open-ended response that was coded into categories by a vendor. The vendor created 50 multi-punch categories stored as 50 true/false variables. By putting this in a ...
393 views

### What, if any, dissimilarity is preserved in partial least squares (PLS)?

When we perform a principal components analysis (PCA) on a multivariate data set we are interested in finding orthogonal components that explain maximal variance in the data set. We can form a biplot ...
7k views

### Rescaling for desired standard deviation

Apologies for what is probably a very basic question. I have looked around both here and in the usual places and haven't had any luck. I have read that there are at least two methods for linearly ...
4k views

### A transform to change skew without affecting kurtosis?

I am curious if there is a transform which alters the skew of a random variable without affecting the kurtosis. This would be analogous to how an affine transform of a RV affects the mean and variance,...
3k views

### Expected value of a transformation

A simple question. If $Y=\frac{1}{X}$ and I know $f_X(x)$, is it true that $E(Y) = E(1/X) = \int_{-\infty}^\infty \frac{1}{x}f_X(x) dx$?
430 views

### Is it necessary to perform a transformation on proportion data if it's reasonably well behaved?

If you're modelling a proportion response against numerous predictors that are also proportions, is it necessary to transform the response if the standard OLS model is seemingly well behaved? By well ...
94 views

### Analyzing historical incident rates and rating future performance

I am analyzing a large of dataset (n>100) of incident rates, with the aim of forming a normal distribution. Then I will know if a future incident rate (x%) is either close to a historical mean or not, ...
21k views

### How to split a numeric variable into a binary low-high variable

I have measured frequency of a certain behavior on 15 individuals. I would like to create two groups based on the amount of this behaviour that was observed (i.e., a group exhibiting high levels of ...
267 views

### Categorical or continuous scale for area chart?

I would like to plot the attached income distribution dataset (rendered as an image) as an area chart. As you can see, personal income is divided into 26 intervals of varying width. I also have the ...
418 views

### Questions about number of groups and group size in planned comparisons in ANCOVA

I have three questions. I want to conduct a planned comparison in ANCOVA which compares three groups (n=2, n=7 and n=5) with two groups (n=11 and n=25). Is it ok to include the group with n=2 if I am ...
3k views

### Are these formulas for transforming P, LSD, MSD, HSD, CI, to SE as an exact or inflated/conservative estimate of $\hat{\sigma}$ correct?

Background I am conducting a meta-analysis that includes previously published data. Often, differences between treatments are reported with P-values, least significant differences (LSD), and other ...
1k views

### How to transform data to have a new mean and maximum value?

I have data relative to image intensity, which range from 0 to 255. For some models I need to run, I must transform this intensity data to a different scale, which must have a certain mean (for ...
7k views

### How to do ANOVA on data which is still not normal after transformations?

I'm looking at the effect defeat and entrapment inducing conditions have on subjective ratings of defeat and entrapment at three different time points (among other things). However the subjective ...
140 views

### Correlation between two nodes of a single layer MLP for joint-Gaussian input

Let's say you have a jointly gaussian vector random variable $\mathbf{x}$, with mean $\mathbf{M}$ and covariance $\mathbf{S}$. I now transform each scalar element of $\mathbf{x}$ , say $x_j$, with a ...
1k views

### Methodology for data transform that best fits normal

SAS EM has a node that generates the transformation for a dataset that makes it best fit a normal distribution. I can't imagine the people at SAS are the only ones to figure out how to do this. Is ...
1k views

### Noise in the Fourier transform

I have a smooth but rather complex curve, sampled with a good frequency. I apply the discrete Fourier transform to it using the fast Fourier transform (FFT) algorithm and get its Fourier image. I need ...
5k views

### SVD dimensionality reduction for time series of different length

I am using Singular Value Decomposition as a dimensionality reduction technique. Given N vectors of dimension D, the idea is to ...
Motivation I have a prior on a random variable $X\sim \beta^\prime(\alpha,\beta)$ that I would like to use in JAGS. JAGS does not support the $\beta^\prime$ distribution, but it does support the $F$ ...