Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

learn more… | top users | synonyms (2)

0
votes
0answers
7 views

Stieltjes transformation for empirical distribution function

If the eigen-decomposition of sample covariance matrix is $S=PDP'$ where $D$ is a diagonal matrix with eigen value of $S$ and $P$ are eigenvectors. If we define the empirical distribution function of ...
2
votes
1answer
38 views

How to extend PDF of normalized sample to original sample?

To calculate the PDF function using Shannon entropy I have scaled my original sample by simply doing $x'=(x-a)/(b-a)$; where $b=\text{max}(x)$, and $a=\text{min}(x)$ and then I found the ...
0
votes
0answers
17 views

Calculate first difference by group in R [migrated]

I was wondering if someone could help me calculate the first difference of a score by group. I know it should be a simple process but for some reason I'm having trouble doing it..... yikes Here's an ...
0
votes
1answer
42 views

Transformation of latitudes to include them a linear model

I have in a dataset a variable latitude that contains values such as 39° 37' 39" N and ...
1
vote
1answer
42 views

How should you fit ANOVA and linear regression models, if the equal variance assumption is violated?

This is my topic for the paper I'm working on for an undergrad stats class. It's supposed to be 20 pages... and I'll be honest, I understand very little beyond the basics and am over my head. From ...
2
votes
0answers
71 views

How to get the regression from a plot?

I have a dependent variable C and an independent variable VPT. VPT is the average volume per ...
1
vote
0answers
39 views

Can a Linear-Log model be used instead of Robust Standard Errors?

If your regression model has heteroskedastic residuals, one should calculate White Standard Errors that correct for the mentioned heteroskedasticity. If the residuals are also autocorrelated one ...
2
votes
0answers
20 views

Curvature of a pencil's stroke

I'm trying to evaluate the curvature of an image of a pencil stroke based on the image's pixels and their shade of gray. I'm trying to get the curvature of the line at every point of the stroke. The ...
0
votes
0answers
45 views

correlation with logarithmic transformation

I have a dataset of 2.000.000 projects. Each project is defined by its size and the number of active developers. After applying a logarithmic transformation on the project size, I've plotted the ...
0
votes
0answers
3 views

how to transfer/export output of rcorr(matrix) into excel table? [migrated]

I use correlation test using code "rcorr", and here it is my script: ...
1
vote
0answers
13 views

Pre-stimulus baseline removal in R

I have the following scenario: trials were conducted where participants were exposed to multiple stimuli during the course of a trial a specific physiological response was continuously recorded ...
2
votes
4answers
52 views

Beta confidence intervals in transformed linear regression

Let's say I have a model: $$Y_i = \beta_0 \beta_1^{X_i} \epsilon_i$$ (note: This is slightly different than the more common example case of $Y_i = \alpha e^{\beta x_i}\epsilon_i$.) I can take the ...
0
votes
1answer
24 views

Equal-size categories vs unequal-size categories

I'm trying to reduce the size of my dataset, which is composed of 200,000 projects. Each project is defined by its size and a binary value that is 1 if the project has active users, 0 otherwise. Most ...
2
votes
2answers
77 views

Reporting regression statistics after logarithmic transformation

I'm a bit troubled about how to report linear regression statistics after log transformation of the dependent variable. I suppose I should report the transformed coefficient, but would they be easily ...
0
votes
1answer
32 views

Transformation of dependent variable: how to interpret it?

I recently reread some statistics books and noted something weird: They all discuss the assumptions of linear regression and mention the need for a normal distributed dependent variable. In the next ...
1
vote
0answers
19 views

Back Transforming Rates in Poisson GLM with Box and Cox Transformation

Suppose I have fitted a Poisson GLM to model rates as follows: > fit.1=glm(response~X1+X2+ offset(log(population)),family=poisson,data=...) I can get the ...
1
vote
1answer
42 views

PIT on a sample with m bins, and KS test used to estimate a good value for m

I know about PIT, but this works only when you know the distribution, or at least have a strong hint. What I am trying to achieve is to transform a given sample into an equivalent sample with ...
1
vote
0answers
32 views

How to interpret Box-Cox Transformation [duplicate]

I have used a Box-Behnken experimental design. I have a full quadratic model. However, I had to transform the response, $Y$ for the model to fit; I did this using a Box-Cox transformation with ...
2
votes
1answer
81 views

Data transformation for count data with many zeros

I have a count dataset that contains many zeros and a discrete variable that contains many zeros as well. I would like to see graphically which kind of correlation exists between these two variables. ...
1
vote
0answers
20 views

“Robust” normalization of features from multiple groups and unknown distributions prior to learning

I'm working on a machine learning project involving statistical analysis (and later discriminatory classification) of different proteins (samples) drawn from multiple, potentially overlapping classes ...
1
vote
0answers
35 views

How deal with exponential distribution of data during principal component analysis

I am trying to do PCA on a series of variables (all are positive, real numbers) using correlation and varimax rotation. All the computation is done in R. Although I got high loadings for all ...
2
votes
1answer
62 views

Interaction term in a linear log model

I am using a linear-log model to test whether overseas development assistance and remittances positively affect FDI in cases of good governance and financial market development. Let's say I want to ...
4
votes
1answer
71 views

Alternatives to one-way ANOVA for heteroskedastic data

I have data from 3 groups of algae biomass ($A$, $B$, $C$) which contain unequal sample sizes ($n_A=15$, $n_B=13$, $n_C=12$) and I would like compare if these groups are from the same population. ...
1
vote
1answer
36 views

Calculating mean and variance with logarithmic sample weights

I have run into a problem that must be pretty simple, but I keep getting snagged somewhere. I have an algorithm that returns a sample and the logarithm of the sample weight (which get themselves ...
0
votes
0answers
17 views

How can Gretl data be exported to use in Stata [migrated]

Very ingenous question I guess, But, we are in a class of introductory econometrics and need export data from gretl in a way that these poor stata users can import it. Seems that there is a tool to ...
0
votes
0answers
101 views

expressing this probability distribution over different variables

I have a likelihood function as follows: $$ P(y|x,w, \phi) = \frac{\phi}{2\pi} \exp ^{-0.5 (y-t(x, w)'\phi (y-t(x,w)) } $$ Here $y$ and $x$ are two observed values. $\phi$ is also some given ...
1
vote
1answer
32 views

Data Preparation – Historical yearly data rolled up or kept as separate observations

I have a data set regarding audit and tax takes which I want to do some analysis (possibly clustering and predicting who to audit in the coming year). The data ranges from 2009-2013. I have (up to) ...
1
vote
1answer
94 views

Does one lose information or precision when calculating effect sizes from statistical values rather than from the raw data?

Let's say that I want to compute the effect size of a certain intervention, for which I have both the raw data and some statistical value that can be transformed to an effect size. For example, if one ...
1
vote
0answers
36 views

OLS with a heavily skewed independent variable

I am regressing a log-normally distributed dependent variable (wage) on a heavily skewed independent variable and I want to make sure I handle it in the best ...
1
vote
0answers
38 views

Asymptotic Distribution of beta distribution parameter

I am trying to create a t-interval for the first shape parameter of a Beta distributed variable based on a random sample. $ f(x;\theta,\beta) = ...
0
votes
1answer
50 views

What is a good transformation for data that looks like an S on the Q-Q plot? Or a good nonparametric alternative for correlations?

I am trying to do a study to determine if average annual temperature is related to number of cases of a particular disease. I have data for 15 different states over ten years. I have done multiple ...
3
votes
5answers
625 views

How do I get “V-shaped” distributed random numbers from uniformly distributed numbers?

I have 1000 uniformly distributed random numbers. How do I manipulate them to get a V-shaped histogram?
0
votes
0answers
22 views

Log transforming dependent variables

Does anyone have a clear explanation or reference as to why it is only important to transform the dependent variable and not necessarily the independent variables for improved skewness and kurtosis ...
0
votes
0answers
38 views

three variables chemical data analysis optimization

Could anybody help me by giving a solved step by step example of response surface methodology (RSM) using Doehler design for leaching of metals or for chemical processes? My supervisor has given me ...
1
vote
0answers
49 views

Generalized $R^2$ for average model

I have some power-law data sets coming from an Ecology study. Some of them seem to be best modeled using linear regression after log-transformation of the data, but other data sets seem to be best ...
0
votes
0answers
8 views

Do I need to do log transformations for probit?

I am doing a (Heckman) probit. Some of my variables (e.g. N of employees, sales) are highly skewed (skew>2000). Do I need to take log-transformation to make them closer to normal? Thanks!
2
votes
0answers
23 views

Generating a categorical variable from an imputed variable

I am using multiple imputation to impute a continuous variable ($X$) with $\approx30\%$ missing values. I have a question regarding the generation of a new categorical variable ($Y$), starting from ...
1
vote
1answer
21 views

Creating a translated sample by subtracting mode

Consider a sample $x_1,...,x_n$ which we will call the original sample. To create a translated sample $x_1+c,...,x_n+c$, we add a constant $c$ to each sample point. Let $y_i=x_i+c, ...
2
votes
2answers
55 views

Should I log the quotient of 2 variables, or use the quotient of 2 logged variables?

I'm analyzing risk to drivers, i.e. driver deaths/distance driven. Over time distance driven increases (people drive more) while ceteris paribus driver deaths decline (vehicles are safer.) To deal ...
15
votes
4answers
943 views

Pitfalls to avoid when transforming data?

I achieved a strong linear relationship between my $X$ and $Y$ variable after doubly transforming the response. The model was $Y\sim X$ but I transformed it to $\sqrt{\frac{Y}{X}}\sim \sqrt{X}$ ...
0
votes
0answers
33 views

Conversion rate clarification

This is a very simple question, but for some reason I cannot wrap my head around it... It's actually a bit weird because it's so simple that I'm kind of laughing that it's not making sense. On my ...
0
votes
0answers
28 views

Can I normalize a data proportion?

Is there any way to normalize this variable (attached)? In fact, this is a proportional data (a percentage). But I need transform it in order to do some contrasts. I try with arcsine transformation: 2 ...
1
vote
0answers
37 views

Zeros in non-negative data: can you multiply by 10^c, round, and then model as poisson GLM?

Potentially stupid question: is there anything wrong with modeling zero-containing non-negative data by first multiplying it by 10^something, then rounding it, and then modeling it as poisson? Or is ...
2
votes
3answers
61 views

Exponential of a standard normal random variable

We know that $Z\sim N(0, 1)$. How do I prove that $e^Z$ has a mean of $e^{0.5}$? I have tried integrating $e^z$ times the pdf of $Z$ but I don't know why it isn't working out. Also what is the pdf ...
1
vote
1answer
71 views

At what point to categorise a continuous variable (and method)?

My dataset contains a lot of variables that appear to me as practically categorical on a continuous scale to differing degrees. Many have a large chunk of zeros or specific value followed by one or ...
2
votes
1answer
149 views

'Uniformization'?

I am looking for a better term for what I call 'uniformification', where I change data to make it more close to uniformly distributed. I am doing a project in which I try to make the output of a ...
0
votes
1answer
51 views

I need to convert a yearly data into a quarterly and monthly data?

Can anybody please help me convert yearly data into monthly and quarterly data?
0
votes
0answers
26 views

How to reduce the dimension of a test data and make it uncorrelated?

I am working on classification of 16000 cell images. Each of them consists of 706 features related to intensity, morphology, colocalisation and texture of the cell. The train set consists of 11 ...
2
votes
2answers
79 views

Fitting a log (or generalized?) linear model

I would like to fit a model of the form $z = k x^\alpha y^\beta$ to some data I have (it's a spatial gravity model). Now I know you could take logs of both sides and fit a linear regression $\log ...
14
votes
5answers
213 views

How to keep exploratory analyses of large datasets in check?

When I start an exploratory analysis on a large data set (many samples, many variables), I often find myself with hundreds of derived variables, and tonnes of different plots, and no real way to keep ...