A way of re-expressing data to make their values lie between 0 and 1 (or 0% and 100%).
5
votes
1answer
179 views
When doing quadrat counting, how do you construct the quadrats?
I want to perform quadrat count analysis on several point processes (or one marked point process), to then apply some dimensionality reduction techniques.
The marks are not identically distributed, ...
1
vote
1answer
103 views
Normalizing when doing count statistics of patient numbers on different arms of a clinical trial
I'm looking at clinical trial data where there are various numbers of patients on different arms (A, B, C, D). All arms receive a drug that is known to cause a toxicity, but some arms receive an ...
2
votes
2answers
58 views
Why Normalization (Standardization) values should be smaller than $1$?
The books gives some examples about content based recommendation. An example of what I understood is at below.
A movie's attributes are values between $1$ and $10$. The duration attribute gets ...
4
votes
3answers
102 views
Should you ever standardise binary variables?
I have a data set with a set of features. Some of them are binary (1=active or fired, 0= inactive or dormant) and the rest are real valued, i.e. 4564.342.
I want to feed this data to a machine ...
2
votes
1answer
97 views
How to compensate for small errors that could greatly distort observed ratios?
Lets say for two samples, treatment and control, there are three constituent molecules each and their corresponding amounts are as follows:
...
6
votes
3answers
3k views
Column-wise matrix normalization in R [closed]
I would like to perform column-wise normalization of a matrix in R. Given a matrix m, I want to normalize each column by dividing each element by the sum of the ...
0
votes
0answers
65 views
Transform non-normal data to normality by rescoring columns
I have a vector with fluency of translation - (0, 0, 1, 3, 3, 3 ,3 ....)
The problem is that it is made by people (for example someone gives too much 3 but only a bit of 2) and we want to normalize ...
1
vote
1answer
67 views
Normalize a periodic parameter
I am using inverse modelling software (PEST) to estimate a periodic parameter for the direction of anisotropy, $\hat{\theta}$, which is somewhere in $[0^{\circ}, 180^{\circ})$ (i.e., has a wavelength ...
1
vote
1answer
29 views
Normalization of three different variables for linear addition
I am a research student and stuck at a point in my work and want your help. I have three different variables for each node of my network (Energy, Traffic Load and Link quality). These three variables ...
1
vote
1answer
71 views
Z scores derived from a regression equation in one group applied to other groups
I work in the developmental psychology arena.
Often papers report using regression to derive a function of age onto a score in a typical group and then applying this function to a second atypical ...
1
vote
1answer
36 views
How to perform Normalization on Call Details Record to perform k-Mean Clustering
I'm new to data mining and currently doing mining project on telecom customer segmentation (based on profile and call details record). I have gender, age, call time and call duration and have to ...
1
vote
1answer
148 views
Comparing the result of a study which has unequal group sizes
I have conducted two user studies and in my studies I didn't have control over the group sizes. In each study users were put in groups and they were asked to perform some group activities. Here is the ...
0
votes
1answer
131 views
Converting spectral data to RGB and normalizing appropriately
I have spectral data which tells me how strong of a response in RGB values for some color-space I get for a particular wavelength of light.
Typically, if I want "plain" white light, I'd have to sum ...
0
votes
1answer
29 views
Data normalization before giving to Neural Nets or Deep Learning algorithm?
What kind of normalization scheman is required for the best of NN algorithms? I saw some people just give the data to signum function before passing to NN and some of those process data by regular ...
1
vote
1answer
35 views
Reshaping a distribution
Not sure what the exact term is for what I'm trying to do.
I have a data set with random variable x with values X1, X2, ..., XN that has a standard deviation sigma and a mean m.
I want to perturb ...
2
votes
1answer
27 views
When to normalize learning?
I'm trying to determine the effect of three types of learning on a group of subjects.
I have their pretest scores and posttest scores.
The current goal is to determine which intervention reduce the ...
2
votes
1answer
51 views
Confusion related to data normalization
I am trying to learn a linear regression model. However, I have some confusion related to the normalization of the data. I have normalized the features/predictors to zero mean and unit variance. Do I ...
1
vote
1answer
51 views
Normalization Factor Wrong? (Bug?)
I'm new to PyMC and Bayesian stuff in general, so I started off with what I thought was a very simple toy problem. I generated some normally-distributed noise with a given mean and standard ...
1
vote
0answers
21 views
getting rid of the batch effects
I have a two data set one with n=15 another with n=25.Each set of data has around 90 variables.The data collected was experimental so when i try to combine my two datasets i see clear differences in ...
5
votes
4answers
2k views
What are the primary differences between z-scores and t-scores, and are they both considered standard scores?
We are currently converting student test scores in this manner :
( ScaledScore - ScaledScore Mean ) / StdDeviation ) * 15 + 100
I was referring to this ...
0
votes
1answer
27 views
Normalization / Moving Average
I have this daily time series of observed prices: $P_1,P_2,..., P_n$.
I want to works with returns: $ 0 , P_2-P_1,..., P_n - P_{n-1}$.
I have been told to "remove" the first term (P_1-P_0= P_1- ?) ...
0
votes
1answer
102 views
Dynamic Time Warping and normalization
I'm using Dynamic Time Warping to match a "query" and a "template" curve and having reasonable success thus far, but I have some basic questions:
I'm assessing a "match" by assessing whether the DTW ...
0
votes
1answer
50 views
Is there a way in python/java/scala to convert/normalize log normal distribution into normal distribution?
We have a data set that looks like a lognormal distribution when we plot it.
We would like to convert/normalize the distribution into normal distribution and see what feature weight got enhanced.
It ...
0
votes
0answers
51 views
Normalizing SVM predications to [0,1]
I have trained an linear SVM which takes a pair of objects, computes features and is expected to learn a semantic similarity function between objects(we can say that it predicts whether the two ...
1
vote
2answers
161 views
How to deal with extreme but “real” data, classify as outliers or no?
I have an explanatory variable, close, which is the daily close price of a firm in the stock market.
The following summarizes this explanatory variable:
...
2
votes
1answer
104 views
LOESS and MA normalization in R?
Attempting to do loess on two variables x and y in R using MA normalization (http://en.wikipedia.org/wiki/MA_plot) like this:
...
3
votes
2answers
232 views
does it make sense for non-negative data to subtract the mean and divide by the std dev?
It is a very usual procedure to subtract the mean and divide by the standard deviation in a set of data. If we deal with non-negative data, i.e. image, (in [0,1] or [0,255]), does this procedure make ...
1
vote
0answers
158 views
Z-scores and normalization in previously transformed data sets
I have two sets of data. The first is multivariate linear regression data consisting of betas and standard errors. It was log(e) transformed before the linear regression model was applied.
The ...
3
votes
1answer
941 views
How should I normalize my accelerometer sensor data?
I'm working with a large set of accelerometer data collected with multiple sensors worn by many subjects. Unfortunately, nobody here seems to know the technical specifications of the devices and I ...
1
vote
0answers
72 views
Test to compare means normalized to different controls
I am running experiments on cells which are given two different treatments. Treatment one is a control virus, and treatment two is an active virus. Due to the variability between cell lines, I always ...
5
votes
1answer
125 views
How can I devise a scoring system for a competition that is more fair than straight percentages?
I am trying to come up with a method for deciding the winner from among eight student groups competing for a prize.
The raw data and corresponding percentages measure participation per group in a ...
0
votes
2answers
116 views
Data normalization and classification
I hope this clearly states the problem I have in hand. Here goes:
I've trained a neural network with one initial data set that was normalized in order to guarantee an equal participation of each ...
2
votes
2answers
155 views
how to avoid 0 determinant when sample covariance matrix has very small values
I have $n$, $p$-dimensional vectors and I am construction the $p$x$p$ covariance matrix using the following formula:
$Cov(j,k) = {1/(n-1)} {{\sum^n_{i=1}} (x_i(j) - {\mu}(j)) * (x_i(k) - {\mu(k)}) ...
1
vote
2answers
208 views
It it legitimate to apply a one-way ANOVA to data that have been normalised to the untreated controls?
I was wondering if it is OK to use one-way ANOVA after normalisation to the untreated controls? This is in an animal model of wound healing where there are 4 wounds per animal, one untreated and 3 ...
2
votes
1answer
105 views
How to calculate expected win percentage where only 2 of 5 values are known?
The idea here is that you have a 5 vs. 5 game where each player is using a unique character (henceforth 'hero'), and thousands of matches of this game have been played. The goal of the analysis is to ...
0
votes
0answers
26 views
Garage Benchmark
I'm using my garage as an experiment in learning how to write a simple benchmarking software but I'm not the best at statistics. Maybe someone can help me here.
My garage is refrigerated 100% of the ...
0
votes
0answers
40 views
Normally and exponentially distributed features
I have several features of which some are exponentially distributed and some are normally distributed.
Can I / should I used both at the same for training a classifier (e.g. neural net) and how ...
0
votes
0answers
114 views
Question about normalizing time series data samples
I have a set of times series biometric data samples for a number of different subjects. In each data sample subjects perform a specified action, but the time spent performing the action differs based ...
1
vote
1answer
193 views
Can I use z-scores to compare results from different measures?
Can I compare outcome data from different literacy measures with different scales (delivered to different subjects) by transforming the data into z scores? Ideally I'd like to use regression and ...
1
vote
1answer
57 views
Limit of quantile normalization
Is quantile normalization adequate for normalizing
data with very few samples?
For example this microarray data. Typically after normalization
we'd like to compare ...
1
vote
2answers
174 views
Confused about proper way to normalize two variables
I have two variables of interest:
Residential Vacancies (res_vac)
Commercial Vacancies (com_vac)
I also have two variables with which I might normalize the above:
Total Residences (res_tot)
...
1
vote
1answer
126 views
Whitening and unwhitening for sparse coding
Is this procedure for whitening and unwhitening correct?
Given an image $i$:
decompose the image in patches:
patch=im2col(i,[8 8],'sliding');
Whitening step:
...
0
votes
0answers
34 views
Does the standard scores analysis make sense in the following scenario?
INTRODUCTION
I've got an image in grayscale.
Say the image is a rectangle with width = 1700 pixels and height = 2338 pixels.
The image is light in the middle and dark in the top and bottom edges.
The ...
3
votes
2answers
176 views
What are good initial weights in a neural network?
I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. ...
1
vote
1answer
170 views
Process for Standardising and Normalising data
First: I'm not well versed in statistics terminology so please forgive me - I'll try to be as verbose as possible with my problem.
This is a problem which I've previously solved very naively. I'm ...
-3
votes
1answer
167 views
On what basis do we differentiate a standard normal distribution from normal distribution?
The normal distribution is a prerequisite for ANOVA. It is not clear what we are supposed to understand by "standard normal distribution". Does ANOVA need a standard normal distribution? The answer ...
6
votes
3answers
529 views
How and why do normalization and feature scaling work?
I see that lots of machine learning algorithms work better with mean cancellation and covariance equalization. For example, Neural Networks tend to converge faster, and K-Means generally gives better ...
4
votes
4answers
471 views
Ratio of Range to IQR vs. Coefficient of Variation — which is the more useful robust measure?
For a given set of data, spread is often calculated either as the standard deviation or as the IQR (inter-quartile range).
Whereas a standard deviation is ...
0
votes
0answers
61 views
Should I normalize for internal standard by taking residuals or by including the standard in the model?
I have some mass-spectroscopy data from several dozen samples, with the abundance of 60 compounds reported for each sample. Four internal standards were run with each sample, and their abundances are ...
1
vote
0answers
67 views
How do you predict the value of new instance, when the training data were normalized?
I estimated a Partial Least Squares model where the X matrix had normalized columns. Now I want to predict the value for a new instance (which is a frequency vector summing to one.) I assume that if I ...
