Questions tagged [normalization]
Usually "normalization" means re-expressing univariate data to make values lie within a specified range.
1,238
questions
0
votes
0
answers
15
views
Normalization by more than one variable [closed]
I want to generate a variable $real \_income$ by normalizing income by population and consumer price index. To do that, I would have to divide by the product of population and consumer price index. ...
0
votes
0
answers
17
views
Can principal components changed by a normalization method be used to construct original data shape with SVD
I'm planning to use an algorithm called Harmony, designed for data normalization, particularly in the context of single cell data analysis. Harmony operates by taking principal components (PCs) as ...
0
votes
0
answers
20
views
How can I normalize Hedges' g from two extreme conditions around the control condition?
The variables I am discussing below are not the ones I am actually using but I think that they should give a better sense of what I am trying achieve.
I am performing a meta-analysis of some chosen ...
2
votes
1
answer
102
views
How to normalise outputs of neural networks with different distribution?
I have a NN model that predicts 8 different variables. I use a multi-task learning approach, where I compute the loss between predictions and targets for each of ...
0
votes
0
answers
28
views
Help interpreting normalized HMM (or otherwise) results
I have run a hidden markov model with five variables on very different scales. Because of this I normalized the input data beforehand using Carets preprocessing:
...
0
votes
1
answer
36
views
Model comparison: with raw or normalized data?
I have developed a index of drug addiction risk whose formula is Index = 1/log10(a_given_variable). The raw values of the calculated ...
1
vote
0
answers
10
views
Do I need to normalize corpus frequency if I am not comparing between corpora?
Do I need to normalize my corpus frequency, given that I am not comparing corpora? For example, if I am going to compare collocations of lexicons A, B and C in a corpus with 13 million tokens, can I ...
0
votes
0
answers
24
views
Normalize age frequency data for PCA
I am working on a project to forecast house ownership rates. One dataset I have consists of number of people of each age from 1-99 per geographic area code. For example, 20 people aged 1, 59 people ...
2
votes
2
answers
113
views
Controlling for number of days when calculating medication adherence
I am trying to evaluate the success of a medication adherence intervention. I am trying to assess the success of the program before and after the intervention in terms of number of days a patient took ...
0
votes
0
answers
31
views
Versatility and validity of aggregate algorithm for a multiple criteria decision problem (averaging)
I'm trying to aggregate different variables in multiple scales and distributions. The distributions are not known beforehand, but I want to generate a general statistical value that represents all the ...
0
votes
0
answers
35
views
Understanding the Normalization/Standardization of geospatial coordinates
I'm building a neural network to predict future [latitude,longitude,altitude], and am having trouble dealing with the features. I've reviewed the answers to the ...
1
vote
0
answers
33
views
Time series normalization
I'm not an expert yet in the field and I have some questions. I have some data of birds and drones taken from a radar. I want to create a classifier that differentiates them. At first I'm trying and ...
1
vote
0
answers
21
views
Normalization for time series comparison
I have a time series Markov Switching model, which is estimated in about 15 different versions. One or two of the time series had to be normalized in order to converge. That is 1-2 out of 15. My ...
0
votes
0
answers
25
views
Normalising series data of variable length
I have a model m which takes as input a starting array of n_input elements which are actually traces of mining processes and ...
2
votes
1
answer
129
views
What are the layer normalization dimensions in transformer?
In transformer training, the activations have three dimensions: batch, feature (i.e. embedding) and time (i.e. token). Layer normalization is applied, calculating statistics (mean, standard deviation) ...
0
votes
0
answers
15
views
Normalize data with a variance-shift in measurement-error to achieve high correlation with true underlying process?
I am discussing the question whether to normalize the data or not in the following setting:
I have a true time series
$$s_t = iid(0,\sigma_s^2)$$
however I only observe different $$\hat{s}_{t,i} = s_t ...
0
votes
0
answers
29
views
Normalization/standardization of time series data
I have energy consumption data where rows represent different users and columns are different measurements. I don't really understand, how and in which order i need to normalize and standardize the ...
0
votes
1
answer
16
views
Interpretation of income rank variable
I have an income rank variable which is calculated using the following formula:
Income rank = (j-1)/(n-1)
where j – 1 represents the number of people within the individuals' comparison group with ...
2
votes
1
answer
29
views
How to Normalise a loss function containing actual values?
I defined a loss function that I am trying to optimise consisting of two elements. let's say a and b:
w1 * (a_actual - a_measured)^2 + w2* (b_actual - b_measured)^2
I am trying to simplify the ...
0
votes
0
answers
40
views
Quantitative Methods for Evaluating Differences Between Two Distributions
I am working with a substantial dataset in which I need to compare the distributions of certain common features across different categories. The challenge I face is that due to the imprecision in ...
0
votes
0
answers
23
views
Interpreting normalized value in logistic regression
I'm trying to interpret results from my logistic regression. I normalized the values between 0 and 100 because I had many variables with different units.
My intercept is -0.741 and I'm using that to ...
0
votes
0
answers
77
views
Normalization of log-returns or normalization of cumulative log-returns
This questions seeks for discussion to find theoretical support for normalizing cumulative log-returns vs normalizing log-returns
By "normalizing" (also known as standarizing) I mean it in ...
0
votes
1
answer
39
views
Transform data domain and maintain mean
I have a dataset, $y$, that is on some arbitrary range. I would like to transform this data to be on the range [-1,1]. This is accomplished using a linear transformation, such as the one described ...
0
votes
1
answer
35
views
Data normalisation for a newbie
I am new to data analysis and I have been given a task to compare safety of trams compared to buses. Since there are far more buses than trams, I was introduced to the concept of normalisation by ...
0
votes
0
answers
14
views
How many number of samples must be taken to ensure no information is lost in normalization?
I was using nn.LayerNorm from Pytorch to normalize my dataset. I tested on a tensor of two points but surprisingly found that my output is almost always the same. Going off the equation of ...
2
votes
1
answer
56
views
Should I apply normalization to predicted probabilities from 7 different models before computing correlation among them?
I'd like to check if there are correlation among predicted probabilities of models in a voting classifier. According to the table below, one of models, Model5, has mean 40.9% and standard deviation 46....
2
votes
1
answer
56
views
Normalizing constant calculation of Strauss Process
Suppose that I have the following Strauss Process up to a proportionality constant
$$p(\mu_{1}, \mu_{2},..., \mu_{K},K)\propto \xi^{K}\prod_{i=1}^{K} I(\mu_{i}\in R) *a^{\sum_{i,j}|\mu_{i}-\mu_{j}|<...
0
votes
1
answer
28
views
Can I transform my output variable in an imbalanced dataset?
I have a dataset that has an output variable that is quite right-skewed and imbalanced. I want to use a neural network as a regressor to predict the output variable. Visually, it looks like there may ...
0
votes
1
answer
138
views
Image pre-processing for Variational Autoencoder
Setting
I am training a Variational Autoencoder (VAE) on the CIFAR10 dataset, which has RGB colors. The VAE uses convolution and transposed convolution layers as well as linear layers to encoder and ...
1
vote
2
answers
48
views
Should I transform my positively skewed predictor in hierarchical regression?
I'm doing a hierarchical regression trying to understand how intelligence (first predictor) and personality traits (second predictor) influence general knowledge (dependent variable). The problem is ...
0
votes
1
answer
33
views
Bias in the normalisation of gene expression due to low counts
I hope it is okay to ask a question because I am one of the silent user or this forum. My background is in biology and I am not so confident in my skills in stats
The goal of the analysis is to ...
0
votes
1
answer
42
views
Normalization of Time Series for GARCH
It is generally observed that financial times series is not normally distributed. So I want to clear the following doubts:
Is it necessary to normalize time series data especially before proceeding ...
0
votes
0
answers
19
views
Can this feature be used as input to cnn based on the q-q plot?
I have normalised my input features and target variable using the quantile transform by sklearn. The Q-Q plot of one of the feature after normalisation is given as below. Is this feature normalised? ...
0
votes
0
answers
79
views
Standard error of mean of normalized scaled data
I had to normalize my data using this equation :
$m ↦ (m - r_{min}) / (r_{max} - r_{min}) * (t_{max} - t_{min}) + t_{min}$
as seen here: scale a number between a range
If I calculate the mean of ...
0
votes
0
answers
24
views
Understanding PCA plot built on data normalized by two normalization methods
I tried two different normalization methods, generated the PCA plot above on the combined data, and colored the samples by the normalization type. Both normalization methods should give similar ...
1
vote
0
answers
19
views
Appropriate tests and normalization approaches for highly non-normal data
I'm trying to ascertain whether a particular type of article, $T$, is associated with higher engagement scores in academic journals, and how substantial the effect size might be. I have raw data of ...
0
votes
0
answers
8
views
Normalised Differences
I have a dataset where my main variable is smoking. I have three definitions of smoking based on different questions and there are overlaps. By the first definition I have 100 people who are smokers, ...
0
votes
0
answers
17
views
Normalizing data when retraining on the train and validation set
It is often good practice to normalize training data for numerical stability and faster convergence. When using a train-validation-test split, it is recommended to calculate normalization parameters (...
1
vote
1
answer
133
views
Normalizing a data set by using z-scores from a related data set
I am trying to come up with a normalization for a set of data. My company puts our products through a series of testers (people) to score our products on key metrics. I am able to access all records ...
0
votes
0
answers
45
views
How to normalize slope of historical financial data?
Im trying to find a method to normalize the slope value for the historical net profits of many companies. But the problem arises when certain companies are clocking profits in thousands, and some in ...
0
votes
0
answers
4
views
How do I minimize the difference between development and production data after normalization relative to measures of central tendency?
I think that normalization of data is performed assuming that the samples you train on are "representative" of real-world or "production" data. This means I have to constantly ...
1
vote
1
answer
204
views
Do I calculate the loss of a regression on the normalized or denormalized data?
I worked on implementing a simple MLP network, which should guess a numerical value based on different values. Basically it's a regression task. This was a few months ago.
Everything worked fine, the ...
0
votes
0
answers
6
views
Comparing quantitative data from a variety of tests
I have a data set comprising about 20 track and fields events, each with about 50 data points from males and 50 from females. Those I've checked so far are normal distributions (within sex).
My null ...
0
votes
1
answer
124
views
In R Limma's `normalizeMedianValues` function, why is its operations preceded with a log transformation followed by an exponential transformation?
In the function normalizeMedianValues in the package limma, column counts are normalised such that their column medians are ...
2
votes
1
answer
151
views
Why it is called "BatchNorm" not "Batch Standardize"?
Regarding the differences between "Normalization" and "Standardization," I found that:
Normalization: Is the process of making a dataset having a specified range, probably [0,1] ...
4
votes
1
answer
306
views
Should you normalize covariates in a linear mixed model
I am using lmer for a set of mixed models, each comparing a protein quantity of interest with a biomarker. Even after experimental batch correction & ...
1
vote
0
answers
83
views
Should standardizing a vector of data lead to a unit length vector? [closed]
I am reading "Introduction to Econophysics" by Stanley and Mantegna and I found the following in Chapter 13.
I can't understand why the 2-Norm of a vector standardized by subtracting the ...
2
votes
1
answer
45
views
How normalizing data cause not problem in prediction?
In algorithms that perform better with data normalization or deep learning problems such as classification, how normalizing data does not bias our algorithm? I mean, in training or even testing, we ...
0
votes
0
answers
12
views
Should the data be scaled before normalisation to enable the use of the model as a pre-trained model?
I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data.
Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I ...
0
votes
0
answers
32
views
Normalisation in feature extraction using pre-trained model
I have a dataset with medical images. I want to implement a network for super-resolution using GANs. One of the criteria of the optimisation is a perceptual loss. For that I will use a pretrained vgg ...