Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [normalization]

Usually "normalization" means re-expressing data to make values lie within a specified range.

0
votes
1answer
10 views

Normalizing value using 3 points / levels

I am using following formula to normalize number: new = ((current - low) / (high - low)) * 100 But i need middle number to be the mode of the data. So in this ...
1
vote
1answer
43 views

Should I discard a feature whose max - min is smaller than 1e-5?

I'm training a neural network for regression. The input vector consists of $140$ entries. For each feature vector entry, I calculate both min and ...
1
vote
1answer
14 views

On masked multi-head attention and layer normalization in transformer model

I came to read Attention is All you Need by Vaswani. There two questions came up to me: 1. How is it possible to mask out illegal connections in decoder multi-head attention? It says by setting ...
0
votes
0answers
11 views

Should Feature Selection be Done Before or After Feature Scaling?

I read feature selection should be done before scaling but did not find the source to be too reputable. Should Feature Selection be Done Before or After Feature Scaling? Why?
0
votes
1answer
20 views

Log-normalization of predictors

I have the following dependent and independent variables for my linear regression model. Since they are all in different scales (some of the are % others continuous variables), I was suggested to take ...
0
votes
1answer
24 views

How to normalize Medical data to further use in Electronic Health Records? [closed]

How can we normalize medical data extracted from patients to be used later in Electronic Health Records? Example for data: Age, temperature, time, blood test. I have come across several papers that ...
0
votes
0answers
10 views

Normalization for different data entry to the same scale?

Let's assume I have three data set. These entities are in terms of particle counting. $$[1] = 10 \times 10^{15}$$ $$[2] = 10 \times 10^{13}$$ $$[3] = 10 \times 10^{10}$$ Since the dataset [3] has ...
0
votes
0answers
16 views

What's the rule of thumb for when to normalize and when to standardize? [duplicate]

I'm reading about normalizing data for predictive analysis. on Statistics how to, they write (https://www.statisticshowto.datasciencecentral.com/normalized/) Normalization usually means to scale a ...
0
votes
0answers
20 views

Normalizing data in a tailed distribution

This is a follow-up from a previous question that I resolved on here earlier. If a have a data set that is essentially gaussian, I can normalize the data using: (x - mean)/std which gives me new ...
0
votes
0answers
32 views

PCA: should standardization be applied on features or samples?

I am struggling a little bit with PCA. I understand that standardization is an important part of the algorithm but I do not understand which elements should be standardized. Let's say I have a 10x100 ...
0
votes
1answer
17 views

Normalizing variables before clustering

I am looking to apply k-means clustering on two features of remote sensing data. The first layer is the Normalized Difference Vegetation Index (NDVI), which is expressed on a scale between 0-1. The ...
0
votes
2answers
22 views

Beta distribution and normalization [duplicate]

Here in the 4 pictures in the last answer, is the vertical axe the probability? I.e. it seems to me that it is somewhat unnormalized : it has the value 2 in the 2nd picture and 3 in the 3rd picture. ...
0
votes
0answers
17 views

NASA TLX Analysis

I am trying to analyze a NASA TLX. The NASA TLX is the de facto standard for subjective workload task load. In my experiment (n=27), participants drove a track in a driving simulator while ...
0
votes
0answers
9 views

How to un-normalize the output of a GAN's generator in tensorflow

I am using the DCGAN framework to learn some specific image distribution as part of a bigger process (compressed sensing using GANs). The problem is that the ouput of the net is in the interval $[-1,1]...
2
votes
3answers
467 views

How to normalize data between 0 and 1?

I have seen the min-max normalization formula in several answers (e.g. [1], [2], [3]), where data is normalized into the interval $\left[0,1 \right]$. However, is there a method to normalize data ...
0
votes
0answers
24 views

P-value for a biased estimator

I am asked the following: Let $X_1,X_2,X_3,X_4$ be $n$ observations, each of which is randomly drawn from normal distribution with mean $\mu$ and variance 25. Consider the null hypthesis $H_0: \mu \...
0
votes
0answers
14 views

Normalization of matrices

I have a dataset of matrices which I want to normalize in such a way that I can feed them into a neural network. On this site I found the following formula to do that: $X \rightarrow \frac{X-\min(X)...
0
votes
0answers
7 views

Normalization of augmented data?

I just came across this question which is about the order of augmentation and normalization. It seems that it does not make any difference if I first do the augmentation and afterwards the ...
0
votes
0answers
13 views

Determine performance difference after software changes

Situation: For our software package, I want to check if there is performance degradation after software changes. Therefore we measure multiple operations and save the duration in milliseconds. There ...
0
votes
0answers
36 views

What is the effect of different goalposts values on MinMax normalization of composite indicators?

I want to compute a synthetic index based on several indicators for one country on a time period. The indicators have different ranges and units. I want to normalize them using a MinMax ...
1
vote
1answer
20 views

Normalize dataset considering future possible values

My question is regarding the normalization or standarization of a dataset that currently has a specific Min and Max value but it is previously known that it might vary and have a lower Min and/or a ...
0
votes
0answers
15 views

TF-IDF Normalisation When Comparing Corpora Similarity

I'm computing the similairity between two text corpora by summarising document-wise distance in some way, wherein each document of each corpora is represented as a TF-IDF vector. My question is the ...
0
votes
0answers
15 views

Neural networks - explanation of normalization of data

I'm beginning with neural networks. Currently I'm struggling with fitting to a data-set, which has a large variance both in input variables. It looks like this: ...
0
votes
0answers
32 views

Is it okay to use non-parametric tests on normalized data?

I'm doing experiments with 3 conditions: A, B, and C. I have done the experiment 3 times, so each condition gets 3 values for a total of 9 values. Each value is actually an average of 5 measurements. ...
0
votes
1answer
40 views

Normalization of inputs in convolutional neural networks

I read that using convolutional neural networks, or any neural networks (?), that the input/features should be normalized. The normalization is typically done for each feature $$x_i \in X \ \ \forall ...
2
votes
0answers
29 views

How can I normalize truncated variables for a neural network?

Generally, I normalize variables using standard normal variates or (x-xmin)/(xmax-xmin) But this only works well for variables that are not truncated, for example ...
-1
votes
0answers
16 views

Significant difference of small count data in 2xn contingency tables

My data consists of multiple contingency tables (2xN, often N > 30) with frequencies of two nominal variables: they are frequencies of shared collocations of various word-pairs taken from the corpus. ...
0
votes
0answers
9 views

Standardize gpx data (GPS Points) to calculate overall distance and duration

I'm using Strave (www.strave.com) quite a lot for my sport activites (3-5 running/cycling activites per week). Strava offers to download all your activites in raw data; means you could download all ...
0
votes
1answer
33 views

How do I normalize a weighted scoring grid?

I have created a weighted scoring grid that looks something like this ...
0
votes
1answer
28 views

Questions concerning Z-Normalization in Dynamic Time Warping

Here I found this very nice presentation. On page 46 one can read the following: Essentially all datasets must have every subsequence z-normalized. There are a handful of occasions where it does ...
0
votes
1answer
50 views

Should you normalize your training data for Local Outlier Factor

Say I'm using scikits implementation of Local Outlier Factor with euclidean distance being used by the reachability function. My input features are magnitudes apart, so is it advisable that I ...
1
vote
0answers
37 views

Linear Regression Results explained

I have a set of stocks fundamentals, and I want to draw some features out of it. One of the feature I'm trying to create is net income of a certain year divided by the sales for that year. Now, I'm ...
0
votes
0answers
13 views

Normalizing differing rating styles of users (recommender system data)

In user provided ratings (5-star scale) for a recommender system, users' varying rating styles are quite noticeable. For instance, some users like to rate either 1 or 5 (dislike / like). Others are ...
0
votes
2answers
63 views

Scaling separately in train and test set?

It seems to me that when you scale a numeric variable you should do it separately in train and test set. For example, if you have a numeric variable X. Normalized X is : ( X - m(X) ) / s(X). When ...
0
votes
0answers
40 views

Normalize Count Data (Poisson Distributed Data) with many Zero Values

I have a count data set for the number of couns of an alert(over speeding alert) for various vehicles(number_rows=206). Please assume this count data is for vehicles who have covered similar amount ...
1
vote
1answer
30 views

Can feature normalization lead to gradient vanishing problem?

I generate 100000 sample at point 1,2,3,4 and 5 so total there are 5 lac samples. I have to classify data into 2 classes. I want to normalize the data in the range of zero to one. If I normalize a ...
1
vote
1answer
146 views

Do you standardize the data before PCA whitening?

I have a data set ranged in different scales as well as some variables are sparse, for example, ...
5
votes
1answer
50 views

What is the correct way of normalizing data in machine learning?

I generate input data for my model at different temperatures. At each temperature I generate 1000 samples and in each sample I have 16 features. So the shape of array is (1000,16) at each temperature. ...
0
votes
0answers
15 views

Normalizing sparse matrix by mean, should the mean be calculated excluding zero?

I have very sparse matrix (70% sparsity) which I want to normalize by mean. I tried using mean both include and exclude zero. The histogram between count (y-axis) and value (x-axis) shows The value ...
1
vote
0answers
42 views

Why normalize when all features are on the same scale? [duplicate]

So I'm doing the tensorflow tutorial found here: https://www.tensorflow.org/tutorials/keras/basic_classification Basically, my input is a [28x28] matrix (image) that I flatten to a [1x784] vector. ...
0
votes
1answer
37 views

Bounded Anomaly Score between 0 and 1

I am using a KNN anomaly detection approach, where the distance to my nearest neighbor is an indication for an anomaly. I am wondering how I can normalize the score between 0 and 1. I can use a test ...
0
votes
0answers
12 views

Compare scores with different baselines

I am trying to compare the 3 students (Tania, Jack, Zania) below based on their test scores. However, each of them has taken a different test with a different total score. Is there anyway I could ...
0
votes
0answers
25 views

Is z-score normalization with hard caps reasonable?

I am currently trying to train an variational autoencoder that is implemented in TensorFlow. I have a training set with around 25 000 samples which I normalize by using the formula where the $i$ ...
0
votes
2answers
21 views

How do I determine significant differences between patient data normalised per individual to 100% (therefore having no SD) and the response decrease?

Patients have very different levels of hormones etc naturally so when measuring the effect of a drug designed to decrease this hormone we normalise each patients original score to 100 (%). Then we can ...
2
votes
1answer
39 views

How to “normalize” standard deviations?

I'm a computer science guy who's recently moved into Performance Engineering. As part of this job, I now find myself needing to analyze results of tests (duh). However, my lack of statistical ...
0
votes
0answers
50 views

What are the different influences of outliers regarding the feature scaling methods: standardization VS. normalization?

I've come to know that normalization (MinMax scaling) and standardization (Z-score normalization) on data have different influences from outliers in the data. In About Feature Scaling and ...
1
vote
1answer
244 views

AutoEncoders and linear activation output function

This is not a duplicate of the Activation functions for autoencoder performing regression because there is a comment that somebody found a linear activation function but: they never said what it was. ...
0
votes
1answer
22 views

Similarity between 2 profiles (observations). Is it possible to generate a % similarity?

I have multiple profiles for 10 different people. Each person has been measured for 5 different continuous variables of different magnitudes. So my dataframe is 10x5 where each row represents a person ...
0
votes
1answer
33 views

How do you transform coefficients from a linear regression on normalized data in Octave so they can be used with the un-normalized data?

What I am currently doing is normalizing an input matrix, using linear regression with that normalized input data vs. the un-normalized output data, then trying to get the coefficients which should ...
0
votes
0answers
26 views

How to normalize data while training implicit data by Alternating Least Squares

I have an order table stored user buying history. I want to use Alternating Least Squares to do collaborative filtering, but there is no score, which make me have to use product buying count as score....