Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [normalization]

Usually "normalization" means re-expressing data to make values lie within a specified range.

0
votes
0answers
10 views

CNN Feature Normalization: Looking into the Future?

It occurs to me as I'm normalizing the features for my CNN that I am inadvertently taking the "future" into account by normalizing using the min and max of the entire time period. In other words, at ...
0
votes
0answers
20 views

Histogram equalization

I'm trying to equalize the following histogram: I tried histogram equalization, but that didn't seem to work the way I wanted. I got something likes this: Which is equally distributed everywhere. ...
0
votes
0answers
10 views

How to use Cullen and Frey graphs for downstream statistical analysis?

How to use Cullen and Frey graphs for downstream statistical analysis? I do not see any difference in normalized and raw data Cullen and Frey plots! Raw data ...
0
votes
0answers
7 views

Should I normalise my data for PCA, Sammon and SOM mapping? [duplicate]

I do not think I need to log-transform my data as the distribution of the components are not skewed. However, the scale of the components are different (i.e. Age, resting blood pressure, maximum heart ...
0
votes
0answers
6 views
0
votes
1answer
14 views

How to interpret normalized difference score?

Lets say you have 2 variables, reaction time (RT) on an easy task and reaction time on a hard task. I've seen papers calculate a normalized difference score using the following: $$ Z = \frac{RT(hard) ...
0
votes
1answer
21 views

K-means clustering scaling

I have a data set of 70 stores with a sales column (ranging from 50M to 70M) and 39 other features, like age group, income categories etc. I need to find the clusters based off of these metrics. A ...
3
votes
2answers
267 views

Simulating rnorm() using runif()

I am trying to 'simulate' rnorm() using only runif(). I don't know if I should do: sqrt(-2*log(U1))*cos(U2) or ...
0
votes
0answers
15 views

Data transform for time series with value regime

I have a time series data which looks as following: I need to understand the relationship of this data vs. daily temperature. The spikes in data for Jan 2009 and Sept 2009 are caused by certain ...
0
votes
0answers
3 views

Advise on how best to categorize unlabeled normalized continuous data set

First, apologies if my ignorance leads me to misstate anything or frame my question poorly. Let's start with the data set I have available. I have a table with about 200K+ salespeople's ...
0
votes
2answers
24 views

Does it make sense to apply PCA or Z-Score to any dataset?

Suppose we have a given dataset whose variables represent different things. For instance, one of them could represent the time a user spends on the phone while another one can represent the continent ...
0
votes
0answers
13 views

Do γ and β “undo” the effects of batch normalization?

Let H be a minibatch of activations for a layer to be normalized, where activations of each example are in a row of the matrix, and each column represents the activation of a given unit in the layer. ...
0
votes
0answers
35 views

How can I even out the output of the sigmoid function?

I'm applying a sigmoid function to an array of values but the output is very concentrated near 1 (i.e. 25th percentile is 0.999). I think this is because the input array takes on values that are quite ...
1
vote
1answer
28 views

Type of normalization [duplicate]

Suppose you have some vector $[a_1, \ldots, a_n]$ where the terms are non-negative. How do you call the normalization method where you divide each element by the sum (to get a probability vector)? i....
0
votes
0answers
14 views

Help with normalising data that has A LOT of 0s [duplicate]

I recently am analysing my results (behavioural, observation-based data), and I realised that my data are non-normal. No problem, this happens in behavioural data a lot, and I thought I just needed to ...
0
votes
1answer
39 views

How do you normalise a histogram with two peaks?

I have a histogram that looks as such and I want to use it as part of a Linear Discriminant Analysis but the lda requires its variables to have a normal distribution. What kind of transformation ...
0
votes
0answers
16 views

Normalization of LC-MS data

I am working on a metabolomics project. I am very new to the world of metabolomics. I have been provided an LC-MS dataset in which all the injections are done with the same concentration of the ...
0
votes
0answers
8 views

Why is normalising not recommended for a factorial analysis?

I have been told that during a factorial analysis you shouldn't center and/or normalise your design matrix. I.e. you should not mean-center and rescale the columns of your design matrix. However, I ...
0
votes
0answers
9 views

division of of values with standard deviations

I need to normalize results from my experiment to the control in that experiment. I have two repeats from each condition (test and control), and each has a standard deviation. When dividing to ...
0
votes
1answer
20 views

Finding the original weights after data normalisation

Suppose we have vectors $x_1$ and $x_2$, each has ($n$) samples. Both $x_1$ and $x_2$ are my independent variables. Suppose we also have a vector $y$ which has ($n$) samples and is y my dependent ...
1
vote
0answers
26 views

Why we normalize mutual information by square root of entropy?

Usually NMI(P,T) is expressed as $\frac{MI(P,T)}{\sqrt{H(P)H(T)}}$. However, I don't know the reason why $\sqrt{H(P)H(T)}$ is a maximum value for Mutual information. Is there any proof or explanation?...
0
votes
0answers
24 views

Testing normality with Confidence Intervals

I've read an article that talks about transforming non-normal data to normal data. In the article it says: "In this case, normality clearly cannot be assumed; the p-value is less than 0.05 and more ...
0
votes
0answers
10 views

Standardisation (normalising) for feature scaling in non-logged data for within transformation model

I wonder if someone could provide an insight for my case. I am interested in normalising my data to obtain unitless state that will allow me to use them in a quadratic model (not translog). Based on ...
1
vote
1answer
60 views

How would the correlations change if we normalized the data first? [closed]

How would the correlations change if we normalized the data first? ( In Xl miner)
0
votes
1answer
42 views

Difference between (log, square, root) transformation and Normalization

I am confused between the Transformation and Normalization/Standardization, The basic understanding I have is Transformation: will be used in situation when we have skewness in data and to distribute ...
0
votes
1answer
18 views

interpret the intercept and coefficients of normalized input variables in regression model

My target variable of regression model is a rate between 0 - 1. When using the original input variables, it is easy to interpret the intercept, say 0.8, and coefficients of each input variable, say 0....
0
votes
1answer
18 views

Normalized regression coefficients - interpratation

I have data containing several variables. I ran a regression model. Prior to running the model I have normalized the dependent variable Y and the independent variables X1 and X2. After receiving the ...
1
vote
2answers
81 views

Tests of normality - qq and Shapiro-Wilk

I am new to the world of stats... My data had a log normal distribution, so transformed by log to get it nearer normal distribution. This is real-world data. From here I want to establish if my data ...
0
votes
1answer
15 views

Normalizing value using 3 points / levels

I am using following formula to normalize number: new = ((current - low) / (high - low)) * 100 But i need middle number to be the mode of the data. So in this ...
1
vote
1answer
48 views

Should I discard a feature whose max - min is smaller than 1e-5?

I'm training a neural network for regression. The input vector consists of $140$ entries. For each feature vector entry, I calculate both min and ...
1
vote
1answer
95 views

On masked multi-head attention and layer normalization in transformer model

I came to read Attention is All you Need by Vaswani. There two questions came up to me: 1. How is it possible to mask out illegal connections in decoder multi-head attention? It says by setting ...
0
votes
0answers
36 views

Should Feature Selection be Done Before or After Feature Scaling?

I read feature selection should be done before scaling but did not find the source to be too reputable. Should Feature Selection be Done Before or After Feature Scaling? Why?
0
votes
1answer
31 views

Log-normalization of predictors

I have the following dependent and independent variables for my linear regression model. Since they are all in different scales (some of the are % others continuous variables), I was suggested to take ...
0
votes
1answer
27 views

How to normalize Medical data to further use in Electronic Health Records? [closed]

How can we normalize medical data extracted from patients to be used later in Electronic Health Records? Example for data: Age, temperature, time, blood test. I have come across several papers that ...
0
votes
0answers
19 views

What's the rule of thumb for when to normalize and when to standardize? [duplicate]

I'm reading about normalizing data for predictive analysis. on Statistics how to, they write (https://www.statisticshowto.datasciencecentral.com/normalized/) Normalization usually means to scale a ...
0
votes
0answers
21 views

Normalizing data in a tailed distribution

This is a follow-up from a previous question that I resolved on here earlier. If a have a data set that is essentially gaussian, I can normalize the data using: (x - mean)/std which gives me new ...
0
votes
0answers
40 views

PCA: should standardization be applied on features or samples?

I am struggling a little bit with PCA. I understand that standardization is an important part of the algorithm but I do not understand which elements should be standardized. Let's say I have a 10x100 ...
0
votes
1answer
27 views

Normalizing variables before clustering

I am looking to apply k-means clustering on two features of remote sensing data. The first layer is the Normalized Difference Vegetation Index (NDVI), which is expressed on a scale between 0-1. The ...
0
votes
2answers
27 views

Beta distribution and normalization [duplicate]

Here in the 4 pictures in the last answer, is the vertical axe the probability? I.e. it seems to me that it is somewhat unnormalized : it has the value 2 in the 2nd picture and 3 in the 3rd picture. ...
0
votes
0answers
22 views

NASA TLX Analysis

I am trying to analyze a NASA TLX. The NASA TLX is the de facto standard for subjective workload task load. In my experiment (n=27), participants drove a track in a driving simulator while ...
0
votes
0answers
21 views

How to un-normalize the output of a GAN's generator in tensorflow

I am using the DCGAN framework to learn some specific image distribution as part of a bigger process (compressed sensing using GANs). The problem is that the ouput of the net is in the interval $[-1,1]...
2
votes
3answers
1k views

How to normalize data between 0 and 1?

I have seen the min-max normalization formula in several answers (e.g. [1], [2], [3]), where data is normalized into the interval $\left[0,1 \right]$. However, is there a method to normalize data ...
0
votes
0answers
25 views

P-value for a biased estimator

I am asked the following: Let $X_1,X_2,X_3,X_4$ be $n$ observations, each of which is randomly drawn from normal distribution with mean $\mu$ and variance 25. Consider the null hypthesis $H_0: \mu \...
0
votes
0answers
14 views

Normalization of matrices

I have a dataset of matrices which I want to normalize in such a way that I can feed them into a neural network. On this site I found the following formula to do that: $X \rightarrow \frac{X-\min(X)...
0
votes
0answers
11 views

Normalization of augmented data?

I just came across this question which is about the order of augmentation and normalization. It seems that it does not make any difference if I first do the augmentation and afterwards the ...
0
votes
0answers
14 views

Determine performance difference after software changes

Situation: For our software package, I want to check if there is performance degradation after software changes. Therefore we measure multiple operations and save the duration in milliseconds. There ...
0
votes
0answers
44 views

What is the effect of different goalposts values on MinMax normalization of composite indicators?

I want to compute a synthetic index based on several indicators for one country on a time period. The indicators have different ranges and units. I want to normalize them using a MinMax ...
1
vote
1answer
24 views

Normalize dataset considering future possible values

My question is regarding the normalization or standarization of a dataset that currently has a specific Min and Max value but it is previously known that it might vary and have a lower Min and/or a ...
0
votes
0answers
48 views

TF-IDF Normalisation When Comparing Corpora Similarity

I'm computing the similairity between two text corpora by summarising document-wise distance in some way, wherein each document of each corpora is represented as a TF-IDF vector. My question is the ...
0
votes
0answers
19 views

Neural networks - explanation of normalization of data

I'm beginning with neural networks. Currently I'm struggling with fitting to a data-set, which has a large variance both in input variables. It looks like this: ...