Questions tagged [normalization]

Usually "normalization" means re-expressing data to make values lie within a specified range.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Standardize species richness by elevation

I am studying the effect of forest structure on recruitment. One of the variables is species richness defined as the number of species. The aim is to quantify the effect of species richness on ...
0
votes
2answers
75 views

Normal distribution, is mean=0 and std_deviation=1?

Am I correct to say that following the formula $ f(x) = \frac{1}{\sigma \sqrt{2\pi} } e^{-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2} $ represents a distribution with mean $\mu$ and standard ...
0
votes
1answer
17 views

Why is non-centered SVD accepted in LSA

In Latent Semantic Analysis (LSA) , we apply SVD to a term-document matrix $A$, then choose to ignore all but $k$ largest singular values. The term-document matrix is not centered, or normalised, ...
0
votes
0answers
8 views

Scaling test set based on training will cause test set to have values greater than the scale

I have a time series data that does not have an upper limit (data is somewhat monotonically increasing). Making the Test set values larger than the training set. (I am not shuffling because time ...
0
votes
0answers
8 views

standardizing output of neural network and calculating likelihood

assume that we have $y_{\text{train}}$ and transform it using $\hat{y}_{\text{train}} = \frac{y_{\text{train}} - \mu}{\sigma}$ for some $\mu$ and $\sigma$. Assume that neural network $n(x)$ predicts $...
0
votes
0answers
5 views

Cancel effect of external factor from dataset

In my problem I have a dataset with features: 1,2,3,...,n and another variable z, that is not part of my dataset, but every sample in the dataset has a corresponding z value. I might even see a trend ...
2
votes
1answer
19 views

Does normalization also help to prevend the vanish/exploding gradients?

I am implementing my own neural network from scratch using numpy. I tested my code with the MNIST dataset and I forgot to normalize the images and my code did not work, because I got an error about a ...
1
vote
0answers
25 views

how to z-normalise time series?

Suppose I have 1000 samples of time series, every one of which has 150 points.(If sample frequency is 150 Hz, then every one of the time series stands for 1s.)What is the correct way to z-normalise ...
1
vote
0answers
11 views

Normalizing product of ordered categorical CDFs

Let $\mathbf{p} = [p_1, \ldots, p_K]^\top$ and $\mathbf{q} = [q_1, \ldots, q_K]^\top$ define categorical distributions over a set of classes $\mathcal{C}_1, \ldots, \mathcal{C}_K$. Define $P_i = \sum_{...
0
votes
1answer
27 views

How to normalize skewed data before clustering?

My question deals with what is the right way to normalize my data. My data consists 6 features, all together representing a state in an environment for reinforcement learning. My goal is to cluster ...
0
votes
0answers
13 views

Transformation of a fixed effect?

I'm working with a data set where I relate the response variable weight gain/loss (it goes from -110 g to 150 g) to multiple explanatory variables. It looks like this: lmer (weight.difference ~ A * B ...
1
vote
0answers
49 views

Standardize first four moments: match sample moments with population moments

Let $X$ be a sample from $N(0,1)$ and $m$, $v$, $s$, $k$ denote sample mean, variance, skewness and kurtosis of $X$. I want to transform the sample $X$ such that the sample moments equal the true ...
0
votes
1answer
11 views

Is there any reason for normalizing the error/loss with respect to the number of data points?

In a lot of the models for the error for supervised learning, the expression appears as $$e(w) = \dfrac{1}{N} \sum\limits_{i = 1}^N L(w,t^i, x^i)$$ where $t^i, x^i$ are the $i$th target and example ...
0
votes
2answers
35 views

how do you transform/standardise a function to always give values between y1 and y2?

Having lost some of my math skills, I am having problems with something that I think should be fairly easy but is eluding me: I have a plateau shaped function that I would like to standardise such ...
0
votes
0answers
10 views

Do percent variables need to be scaled for machine learning?

I have 101 independent variables in a logistic regression predictive model. 55 variables are continuous and 46 are categorical one-hot encoded. 5 of the 55 continuous variables are expressed as ...
1
vote
0answers
17 views

Normalization and standardization

I want to use techniques of normalisation and standardisation to pre-process my data in readiness for ML algorithms. My understanding is that this boils down to: (1) squashing my data within a range (...
0
votes
0answers
21 views

How to preprocess performance counter input data for anomaly detection using autoencoders

I am working with more than 250 input features that include system performance counters and SQL Server database counters to predict anomalies / system outages. I am looking to use an autoencoder ...
0
votes
1answer
13 views

Stnadard deviation of values containing different ranges

Taking the $\sigma$ of these values : np.std([55,50,40,45]) returns: 5.59 If I take the $\sigma$ of these values: ...
0
votes
2answers
18 views

Validation of Models with Different Scale Data

I created a model for two different datasets that have different scales. When checking which one performed better, I am struggling with figuring out the best methodology. My top choices right now are <...
1
vote
0answers
11 views

Validation of Models with Different Scale Data [duplicate]

I created a model for two different datasets that have different scales. When checking which one performed better, I am struggling with figuring out the best methodology. My top choices right now are <...
0
votes
1answer
33 views

Normalize target value for linear regression

I'm building a regression model to predict sensor value over time. Bellow is a figure of my sensors data over time: Based on this video about transforming nonlinear data with a log function, What ...
0
votes
0answers
20 views

Min Max Normalization [duplicate]

What is the difference between Min Max and z score normalization.
1
vote
1answer
27 views

Normalise different thresholds for binary prediction

I'm working in a module that outputs the risk of an event happening i.e. risk of a crime happening depending on the district of the city. What I've done is to calculate for each district a binary ...
0
votes
0answers
11 views

Handling fractions when normalizing a set

I currently have the following normalization function: ...
0
votes
0answers
15 views

RMSE normalization. Number of bins

I am using RMSE (Root mean squared error) as a measure of goodness of fit. I am fitting a formula to binned data. The number of bins is not fixed: if there are less than 5 data values in a certain bin,...
0
votes
1answer
17 views

Can we use the results achieved in forecasting after employing min-max normalization on data?

I am working on time series forecasting with univariate data. After applying min-max normalization, I am getting results in terms of Mean Error, Root Mean Square Error etc, less than 1 of course and ...
0
votes
0answers
24 views

What is the second derivative (Hessian) of normalization function?

The normalization function over a $n$ elements column vector $\mathbf{x} $ is described as: $\frac{\mathbf{x}}{\|\mathbf{x}\|_2}$。(x divide by its L2-norm) The gradient of x is written as : $$\frac{\...
2
votes
2answers
90 views

Monte Carlo Integration and pdfs?

Let's say I have an un-normalized probability density function $f(x)$, which is related to $\xi$ via $\xi = \frac{f}{c}$ I also have a sample set $S = \{x_i\}_{i=1}^n \sim \xi$ which is sampled from ...
0
votes
0answers
5 views

what are the implications of standard normalizing non gaussian distributed data?

Many Bayesian regression neural network models and experiments in the corresponding papers (example: https://arxiv.org/abs/1705.07832) use standard normalized data $\frac{x - \mu}{\sigma}$, but the ...
0
votes
1answer
23 views

Does centering data destroy information if the data are not gaussian distributed?

I've alway wondered if you are given tabular data where things have different distributions (some features are normally distributed, some are log-normally distributed) if centering the data by... $$ \...
0
votes
1answer
22 views

Normalise variables for calculation of correlation coefficient

I have a dataset showing monthly stock index returns for the last twenty years across 7 regions. Each region has 6 stock indices( both growth and value stock indices for small cap, standard cap and ...
0
votes
0answers
16 views

Normalising time series data by group

I trained the LSTM model to forecast the number of people infected with COVID-19. Since each country and state (geo location) has different number of population as well as the infected over time, I ...
0
votes
0answers
22 views

Normalization and Standardization of variables before clustering [duplicate]

my data set has got 821.000 rows and 18 columns. It is about online clickstream behavior. My variables are number of shopping baskets, number of items in the shopping basket, number of product pages ...
1
vote
2answers
31 views

Normal distribution question

I encountered a question here: Sodium content of hot dogs are normally distributed with mean = 140 mg and standard deviation = 8 mg. If we randomly select 49 hot dogs, what is the probability that ...
0
votes
0answers
10 views

How to improve neural network training against a large data set of points with varying magnitude

I am currently using TensorFlow and have simply been trying to train a neural network directly against a large continuous data set, e.g. $y = [0.014, 1.545, 10.232, 0.948, ...]$. The loss function in ...
0
votes
1answer
16 views

Batch Normalization vs Full Whitening

In section 3 of the Batch Normalization paper, it says Since the full whitening of each layer's inputs is costly and not everywhere differentiable, we make two necessary simplifications. I ...
2
votes
0answers
12 views

Normalized statistical measure for agreement of biological replicates

I've conducted an experiment collecting samples from various depths in the ocean. At each depth, I collected 3 samples to act as biological replicates. I now want to show that all of my replicates ...
0
votes
0answers
4 views

Creating predictions from RNN model built on scaled data

As the title says, how would I create predictions from a RNN model trained on scaled (-1,1) data, especially when the test data/real world data isn't in the range of the non scaled data?
0
votes
0answers
23 views

How do I apply Min Max scaling for numerical forecast when both dependent and independent volumes are increasing over time?

I'm want to build a numerical regression to forecast. From my initial analysis, it shows linear models (glm) out performs the typical decision tree models (xgboost, ranger...etc). I hypothesized that ...
1
vote
0answers
28 views

Data normalization for linear regression

I'm currently writing my thesis on the comparison of ultrasound measurements with DEXA scan measurements for specific fat distributions (40 participants). I would like to perform a linear regression ...
0
votes
0answers
10 views

Normalization and positive reward in PPO

What does normalization of inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This means ...
0
votes
0answers
45 views

How to normalize multivariate data for neural networks?

I am training a neural network with several different types of data, which are difficult to put together. Some of them are integers (e.g. age), or binary (e.g. gender). Others are integers which ...
0
votes
0answers
9 views

Is there a way to normalize my data of multiple groups to use as a random effect or to incorporate it into my model

I have Retention Efficiency(RE) percentages of 5 food types for sponges. RE is found by ((incurrent food - excurrent food)/incurrent food). I am running a generalized mixed effect model to look to see ...
0
votes
0answers
8 views

Priority between feature engineering and normalisation

My question is related to the priority between feature engineering (for example a simple transformation) and normalisation. It is a general question and I am not sure I understand all the ...
0
votes
0answers
20 views

Mean Normalization vs Standardization (Z-score Normalization)

I am training a neural network on the MNIST dataset. When normalizing the data, I used standardization, but later on did not get the desired accuracy. I tried different learning rates and more neurons ...
0
votes
1answer
14 views

how to plot different proportions (different sample sizes/different denominator) in a pie chart

I have 5 samples from Asia, Africa, Europe, Oceania and America of different sample sizes. I am looking for particular mutations in a gene in these samples and I got the following proportions ...
1
vote
2answers
32 views

Should I normalize all data prior feeding the tensorflow models?

Appreciate your wisdom on this, My understanding is most of the tutorials recommend normalizing / scaling the data prior feeding the tensorflow models. Doesn't normalization require that data ...
0
votes
0answers
10 views

Normalization of RNA seq data

Currently, I am analysing sc-RNA sequencing data. As far as I know, there are several normalization methods available when differential gene expression analysis is performed. However, in my case, I ...
0
votes
0answers
25 views

Why is normalization not nescessary for random forests?

I think I remember that normalization is not nescessary for random forest classifiers. Why is this? In each individual tree when calculating Gini impurities to determine cutoffs in a numerical node, ...
0
votes
0answers
9 views

How to standardize/normalize 16 bit images with a small standard deviation

I'm trying to detect silhouettes on thermal 60x80 and depth 480x640 images by using the SSD model. I have good results on thermal images, but realy poor results on depth images. I started to ...

1
2 3 4 5
20