Questions tagged [normalization]

Usually "normalization" means re-expressing data to make values lie within a specified range.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Do random forests normalise data?

My question is about random forests and whether they normalise the data through transformation, and whether this transformation is coherent with the original methodology proposed by a RF package like <...
0
votes
0answers
30 views

Using normal distribution and CDF to normalize data between 0-1

I need to do the following procedure: 1) first need to transform may data to a normal distribution and 2) to calculate CDF curve to finally get my values distributed between 0-1 range, I mean ...
0
votes
0answers
9 views

Why does tf.keras.experimental.preprocessing.Normalization sum over all samples, and why can't this be changed? (Time-series)

From my understanding, when dealing with time-series data, it makes sense to sum normalize the features channel-wise in the time-domain. This means that we treat each channel separately and sum over ...
0
votes
0answers
9 views

Modelling the splitting of resources

I recently came across the following data generating process which I would like to model: A fixed amount of a resources has be to split between a number of receivers with no remainder: In case a model ...
0
votes
0answers
10 views

In what way are the coefficients normalized in a static Difference-in-Difference regression?

I'm a beginner in the field of econometrics. I'm currently working on a Difference-in-difference regression, and I have a (probably a very basic) question in this regard: In what way are the ...
0
votes
0answers
4 views

Gaussian Linear Model with Normalized dependent variable - statsmodels

I used MinMax normalization technique before applying GLM regression to my dataset. I have a few questions: Can I apply such transformation before fitting linear models? Which transformations could ...
0
votes
0answers
25 views

Histograms with Normalized Data?

I have two datasets I am looking to perform some simple visualization on. One dataset is approx. seven times larger than the other. My first step was to compare the distribution of each, and I thought ...
0
votes
0answers
17 views

Should I normalize time series data?

The time series data I have differ in magnitudes but seem highly correlated (and hence cointegrated?). So should I subtract the mean from all values and divide by sd for all values in the time series? ...
1
vote
1answer
37 views

Undo Batch Normalization in NN

I am using 2 BatchNormalization layers in Keras for a huge dataset that does not fit into memory. I can train with normalized values, but since I want to do a ...
0
votes
1answer
40 views

Clustering and why it might not be a good idea to normalize your data?

I came across this post on Quora and the first answer cleared up why it might be a good idea to always normalize your data, but I want to understand what happens conversely. More specifically, what ...
2
votes
0answers
18 views

Analytical distribution of $N$ standardized samples? [duplicate]

When standardizing a set of Gaussian i.i.d. samples $\{x_i\}^N_{i=1}, X_i \sim \mathcal{N}(\mu, \sigma)$, are there analytical forms for the distributions of standardized values dependent on $N$? ...
0
votes
0answers
35 views

Normalizing weight vector of a linear SVM

From what I've learned, training a (hard-margin) linear SVM on training data gives weights $w$ and intercept $b$ such that it forms hyperplane $\{x: x^Tw + b = 0\}$- this is the hyperplane that ...
1
vote
1answer
19 views

Does Kernel density estimation normalise the distributions?

I am analysing polymorphisms distribution data from Next Generation Sequencing data using Kernel density estimation (KDE). However I would like to know if this method permit an unbiased comparisons, i....
0
votes
0answers
5 views

Analysis of correlation for individuals' answers to repeated questions

My situation is this: I have collected data in a user study where I asked people to play through and rate multiple (20) Super Mario levels in sequence. I generated these levels using an AI powered ...
0
votes
0answers
10 views

Scaling [0,1] values so that their l1-norm is not uniform?

Let's say I have am sampling of $k$ values uniformly sampled from [0, 1], call the sample $X$. I would like to apply a transformation to them $f$, such that when I apply the $\ell_1$ norm to $f(X)$, ...
0
votes
0answers
9 views

Why is loss after training with normalized data higher than the loss of the same, but non-normalized, data?

I am toying around with training a simple LSTM model using time series data generated from the f(x) = sin(x) function to make predictions about the next value. I know this is non-sensical but I use it ...
0
votes
0answers
55 views

How do I normalise severely right-skewed data?

I have a few continuous variables in my dataset that are severely right-skewed. I have tried several log transformations, incrementally increasing it to log(10^8). Nothing has worked. The log ...
0
votes
0answers
10 views

Issue with using min/max method of normalisation in KNN

I have inherited code for a knn-based classification (intrusion-detection) algorithm that normalises based on the min/max method so that all feature values lie within the scale of 0-1. My issue is ...
0
votes
0answers
13 views

SVM Geometric Margin how to find w*

I have a SVM with training points $x_1$, and $x_2$ I calculated w by $x_1 - x_2$ $$ x_{1} = [1,0,0] $$ $$ x_{2} = [1,2,2] $$ $$ w = [0,-2,-2] $$ I have been asked to solve for $ w^* $ knowing that $$ ...
0
votes
0answers
8 views

Normalizing odd data for machine learning

I am using the kddcup99 data for some machine learning tasks (data set here: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html). I was wondering the best way to normalize some of the data that ...
0
votes
0answers
10 views

Normalizing pair of variables

I am new in statistics and have a question about normalizing pair of variables. I tried to explain below; Data Structure=(Capital, Profit) Example data = [($2M, $100K), ($3M, $100K), ($1M, $200k), ($...
1
vote
1answer
23 views

Should I use different types of normalization on the same dataset when preprocessing for machine learning

I am working to preprocess a dataset where half of it is already normalized between 0 and 1. I was planning on using z-score to normalize the rest of the dataset but I was wondering if that was a bad ...
0
votes
0answers
8 views

Normalize array in terms of correlation coefficient

Given two arrays: T and E recording temperature elastic modulus over time. They have a negative correlation coefficient r_{TE} Question: can I compute a “temperature adjusted E” (E_T) that takes all ...
1
vote
1answer
50 views

Are skewed distributions problematic to binary classification models?

I am trying to build a binary classification model and while trying to build the model, I have done some research about skewed distributions. I learnt that skewed models should be normalized to be ...
4
votes
1answer
50 views

Practical method to do MLE for natural parameters in exponential family

I encountered the following question in my research and I hope this is the correct place to post it. I'm following the notation in this lecture note by Michael I. Jordan. Assume random vector $X$ ...
0
votes
0answers
10 views

Normalization of 3D medical images

I am trying to normalize a medical (MRI) 3D image of the brain with shape DxHxW. My question is, how should I normalize this image since there is correlation between each slices? Is there a way to do ...
0
votes
1answer
20 views

Normalized 2D tensor values are not in range 0-1

Below function takes in 2D tensor and normalizes it using broadcasting .The issue is except all values to be in range 0-1 but the result has values outside this range . How to get all values in 2D ...
1
vote
1answer
22 views

Interpretation of Cluster Distortion on Normalized data

I have a clustering problem which I solved using KMeans clustering. I also know that the Elbow Method for cluster evaluation can be used to approximate a feasible pick for the number of clusters. I ...
2
votes
0answers
29 views

The impact of Normalization when training MLP

I come across a problem where I trained two MLPs using the same dataset, but one was trained using the raw data and the second one was trained using the normalized version of the dataset. In this case,...
0
votes
1answer
20 views

When Normalize is true are the coefficients arising from LASSO normalized or in the original state?

From this question: Are LASSO coefficients raw or standardized? I understand that when standardizing the data, the coefficients are returned to the original scale. Is this correct? Can I just plug in ...
0
votes
0answers
19 views

How to deal with features encoding and normalization for deployment of ML model?

I have a dataset that has many different features such as categorical, ordinal and continuous ones. Categorical Features I have great difficulty understanding how should I apply label encoding to ...
1
vote
0answers
19 views

Can I compare two different scores with two different maxes if I scale them to 1?

I have two scores based on the same population: Score A has a max of 6 Score B has a max of 9 Score A's formula is 1 * Multiplier_A * Multiplier_B + FlatNumber_A + FlatNumber_B Score B's formula is 1 ...
0
votes
1answer
26 views

Does it make sense to expect zero mean and standard deviation 1 for errors from a gaussian process model?

ML newbie here. If whatever information I have provided is not sufficient feel free to let me know what more I need to add. Now, the question: I am working with multi-task Gaussian processes. I have 3 ...
0
votes
0answers
15 views

Normalizing relevancy score in recommendation system

I am designing a recommendation system that gets some data from the user, calculates a score per each of the products, and based on that recommends some products. Since there are lots of entry points, ...
0
votes
0answers
21 views

Tensorflow unit scale preprocessing layer

I would like to have a keras model self-contained to reduce the training/serving skew. It would mean here having a preprocessing layer that is doing essentially what MinMaxScaler from scikit learn is ...
0
votes
0answers
12 views

How do you normalize results when dealing with two or three datasets?

I'm needing some help with a data analysis. I have taken advanced engineering calculus classes, but no advanced statistics, so I will need answers dumbed down a little, but hopefully I can understand ...
0
votes
0answers
13 views

What is the most efficient way of adding values from different scales to create a metric? Normalization v/s Scaling v/s?

I'm working on creating a metric to evaluate medical protocols with different factors. The proportion of weightage of each factor is determined by a survey which we have calculated as below : Factor a ...
0
votes
0answers
21 views

Should one-dimensional data be normalized for K-Means clustering?

Data normalization is important prior to K-Means clustering when there are multiple variables in the clustered data set. Data centering and scaling (for instance using Z-score) can change the relative ...
0
votes
0answers
11 views

Quantify a score (or figure-of-merit) of the code style for source code of different sizes

I want to have a numeric score between 0 and 100 to reflect the quality of code style for the R source code written by my team members based on some pre-defined rules. The rules and checks are taken ...
0
votes
0answers
9 views

Normalization changes the effect of a variable in fixed effect logit

I have constructed a conditional/fixed effect logit model. My data consists of 6k+ groups in which I get two different coefficients for the same variable if I divide that variable to the the size of ...
0
votes
0answers
31 views

Normalize skewed distribution

My data is right-skewed. Log-transform data only shifts it, not changing the distribution shape. Tried to use QuantileTransformer but output seems to be really messy. Any suggestions on how to ...
0
votes
0answers
27 views

How to normalize predicted values for an outcome event?

I work on predictive models for crime forecasting, meaning I try to model the risk for crimes. In the end of my modeling, I have the following values: number of predicted crimes for each state (...
1
vote
1answer
159 views

Why does normalizing image twice work?

I made a 'mistake' while training a neural network, it is a typical image classification problem like this. However the data is much larger and came from Kaggle. In my ...
0
votes
1answer
57 views

What is the l1-normalization of some data?

From this page and in this paper (first paragraph of chapter 2.1) there is the term of "$l_1$-normalization" or absolute normalization of a vector (i.e. some data). The scope is to turn the ...
2
votes
1answer
38 views

Scaling embedding layer's outputs in Tensorflow

I have a neural network that takes categorical and quantitative features as inputs. The quantitative features are scaled in $[0,1]$. I apply an embedding layer to get a continuous representation of ...
0
votes
0answers
22 views

Comparing Mean and standard deviation from a normalized histogram and scipy.stats.lognormal fit

I am trying to fit a lognormal distribution to my experimental Data. The experimental data provides the frequency of occurrence of a certain length. The data is given in the code below. The counts_exp ...
0
votes
0answers
38 views

Is it wrong to use tanh with images normalized in [0,1] range?

I've seen in some repositories, mostly related to GANs (Generative Adversarial Networks) using tanh activation function whilst having input images in the range of <...
0
votes
0answers
35 views

Standardizing Multiple Multivariate Time Series

I have a set of devices that I am using to collect data, each device collects a multivariate time series at the same sampling rate. (around 10 minutes). I have done first-order differencing on the ...
0
votes
0answers
11 views

Data hybrid logarithmic normalization for ANN

is it ok to normalize data by using log normalization before trained the ANN model as following: xn=log(x) yn=log(y) then performing min-max normaliztion as following: xnn=(xn-min)/(max-min) ynn=(yn-...
1
vote
0answers
21 views

Normalizing a Categorical Ordinal Variable

I am trying to normalize my data in which I collected number of time people pick their hair and their stress level which ranges from 1 (no stress) to 5 (high stress) and try to create a graph in order ...

1
2 3 4 5
22