Questions tagged [normalization]

Usually "normalization" means re-expressing data to make values lie within a specified range.

Filter by
Sorted by
Tagged with
1
vote
1answer
20 views

Why does normalising a conditional probability sufficient to ensure ensure that the resulting quantity is a distribution?

The following is said in David Barber's Bayesian Reasoning and Machine Learning: "The relation between the conditional p(A=a | B=b) and the joint p(A=a, B=b) is just the normalisation constant ...
0
votes
1answer
23 views

Does PCA normalize data with respect to standard deviation?

In my textbook it says the data is assumed to be zero-centered. I am not sure what this means. I read it stand for substracting the mean from the individual values, however I would like to know do you ...
0
votes
0answers
24 views

Yeo-Johnson did not improve model. Can that really be? [closed]

I have a dataset which has several continuous columns including my y-variable. Most of these columns are non-normally distributed and some of them have also negative values. For that reason, I tried a ...
0
votes
0answers
17 views

How to normalize data for LSTM?

For learning purpose I am making simple dataset for LSTM, which can predict next number in sequence. Here is my x value ...
0
votes
0answers
9 views

How to normalize a value from a range to another [duplicate]

I have a set of value with range 0 to 1. I have to change the dataset from 0 to x. Which formula i have to use?
0
votes
0answers
14 views

Regression : Variable transformation necessary? If yes, why?

I am working on a real estate project and have historical rental prices and vacancy data. Interested in exploring the relationship between vacancy rates and change in rental prices. The unit of ...
0
votes
2answers
39 views

How to interpret normalized coefficients in logistic regression?

I trained a logistic regression model with 5 features per sample. Before training, I normalized the range of my features into [0,1] (MinMax scaler). After training, I received the following ...
1
vote
1answer
32 views

Normalising data for a learning experiment on alpine skiing

We are currently performing a large learning experiment on alpine skiing. In this experiment, we test skiers on three different slalom courses. We compute performance by calculating the average of the ...
0
votes
0answers
31 views

How to convert standard normal output to predictions?

I am using a linear model to perform predictions. My loss function is minimizing three seperate objective functions. In order to perform regression, I converted my 3 objective outputs to standard ...
1
vote
1answer
25 views

Normalize sample to match the mean and the standard deviation

There are two samples (sufficiently large and independent). One, size $n_1$ has the mean $m_1$ and the standard deviation $s_1$, the other, size $n_2$ with the mean $m_2$ and the standard deviation $...
0
votes
0answers
6 views

A question about normalization in probit model (binary response model with normal error)

Suppose I have data on $\{Y_i,X_{1i},X_{2i}\}_{i=1}^{N}$ and the data generating process is $Y_i=\mathbf{1}(\beta_1X_{1i}+\beta_2X_{2i}>e_i)$, where $e_i\sim N(0,\sigma^2)$. Usually, we do a ...
0
votes
0answers
12 views

normalization or standardization for outlier?

I am trying to build a machine learning model that uses the age of the student as a feature. But my data has some erroneous value. which technique should I use to ignore those values? Z-score or ...
2
votes
2answers
31 views

Comparing yearly data that has different $n$ values per year?

I have been collecting soil moisture data at my research site for almost 20 years. However, I have not been consistent with the number of times that I have been taken these measurements. The problem ...
0
votes
0answers
17 views

Fitting time series $b$ to time series $a$ stock prices

This is a bit hard to explain by words but I will do my best and then provide an illustration afterwards. In short I want to fit time series $b$ to time series a while retaining the "uniqueness&...
1
vote
1answer
23 views

Can we normalize both continuous and discrete numerical values

I have a sensor dataset with 16 features as numerical values (12 are continuous and 4 are discrete). I am using LSTM model to fit the data and do some classification. As both continuous and discrete ...
1
vote
0answers
42 views

Why clustering in a linear scale using correlation based distance gives better results than clustering in a log2 scale?(PAM clustering)

I have questions regarding cluster analysis. I am trying to cluster data made up of proteins. (23 columns and 1800 rows) I have the data in a log2 scale, some variables range between 2-10 and others ...
2
votes
1answer
34 views

Preprocessing of target data set in Transfer learning approach

So the idea of transfer learning approach is to pre-train a model on source data set and then re-train (or fine-tune) the model on the target data set. But what about preprocessing? If I choose to ...
1
vote
1answer
24 views

Does model need normalized data?

With a very limited time to look at the model's architecture, how does one decide whether or not an arbitrary model need normalized input data? There are tons of ML libraries out there and most of the ...
1
vote
0answers
25 views

How to normalize correlated variables for ANOVA

I have district level data about number of COVID-19 cases $X=\{x_i\}_{i=0}^n$ and frequency of newspaper reports about COVID cases $Y=\{y_i\}_{i=0}^n$. Here $n$ is the number of districts. I want to ...
0
votes
1answer
8 views

how to normalize data such that an estimated OLS regression vector has pre-specified length (= L_2 norm)

I have the following data: $n$ observations on $d$ variables $X$ and one outcome variable $Y$; i.e. $X$ is a $n \times d$ matrix and $Y$ an $n \times 1$ vector. I consider the following Ordinary Least ...
1
vote
0answers
39 views

Is group normalization with G=1 equivalent to layer normalization?

References: Batch normalization (BN) Layer normalization (LN) Group normalization (GN) I will use pseudo TensorFlow-like code to be very specific about the tensor axes. I assume an input tensor <...
0
votes
0answers
12 views

Can I use q-values on true positive and true negative datasets?

I'm evaluating a program that predicts whether a DNA sequence is bacterial or not. I have a true positive dataset consiting of bacterial DNA an a true negative dataset made up of non-bacterial dna ...
0
votes
2answers
22 views

What does the data sample (xi) means in z-score normalization?

Is $\mu_i$ computed by averaging over the entire features or by averaging over each feature respectively? Here is a snapshot I found from a journal article? Can anyone help to answer the above two ...
2
votes
0answers
31 views

Standarization vs Scaling

As part of the data pre-processing in the paper Cloud Security: LKM and Optimal Fuzzy System for Intrusion Detection in Cloud Environment numeric attributes were normalized using the following ...
1
vote
1answer
50 views

Data normalization by using mean and standard deviation - Strange example

I read a paper which talks about two samples; the first one concerning data (e.g. resistance $R_{ct}$) from a biosensor1 (called module), the second one concerning data from biosensor2 (called CHI). ...
1
vote
0answers
22 views

How to preprocess time series data with a high range for a neural network?

I have a multivariate time series where one feature ranges from 0 to 25 million while another simply goes from 0 to 800 thousand. Here is an example of my data: Giving these values to a neural ...
0
votes
0answers
22 views

What level of measurement can I use to compare a student's different test scores?

Let's say a student takes multiple different tests. I get a percentage for each of the test scores (the tests have a different amount of multiple choice questions, and the multiple choice questions ...
1
vote
2answers
30 views

Normalize across groups, individuals or population?

I would like to compare sensors of a manufacturer A with those of a manufacturer B. As they provide different measurement magnitudes, I want to scale the variables such that it makes more sense to ...
0
votes
0answers
9 views

Which scaler to use for combining different embeddings?

I am working on an experiment, where I am combining ( average and concatenate ) different embeddings like Elmo, Bert etc, since both embeddings are from different models, I thought, it's better to ...
2
votes
2answers
41 views

Normalization of financial price to use as input in a neural network

I'm looking for the best method to normalize/standardize financial prices in order to use them as inputs for my neural network. As you probably know financial prices do not follow a normal ...
1
vote
0answers
17 views

Scaling revenues - Drawbacks and advantages of average vs. median scaling

Context Currently I am doing some regression-predictions with various machine learning algorithms (still in the experimental phase). Some features I use for the prediction are revenues customers ...
1
vote
2answers
38 views

How can PCA maximise variance after I standardise all predictor variance = 1?

I have been reading about Principal Components Analysis, and I think it is in general trying to extract as much "variance" out of the predictors $ \vec{X} = (X_1, X_2, ..., X_n)$ by ...
0
votes
1answer
28 views

Transform general multivariate normal to standard multivariate normal

I have data that I can assume will be multivariate normal with a known mean vector mu and known covariance matrix sigma, and I'm ...
0
votes
1answer
43 views

Standardize-back the Standard deviation

I run an lmer model using standardized data like scale(y) ~ 1 + (1|categorical) Now, I have a standard deviation for the random effect in normalized world but I ...
0
votes
0answers
10 views

Assess how different types of normalization (global vs. local) alter my groups of 2D matrices

I have a group of 2D matrices that passed by two distinct normalization types: in the first, each matrix was normalized by dividing its elements by the local maximum (the maximum value observed in ...
1
vote
1answer
20 views

What is considered “normal” in a dataset, relative to other datasets and through time?

I currently have time series datasets of GDP across countries (lets say USA, Australia, and Japan). I want to be able to create a number from -1 to 1 for some point in time that both considers how ...
1
vote
0answers
43 views

How to normalize data in R

This is my data: ...
0
votes
1answer
39 views

Transforming dependent and independent variables with different techniques

I am learning how to transform and standardize data for building more precise models. I did not find any information if it is legit to transform independent variables with one technique (for instance, ...
0
votes
0answers
11 views

How to normalize my data between -1 and 1? [duplicate]

How would I normalize my data between -1 and 1? I have both negative and positive values in my data c# array ?
2
votes
0answers
18 views

Correction of systematic multiplicative confounder effect

Dear community members, I have a very simple question. I have thousands of independent large samples (>1000 data points each) from complex 1D distributions. To imagine the problem better, I ...
2
votes
0answers
96 views

How do I scale test score improvements with normally distributed test scores?

I have data on an entire population's test scores at two points in time. The entire set of test scores are normally distributed. There are 10 groups in the population. I want to be able to identify ...
0
votes
0answers
19 views

How to make scores with different baselines comparable---compare the quality of edible vegetation for several bird species in the same area?

To give some background to my problem, I want to compare the quality of edible vegetation for several bird species in the same area. The birds eat some of the same food resources, but there are also ...
1
vote
1answer
82 views

Normalizing continuous features using sigmoid function

Can you use the sigmoid function to normalize continuous features that have no theoretical maximum value but tend to cluster around [-1, 1]? Although using the sigmoid function would be a non-linear ...
1
vote
1answer
38 views

Elbow curve response for Normalized vs Standardized data

I was trying K prototypes for a Blood transfusion dataset. When I tried to find the optimal number of clusters by normalizing(range 0-1) the data it was not giving proper curve, I got the following ...
3
votes
1answer
41 views

Transformation to normality when data is trimmed at a specific value

So let's say I have an original dataset $x\sim N(58, 3.5)$, but then I go and take only the records with $x \geq 54$ and continue working with this trimmed data. Now, as I understand, this trimmed ...
1
vote
0answers
23 views

Equation for weighted mean of percentages

I am trying to get the equation for a weighted average of percentages to be normalized between 0 and 1. Am I doing it correctly? Thank you so much!
2
votes
1answer
33 views

Normalizing posterior distribution

This is from task 5.3. in Bayesian Data Analysis 3rd edition by Gelman et al. It deals with a hierarchical model where I am supposed to simulate the posterior distribution for $\tau$, which is the ...
1
vote
0answers
18 views

Should multivariate time series by normalized in any way before fitting a VAR model?

I have many time series on very different scales; some take on values in the millions, while some are restricted to $[-1,1]$. Should I preprocess them in any way, or will the found coefficients take ...
2
votes
0answers
30 views

Understanding the difference between standardization and normalizing for hierarchical clustering

I am trying to understand the difference between normalization and standardization for the hierarchical clustering. I read the documentation and some posts like this and this as well as several SO ...
0
votes
1answer
40 views

Should I scale all features?

I was wondering whether or not one should generally scale all numerical features. For instance, I had a state column with string values like KY, TX, or AL. These I converted to an indexed column with ...

1
2 3 4 5
21