Questions tagged [normalization]

Usually "normalization" means re-expressing univariate data to make values lie within a specified range.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Should the data be scaled before normalisation to enable the use of the model as a pre-trained model?

I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data. Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I ...
Janikas's user avatar
  • 101
0 votes
0 answers
32 views

Normalisation in feature extraction using pre-trained model

I have a dataset with medical images. I want to implement a network for super-resolution using GANs. One of the criteria of the optimisation is a perceptual loss. For that I will use a pretrained vgg ...
Janikas's user avatar
  • 101
-1 votes
1 answer
44 views

Denormalize prediction values for unseen data

For my model, I need to apply z-normalization to the input data. For train and test, I can denormalize the output since I have the mean and standard deviation for both y-true train and test sets. ...
Jas's user avatar
  • 99
0 votes
0 answers
38 views

Normalize data without losing confidence interval of reference

I have a small data frame of lower and upper confidence levels along with means. What I would like to do is compare the means of all rows directly against the first. Here is the DF: ...
Tjippie's user avatar
0 votes
1 answer
86 views

Normalization of samples before applying the t-test

I am working on an algorithm for recommender systems. I want to compare its results to another method. For this purpose, I need to apply a t-test on their results to see if there's a significant ...
samiravz's user avatar
2 votes
1 answer
32 views

In the context of machine learning, at what stage is the normalization process implemented?

I have several data types and I want to use them as features for binary classification task. The data is as follows: 1- Genes: I have several datasets containing bulk RNA-seq data. Some of these ...
Programming Noob's user avatar
0 votes
0 answers
88 views

Per Mel Spectrogram min-max normalization vs full training set min-max normalization for CNN classification of audio

I am watching a tutorial on using mel spectrograms to classify the audio's genre via CNN. My question is why apply local min-max normalization to each individual mel spectrogram? What I mean by local ...
Hayden LaBrie's user avatar
0 votes
0 answers
46 views

How to normalize dataset with mixed data (continuous and count)?

I am trying to determine what procedure should I use to feature engineer the most descriptive possible dataset to predict a binary outcome. The dataset has variables that are count-valued with ...
The Bosco's user avatar
  • 253
2 votes
1 answer
74 views

How should I scale data that has been assembled from different data sources?

The data I'm woking with consists of 3 types of data: 1- binary features: those features are either 0 or 1. I have about 6 or 7 columns. 2- cells: the values here range from 0 to 0.8 at max. Here I ...
Programming Noob's user avatar
1 vote
0 answers
170 views

Batch normalization before or after channel-wise concatenation?

I have a block in a CNN that splits the input channel-wise in half, and one half goes through a regular 3x3 2d convolutional layer, and the other goes through a dilated 3x3 2d convolutional layer. ...
92carmnad's user avatar
1 vote
0 answers
75 views

Comparing means in Box-Cox transformed data

Deeply sorry as this wasn't really covered in my statistics classes. I am current comparing datasets for people divorcing vs. dissolving their marriages. I have two variables, length of marriage, and ...
Sophie's user avatar
  • 11
1 vote
0 answers
312 views

Normalization or standardization in stock prediction

Currently I watched the videos (links below) that argues using the normalization (max-min scale) is the bad idea when it comes to the stock prediction. In the videos, the editor aruges that people ...
Chi-Yuan Li's user avatar
1 vote
0 answers
14 views

Scaling datasets for multi-dataset time series

Suppose that I have training data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. Normalizing each dataset over the ...
Hadar's user avatar
  • 125
1 vote
1 answer
284 views

How to normalize scores on different versions of an exam?

Let's say a professor gives different versions of an exam to her two sections. Despite attempts to make the exams equal in difficulty, the minimum score, median, and mean are all significantly higher ...
Ellen Spertus's user avatar
0 votes
1 answer
75 views

Comparing RMSE values across different datasets

I am working on a PV energy production forecasting problem. With various ML models (ANN, RNN, LSTM) I am trying to predict the energy for the following day, based on the historical data. The ...
GCMeccariello's user avatar
0 votes
0 answers
20 views

On effects of dummy variables on scaled data

I have a dataset consisting of 3 variables: a time series related to hourly energy load (expressed in thousands of MWh), a dummy variable related to business hours (e.g. the dummy is 1 the current ...
Lorenz's user avatar
  • 1
0 votes
0 answers
5 views

Which mean value to consider for checking over indexing accounts

Problem statement: I am working on a project for a major game producer. The company is planning to sell the SKUs of one of their game titles (a title has multiple games) to the people who are ...
Ritik P. Nayak's user avatar
1 vote
0 answers
31 views

How to create a composite variable out of 3 non-correlated variables?

I am working on a model to account for flood risk and it is based on three variables: Variable 1: drainage (float: 0 - 80) Variable 2: estimated population (float: 0-2,000) Variable 3: road network ...
Felipe S's user avatar
5 votes
1 answer
296 views

Is standardization still needed after a LASSO model is fitted?

We know that it's better to standardization the training data (i.e. X_train) before fitting a LASSO model, especially when features are not in the same scale (Ref. Is standardisation before Lasso ...
YaOzI's user avatar
  • 153
2 votes
1 answer
243 views

Normalization of time series data for classification

Explaining my dataset: I have a univariate timeseries dataset of energy consumption. Each row of the dataset are half-hourly records of the energy consumption of a consumer. So every ...
Murilo's user avatar
  • 179
2 votes
1 answer
829 views

Does normalizing/changing the scale of the target variable impact the shape of the loss function equation?

I was under the impression that changing the scale/normalizing the target variable in a regression task would not change the overall shape of the loss function equation but would simply translate/move ...
Kiran Manicka's user avatar
0 votes
0 answers
154 views

Loss function weighting in regression when the target varies orders of magnitude between groups

I have a dataset with 200 groups, and 50-300 observations per group. The target I'm trying to predict is a strictly positive financial metric, which varies 5+ orders of magnitude between groups but is ...
Marcin Kozłowski's user avatar
1 vote
1 answer
98 views

Normalising AIC to compensate for missing data?

AIC is simply penalised log-loss, and log-loss depends directly on the dataset size. To create a model from data, missing data need to be excluded first. Assuming missing data are spread across ...
Igor F.'s user avatar
  • 9,069
1 vote
0 answers
19 views

When should one normalise the data and when should one standardize the data as a part of data pre-processing while building ML models?

I have seen people using both normalisation which is min-max normalization ( all values will be between 0,1) and standardize( normal distribution) the data as part of pre-processing. It's given that ...
Arpit Sisodia's user avatar
0 votes
1 answer
328 views

How to use slopes in PCA?

I would like to use slope values in PCA. The problem I face is that the slopes I calculate per group could be within different ranges of values. We know that it is important to normalize your data ...
Quinten's user avatar
  • 369
1 vote
1 answer
1k views

ARIMA or SARIMA scale and normalize data

Good evening everyone, I am here to ask a question regarding the statistical models ARIMA & SARIMA use to build predictive models based on past values and with the intent of predicting future ...
Alessandro Pio Budetti's user avatar
1 vote
1 answer
179 views

Neural Networks - Can I Use Any Activation for the Output Layer?

I'm new to neural networks, and in almost everything I'm reading, the activation function recommended on the output layer follows a specific pattern: If the network does binary classification (1 ...
Krusty the Clown's user avatar
2 votes
1 answer
446 views

How to choose the Normalization method for a co-occurence matrix?

I have a co-occurrence matrix about hashtags usage (The value in the cell means the number of times two hashtags appear together in a single tweet), it is transformed from a 2-mode matrix. Now I want ...
Xinmeng Lien's user avatar
1 vote
0 answers
23 views

Visualize difference between time series lines, with similar changes over time and missing data

I have a large set of objects that each generate a multivariate time series at a daily to hourly resolution. As an example, let's say that they are weather stations generating variables for ...
The Guy with The Hat's user avatar
0 votes
0 answers
25 views

Conceptual understanding of effect of standardizing after normalization for clustering

My general understanding is that, before running a clustering algorithm, one typically wants to consider trying to normalize or standardize the data depending on its content and use. In my case, I ...
Josh's user avatar
  • 308
1 vote
0 answers
29 views

Normalisation methodology and batch effect

I have a question about the normalization of datas and batch effect ... For example, here is a dataset : ...
Nicolas's user avatar
  • 13
0 votes
0 answers
19 views

Normalization and patient effect

I have a very simple question but I don't really have answer myself... I have a set of data like this : ...
Nicolas's user avatar
  • 13
0 votes
1 answer
118 views

Preprocessing of spectroscopy data for PLSR: do I need to normalize the data for every wavelength?

I want to apply a partially least square regression on spectroscopy data to model a chemical content of my probe. So, every wavelength of the spectrum serves as one variable in the model. Doing some ...
iris's user avatar
  • 11
0 votes
2 answers
47 views

How can I "normalize" one data set based on a third variable?

My data set is divided into two groups: (1) patients who received physical therapy and (2) patients who did not. Our primary outcome is looking at how long it took them for their symptoms to resolve (...
Anthony B's user avatar
0 votes
0 answers
75 views

Comparing probability of different length sequences of independent events

Assume I have two different datasets $D_1$ and $D_2$. Each of these datasets has a different number of samples $|D_1| = m$, $|D_2| = n$. Assume each sample is i.i.d drawn from the same underlying ...
BeginnersMindTruly's user avatar
0 votes
1 answer
46 views

Standard score applied to data that is not normally distributed

If I scale data from an arbitrary distribution using the standard score, will the property of the normal distribution that 75% of data lies between +/- 2 standard deviations from the mean, still hold?
user378377's user avatar
0 votes
0 answers
74 views

Comparing gradients for 3 slopes on different scales

I am trying to compare slopes/gradients for three Groups (A, B, and C): the values for each Group are on essentially the same scale for the independent variable (Age), but are on markedly different ...
PhelsumaFL's user avatar
2 votes
1 answer
39 views

Correct renormalization of the product of two probabilistic models

I came across these two papers combining an energy-based model (EBM) with a probabilistic generative model: [Xiao et al., 2020] and [Xiao et al., 2021]. In [Xiao et al., 2020], the new model is ...
Njw96's user avatar
  • 43
1 vote
0 answers
34 views

Do you need to normalize labels for models other than neural nets?

As mentioned here, normalizing the target variable often helps a neural network converge faster. Does it help in convergence, or is there otherwise a reason to use it, for any type of model other than ...
bdavidson's user avatar
1 vote
0 answers
70 views

Does the transformation of variables affect the performance of gradient descent?

Suppose we want to minimize the objective function $f(x)$ with respect to $x\in \mathbb{R}^p$. Wtih a linear transformation of $x$, e.g., $z = Qx$ for a proper $Q\in\mathbb{R}^{p\times p}$, the new ...
Miles N.'s user avatar
  • 145
0 votes
0 answers
175 views

What's the best data transformation to build a radar chart?

I need to build some radar charts using NBA players stats, but obviously if I don't use the statistics expressed as a percentage (like for example "3-Point Field Goal Percentage" or "...
TF7's user avatar
  • 37
0 votes
0 answers
39 views

How do I impose restrictions $ 0\leq \alpha \leq \beta <1$?

I want to restrict values s.t. I get $\theta = (\alpha, \beta) =g(\theta_1, \theta_2)$ with the following restrictions. $ 0\leq \alpha \leq \beta <1 $ I know the correct answer should be $\beta = (...
user773674's user avatar
1 vote
0 answers
23 views

Best way to analyse two experiments

I'm looking for help/indication on approaching a data integration problem. I have a dose-response curve that is described by a log curve. Because of measurement difficulties, the dose and response can'...
Lu_Ste's user avatar
  • 11
0 votes
0 answers
218 views

Median, MAD Normalization

I am trying to normalize prices based on the (median, MAD) transformation. I attach a image below describing it in comparison to the (mean, std) normalization. I have trouble underatanding if ...
Simon Rydstedt's user avatar
1 vote
0 answers
174 views

Is the min-max rescaling or z-score normalization more appropriate when comparing data of vastly different mean, SD, and range?

I have six lists of data in Python where each list contains 23 items as follows. ...
Philipp's user avatar
  • 225
0 votes
0 answers
328 views

Can I use normalization and standardization on the same dataset?

I'm working on an ML project to predict wine quality from a wine's physical characteristics. The features of my data are on vastly different scales so I've been experimenting with different ...
Sylith's user avatar
  • 101
1 vote
0 answers
62 views

What's the proper way to analyze data from an experiment where participants perform different tasks?

I've recently conducted an experiment on my university where participants had to perform various tasks using two different virtual reality interaction methods (each participant performed all tasks ...
Wojtek Wencel's user avatar
1 vote
0 answers
32 views

Regularize variables for Johansen coinegration

I am looking into applying the Johansen cointegration procedure to variables of very different scales, going from units to tens of thousands. I use log(prices) but the scale difference is still very ...
StatArb's user avatar
  • 85
0 votes
1 answer
134 views

Prediction with normalization of features

I am working with a time series data (date column and a value column) and I have extracted date features(weekday, weekend etc) and rolling means or the prediction. For features to be given equal ...
bella_pa's user avatar
  • 103
0 votes
0 answers
31 views

Hello, I am doing a machine learning assesment project in R where I have to perform multiple linear regression and write report on all my steps

So I did some data exploratory and found that the range of my features are variable, like some columns have range between 1 to 5, while other columns like 'square_feet' has range of 500 to 2000. In ...
Rishabh Parmar's user avatar

1
2
3 4 5
25