Questions tagged [normalization]
Usually "normalization" means re-expressing univariate data to make values lie within a specified range.
1,239
questions
0
votes
0
answers
13
views
Should the data be scaled before normalisation to enable the use of the model as a pre-trained model?
I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data.
Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I ...
0
votes
0
answers
32
views
Normalisation in feature extraction using pre-trained model
I have a dataset with medical images. I want to implement a network for super-resolution using GANs. One of the criteria of the optimisation is a perceptual loss. For that I will use a pretrained vgg ...
-1
votes
1
answer
44
views
Denormalize prediction values for unseen data
For my model, I need to apply z-normalization to the input data. For train and test, I can denormalize the output since I have the mean and standard deviation for both y-true train and test sets.
...
0
votes
0
answers
38
views
Normalize data without losing confidence interval of reference
I have a small data frame of lower and upper confidence levels along with means.
What I would like to do is compare the means of all rows directly against the first.
Here is the DF:
...
0
votes
1
answer
86
views
Normalization of samples before applying the t-test
I am working on an algorithm for recommender systems. I want to compare its results to another method. For this purpose, I need to apply a t-test on their results to see if there's a significant ...
2
votes
1
answer
32
views
In the context of machine learning, at what stage is the normalization process implemented?
I have several data types and I want to use them as features for binary classification task.
The data is as follows:
1- Genes: I have several datasets containing bulk RNA-seq data. Some of these ...
0
votes
0
answers
88
views
Per Mel Spectrogram min-max normalization vs full training set min-max normalization for CNN classification of audio
I am watching a tutorial on using mel spectrograms to classify the audio's genre via CNN. My question is why apply local min-max normalization to each individual mel spectrogram? What I mean by local ...
0
votes
0
answers
46
views
How to normalize dataset with mixed data (continuous and count)?
I am trying to determine what procedure should I use to feature engineer the most descriptive possible dataset to predict a binary outcome. The dataset has variables that are count-valued with ...
2
votes
1
answer
74
views
How should I scale data that has been assembled from different data sources?
The data I'm woking with consists of 3 types of data:
1- binary features: those features are either 0 or 1. I have about 6 or 7 columns.
2- cells: the values here range from 0 to 0.8 at max. Here I ...
1
vote
0
answers
170
views
Batch normalization before or after channel-wise concatenation?
I have a block in a CNN that splits the input channel-wise in half, and one half goes through a regular 3x3 2d convolutional layer, and the other goes through a dilated 3x3 2d convolutional layer. ...
1
vote
0
answers
75
views
Comparing means in Box-Cox transformed data
Deeply sorry as this wasn't really covered in my statistics classes. I am current comparing datasets for people divorcing vs. dissolving their marriages. I have two variables, length of marriage, and ...
1
vote
0
answers
312
views
Normalization or standardization in stock prediction
Currently I watched the videos (links below) that argues using the normalization (max-min scale) is the bad idea when it comes to the stock prediction. In the videos, the editor aruges that people ...
1
vote
0
answers
14
views
Scaling datasets for multi-dataset time series
Suppose that I have training data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. Normalizing each dataset over the ...
1
vote
1
answer
284
views
How to normalize scores on different versions of an exam?
Let's say a professor gives different versions of an exam to her two sections. Despite attempts to make the exams equal in difficulty, the minimum score, median, and mean are all significantly higher ...
0
votes
1
answer
75
views
Comparing RMSE values across different datasets
I am working on a PV energy production forecasting problem. With various ML models (ANN, RNN, LSTM) I am trying to predict the energy for the following day, based on the historical data.
The ...
0
votes
0
answers
20
views
On effects of dummy variables on scaled data
I have a dataset consisting of 3 variables: a time series related to hourly energy load (expressed in thousands of MWh), a dummy variable related to business hours (e.g. the dummy is 1 the current ...
0
votes
0
answers
5
views
Which mean value to consider for checking over indexing accounts
Problem statement:
I am working on a project for a major game producer. The company is planning to sell the SKUs of one of their game titles (a title has multiple games) to the people who are ...
1
vote
0
answers
31
views
How to create a composite variable out of 3 non-correlated variables?
I am working on a model to account for flood risk and it is based on three variables:
Variable 1: drainage (float: 0 - 80)
Variable 2: estimated population (float: 0-2,000)
Variable 3: road network ...
5
votes
1
answer
296
views
Is standardization still needed after a LASSO model is fitted?
We know that it's better to standardization the training data (i.e. X_train) before fitting a LASSO model, especially when features are not in the same scale (Ref. Is standardisation before Lasso ...
2
votes
1
answer
243
views
Normalization of time series data for classification
Explaining my dataset: I have a univariate timeseries dataset of energy consumption. Each row of the dataset are half-hourly records of the energy consumption of a consumer. So every ...
2
votes
1
answer
829
views
Does normalizing/changing the scale of the target variable impact the shape of the loss function equation?
I was under the impression that changing the scale/normalizing the target variable in a regression task would not change the overall shape of the loss function equation but would simply translate/move ...
0
votes
0
answers
154
views
Loss function weighting in regression when the target varies orders of magnitude between groups
I have a dataset with 200 groups, and 50-300 observations per group. The target I'm trying to predict is a strictly positive financial metric, which varies 5+ orders of magnitude between groups but is ...
1
vote
1
answer
98
views
Normalising AIC to compensate for missing data?
AIC is simply penalised log-loss, and log-loss depends directly on the dataset size. To create a model from data, missing data need to be excluded first. Assuming missing data are spread across ...
1
vote
0
answers
19
views
When should one normalise the data and when should one standardize the data as a part of data pre-processing while building ML models?
I have seen people using both normalisation which is min-max normalization ( all values will be between 0,1) and standardize( normal distribution) the data as part of pre-processing.
It's given that ...
0
votes
1
answer
328
views
How to use slopes in PCA?
I would like to use slope values in PCA. The problem I face is that the slopes I calculate per group could be within different ranges of values. We know that it is important to normalize your data ...
1
vote
1
answer
1k
views
ARIMA or SARIMA scale and normalize data
Good evening everyone,
I am here to ask a question regarding the statistical models ARIMA & SARIMA use to build predictive models based on past values and with the intent of predicting future ...
1
vote
1
answer
179
views
Neural Networks - Can I Use Any Activation for the Output Layer?
I'm new to neural networks, and in almost everything I'm reading, the activation function recommended on the output layer follows a specific pattern:
If the network does binary classification (1 ...
2
votes
1
answer
446
views
How to choose the Normalization method for a co-occurence matrix?
I have a co-occurrence matrix about hashtags usage (The value in the cell means the number of times two hashtags appear together in a single tweet), it is transformed from a 2-mode matrix. Now I want ...
1
vote
0
answers
23
views
Visualize difference between time series lines, with similar changes over time and missing data
I have a large set of objects that each generate a multivariate time series at a daily to hourly resolution. As an example, let's say that they are weather stations generating variables for ...
0
votes
0
answers
25
views
Conceptual understanding of effect of standardizing after normalization for clustering
My general understanding is that, before running a clustering algorithm, one typically wants to consider trying to normalize or standardize the data depending on its content and use. In my case, I ...
1
vote
0
answers
29
views
Normalisation methodology and batch effect
I have a question about the normalization of datas and batch effect ...
For example, here is a dataset :
...
0
votes
0
answers
19
views
Normalization and patient effect
I have a very simple question but I don't really have answer myself... I have a set of data like this :
...
0
votes
1
answer
118
views
Preprocessing of spectroscopy data for PLSR: do I need to normalize the data for every wavelength?
I want to apply a partially least square regression on spectroscopy data to model a chemical content of my probe. So, every wavelength of the spectrum serves as one variable in the model. Doing some ...
0
votes
2
answers
47
views
How can I "normalize" one data set based on a third variable?
My data set is divided into two groups: (1) patients who received physical therapy and (2) patients who did not.
Our primary outcome is looking at how long it took them for their symptoms to resolve (...
0
votes
0
answers
75
views
Comparing probability of different length sequences of independent events
Assume I have two different datasets $D_1$ and $D_2$. Each of these datasets has a different number of samples $|D_1| = m$, $|D_2| = n$. Assume each sample is i.i.d drawn from the same underlying ...
0
votes
1
answer
46
views
Standard score applied to data that is not normally distributed
If I scale data from an arbitrary distribution using the standard score, will the property of the normal distribution that 75% of data lies between +/- 2 standard deviations from the mean, still hold?
0
votes
0
answers
74
views
Comparing gradients for 3 slopes on different scales
I am trying to compare slopes/gradients for three Groups (A, B, and C): the values for each Group are on essentially the same scale for the independent variable (Age), but are on markedly different ...
2
votes
1
answer
39
views
Correct renormalization of the product of two probabilistic models
I came across these two papers combining an energy-based model (EBM) with a probabilistic generative model: [Xiao et al., 2020] and [Xiao et al., 2021].
In [Xiao et al., 2020], the new model is ...
1
vote
0
answers
34
views
Do you need to normalize labels for models other than neural nets?
As mentioned here, normalizing the target variable often helps a neural network converge faster.
Does it help in convergence, or is there otherwise a reason to use it, for any type of model other than ...
1
vote
0
answers
70
views
Does the transformation of variables affect the performance of gradient descent?
Suppose we want to minimize the objective function $f(x)$ with respect to $x\in \mathbb{R}^p$. Wtih a linear transformation of $x$, e.g., $z = Qx$ for a proper $Q\in\mathbb{R}^{p\times p}$, the new ...
0
votes
0
answers
175
views
What's the best data transformation to build a radar chart?
I need to build some radar charts using NBA players stats, but obviously if I don't use the statistics expressed as a percentage (like for example "3-Point Field Goal Percentage" or "...
0
votes
0
answers
39
views
How do I impose restrictions $ 0\leq \alpha \leq \beta <1$?
I want to restrict values s.t. I get $\theta = (\alpha, \beta) =g(\theta_1, \theta_2)$
with the following restrictions.
$
0\leq \alpha \leq \beta <1
$
I know the correct answer should be $\beta = (...
1
vote
0
answers
23
views
Best way to analyse two experiments
I'm looking for help/indication on approaching a data integration problem.
I have a dose-response curve that is described by a log curve. Because of measurement difficulties, the dose and response can'...
0
votes
0
answers
218
views
Median, MAD Normalization
I am trying to normalize prices based on the (median, MAD) transformation. I attach a image below describing it in comparison to the (mean, std) normalization.
I have trouble underatanding if ...
1
vote
0
answers
174
views
Is the min-max rescaling or z-score normalization more appropriate when comparing data of vastly different mean, SD, and range?
I have six lists of data in Python where each list contains 23 items as follows.
...
0
votes
0
answers
328
views
Can I use normalization and standardization on the same dataset?
I'm working on an ML project to predict wine quality from a wine's physical characteristics. The features of my data are on vastly different scales so I've been experimenting with different ...
1
vote
0
answers
62
views
What's the proper way to analyze data from an experiment where participants perform different tasks?
I've recently conducted an experiment on my university where participants had to perform various tasks using two different virtual reality interaction methods (each participant performed all tasks ...
1
vote
0
answers
32
views
Regularize variables for Johansen coinegration
I am looking into applying the Johansen cointegration procedure to variables of very different scales, going from units to tens of thousands. I use log(prices) but the scale difference is still very ...
0
votes
1
answer
134
views
Prediction with normalization of features
I am working with a time series data (date column and a value column) and I have extracted date features(weekday, weekend etc) and rolling means or the prediction.
For features to be given equal ...
0
votes
0
answers
31
views
Hello, I am doing a machine learning assesment project in R where I have to perform multiple linear regression and write report on all my steps
So I did some data exploratory and found that the range of my features are variable, like some columns have range between 1 to 5, while other columns like 'square_feet' has range of 500 to 2000. In ...