Questions tagged [normalization]
Usually "normalization" means re-expressing univariate data to make values lie within a specified range.
1,185
questions
0
votes
0
answers
13
views
Loss function weighting in regression when the target varies orders of magnitude between groups
I have a dataset with 200 groups, and 50-300 observations per group. The target I'm trying to predict is a strictly positive financial metric, which varies 5+ orders of magnitude between groups but is ...
0
votes
1
answer
23
views
Normalising AIC to compensate for missing data?
AIC is simply penalised log-loss, and log-loss depends directly on the dataset size. To create a model from data, missing data need to be excluded first. Assuming missing data are spread across ...
- 7,621
1
vote
0
answers
9
views
When should one normalise the data and when should one standardize the data as a part of data pre-processing while building ML models?
I have seen people using both normalisation which is min-max normalization ( all values will be between 0,1) and standardize( normal distribution) the data as part of pre-processing.
It's given that ...
- 1,129
0
votes
0
answers
9
views
Different normalization in regularization term for scale invariant problem
Suppose, I have a regression problem, where I have the lables of my training data $\bf{y}$ and two measurement matrixes $A$ and $B$. The cost function is $||\sin(\frac{A*\textbf{w}}{B*\textbf{w}})-\...
0
votes
1
answer
123
views
How to use slopes in PCA?
I would like to use slope values in PCA. The problem I face is that the slopes I calculate per group could be within different ranges of values. We know that it is important to normalize your data ...
- 301
1
vote
1
answer
43
views
ARIMA or SARIMA scale and normalize data
Good evening everyone,
I am here to ask a question regarding the statistical models ARIMA & SARIMA use to build predictive models based on past values and with the intent of predicting future ...
1
vote
1
answer
22
views
Neural Networks - Can I Use Any Activation for the Output Layer?
I'm new to neural networks, and in almost everything I'm reading, the activation function recommended on the output layer follows a specific pattern:
If the network does binary classification (1 ...
- 111
2
votes
1
answer
29
views
How to choose the Normalization method for a co-occurence matrix?
I have a co-occurrence matrix about hashtags usage (The value in the cell means the number of times two hashtags appear together in a single tweet), it is transformed from a 2-mode matrix. Now I want ...
- 23
1
vote
0
answers
15
views
Visualize difference between time series lines, with similar changes over time and missing data
I have a large set of objects that each generate a multivariate time series at a daily to hourly resolution. As an example, let's say that they are weather stations generating variables for ...
0
votes
0
answers
17
views
Conceptual understanding of effect of standardizing after normalization for clustering
My general understanding is that, before running a clustering algorithm, one typically wants to consider trying to normalize or standardize the data depending on its content and use. In my case, I ...
- 278
0
votes
0
answers
24
views
Normalisation methodology and batch effect
I have a question about the normalization of datas and batch effect ...
For example, here is a dataset :
...
- 11
0
votes
0
answers
15
views
Normalization and patient effect
I have a very simple question but I don't really have answer myself... I have a set of data like this :
...
- 11
1
vote
1
answer
102
views
Expected value of the outer product of normalized, non-centered Gaussian vector
I have a multidimensional random variable $X \sim \mathcal{N} \left(\mu, I_d \right)$.
Ideally, I would like to know the expected value of the normalized outer product of the latent variable with ...
- 168
0
votes
0
answers
10
views
Preprocessing of spectroscopy data for PLSR: do I need to normalize the data for every wavelength?
I want to apply a partially least square regression on spectroscopy data to model a chemical content of my probe. So, every wavelength of the spectrum serves as one variable in the model. Doing some ...
- 11
0
votes
0
answers
12
views
How can I calculate growth per year when some of the size values equal zero?
I am trying to retrospectively evaluate growth of tumors over the years. I am looking at the tumor's initial presenting characteristics:size, location etc., and then following to see which of the ...
- 1
0
votes
2
answers
29
views
How can I "normalize" one data set based on a third variable?
My data set is divided into two groups: (1) patients who received physical therapy and (2) patients who did not.
Our primary outcome is looking at how long it took them for their symptoms to resolve (...
0
votes
0
answers
6
views
Should the order of magnitude of response affect machine learning model performance
Does the range for the response variable affect the modeling performance of supervised machine learning models (such as ANN, SVR,...). Many test functions (e.g. perm, zakaharov, etc.) used to create ...
- 307
0
votes
0
answers
18
views
Comparing probability of different length sequences of independent events
Assume I have two different datasets $D_1$ and $D_2$. Each of these datasets has a different number of samples $|D_1| = m$, $|D_2| = n$. Assume each sample is i.i.d drawn from the same underlying ...
0
votes
1
answer
35
views
Standard score applied to data that is not normally distributed
If I scale data from an arbitrary distribution using the standard score, will the property of the normal distribution that 75% of data lies between +/- 2 standard deviations from the mean, still hold?
- 45
0
votes
0
answers
19
views
Comparing gradients for 3 slopes on different scales
I am trying to compare slopes/gradients for three Groups (A, B, and C): the values for each Group are on essentially the same scale for the independent variable (Age), but are on markedly different ...
- 11
0
votes
0
answers
47
views
What is the best way to normalize a timeserie with a trend without differencing it?
On a multivariate forecasting problem (a target and some covariates with known history used to predict future of the target) i'm struggeling with the normalization of my data (covariates and target).
...
- 21
1
vote
1
answer
29
views
Correct renormalization of the product of two probabilistic models
I came across these two papers combining an energy-based model (EBM) with a probabilistic generative model: [Xiao et al., 2020] and [Xiao et al., 2021].
In [Xiao et al., 2020], the new model is ...
- 13
0
votes
0
answers
19
views
How to scale different values to same range?
I'm facing a problem. I have several numerical values in the range from 0 to 1, let's call them "r_i".
Then for every "r_i" I have a threshold "e_i", like a percentage ...
- 23
1
vote
0
answers
23
views
Do you need to normalize labels for models other than neural nets?
As mentioned here, normalizing the target variable often helps a neural network converge faster.
Does it help in convergence, or is there otherwise a reason to use it, for any type of model other than ...
- 11
1
vote
0
answers
28
views
Does the transformation of variables affect the performance of gradient descent?
Suppose we want to minimize the objective function $f(x)$ with respect to $x\in \mathbb{R}^p$. Wtih a linear transformation of $x$, e.g., $z = Qx$ for a proper $Q\in\mathbb{R}^{p\times p}$, the new ...
- 35
0
votes
0
answers
38
views
What's the best data transformation to build a radar chart?
I need to build some radar charts using NBA players stats, but obviously if I don't use the statistics expressed as a percentage (like for example "3-Point Field Goal Percentage" or "...
- 37
0
votes
0
answers
38
views
How do I impose restrictions $ 0\leq \alpha \leq \beta <1$?
I want to restrict values s.t. I get $\theta = (\alpha, \beta) =g(\theta_1, \theta_2)$
with the following restrictions.
$
0\leq \alpha \leq \beta <1
$
I know the correct answer should be $\beta = (...
1
vote
0
answers
20
views
Best way to analyse two experiments
I'm looking for help/indication on approaching a data integration problem.
I have a dose-response curve that is described by a log curve. Because of measurement difficulties, the dose and response can'...
- 11
0
votes
0
answers
56
views
Median, MAD Normalization
I am trying to normalize prices based on the (median, MAD) transformation. I attach a image below describing it in comparison to the (mean, std) normalization.
I have trouble underatanding if ...
0
votes
0
answers
70
views
Is the min-max rescaling or z-score normalization more appropriate when comparing data of vastly different mean, SD, and range?
I have six lists of data in Python where each list contains 23 items as follows.
...
- 1
0
votes
0
answers
12
views
Interpreting negative binomial coefficients from Yeo-Johnson-transformed independent variables
I am running a multiple negative binomial regression with some transformed independent variables. The IVs were skewed, so I performed a Yeo-Johnson power transformation on the data, and then ...
- 23
0
votes
0
answers
72
views
Can I use normalization and standardization on the same dataset?
I'm working on an ML project to predict wine quality from a wine's physical characteristics. The features of my data are on vastly different scales so I've been experimenting with different ...
- 101
1
vote
0
answers
56
views
What's the proper way to analyze data from an experiment where participants perform different tasks?
I've recently conducted an experiment on my university where participants had to perform various tasks using two different virtual reality interaction methods (each participant performed all tasks ...
- 143
1
vote
0
answers
27
views
Regularize variables for Johansen coinegration
I am looking into applying the Johansen cointegration procedure to variables of very different scales, going from units to tens of thousands. I use log(prices) but the scale difference is still very ...
- 85
0
votes
1
answer
39
views
Prediction with normalization of features
I am working with a time series data (date column and a value column) and I have extracted date features(weekday, weekend etc) and rolling means or the prediction.
For features to be given equal ...
- 103
0
votes
0
answers
29
views
Hello, I am doing a machine learning assesment project in R where I have to perform multiple linear regression and write report on all my steps
So I did some data exploratory and found that the range of my features are variable, like some columns have range between 1 to 5, while other columns like 'square_feet' has range of 500 to 2000. In ...
0
votes
0
answers
34
views
Justification for normalization in a ratio scale data
Without loss of generality, I asked a group of participants $X=\{x_1,x_2,\ldots ,x_m\}$ to give scores in a $L$-point ratio scale $0,\ldots,l$ to different items $C=\{c_1,c_2,\ldots ,c_n\}$ based on ...
- 1
0
votes
0
answers
22
views
Help creating a single score using normalization
I'm attempting to create a single performance score based on key performance indicators that have been identified for a brand. The three specific data sets are: social net sentiment, search interest ...
1
vote
0
answers
29
views
Intuition on normalizing to 0 mean and unit variance [duplicate]
What is the intuition, whether geometric or otherwise, to scale 1D scalar data to 0 mean and unit variance? The zero mean seems straightforward, particularly for ML algorithms, but what is the ...
0
votes
0
answers
61
views
Shouldn't I do standardization when data is not normally distributed?
I am trying to scale the data prior to clustering analysis, and got a question.
The goal of scaling at this point is to unify(eliminate) the unit of all input variables so that make influence of each ...
- 23
-2
votes
1
answer
119
views
Why does plot between same scaled and unscaled data look different? [closed]
I am using the StandardScaler of Scikit-learn to scale my data. When plotting the data and comparing scaled with unscaled I get different results in how the plots ...
- 129
0
votes
0
answers
38
views
How do I measure goodness of fit of data transformations that standardize for variables?
I am modeling a dependent variable which has significantly different distributions when grouped by various independent variables. Consequently, it is difficult to compare previous values of the ...
- 133
1
vote
2
answers
330
views
how to normalize data 'with a sample range from -1 to 1 and a mean value of 0'?
I am trying to pre-process data following a statement in a paper.
They said
for the normalization, each dataset is normalized on a per channel basis with a sample range from -1 to 1 and a mean value ...
- 13
0
votes
0
answers
38
views
How normalized, weighted composite score measaured?
How can we convert a vector of repeated measures of a variable into one value when the vector has varying length in different instances? For instance, let's assume ...
- 197
0
votes
0
answers
8
views
Sequence of pre processing methods of machine learning
I have a big dataset. It is a categorical data set. I used label encoder to change the categorical values to the integer values. I would like to find out the co relation between class attribute and ...
- 155
1
vote
1
answer
107
views
Normalize the number of cells per patient to achieve equal contribution of patients for Wilcoxon rank sum test
I'm analyzing a single-cell RNAseq dataset where each patient has a few hundreds of sequenced cells before and after a medical intervention (blue and orange colors respectively in the graph below):
I ...
- 123
2
votes
0
answers
35
views
Normalization/standardization impact on T-SNE and K-means
I have a dataset of 20K samples on 27 features that I am trying to cluster with k-means. The dataset is in its majority rather sparse, i.e. 98% of samples have a single nonzero value in one of its ...
- 21
0
votes
1
answer
24
views
How to normalise sequencing and qRT-PCR data for joint correlation analysis
We would like to analyse the correlation between cytokine (IL-6) and miRNA expression.
However, we quantified the cytokine level in samples, in which the miRNA expression was quantified using either ...
- 21
4
votes
1
answer
378
views
Why do I get a non-zero intercept using the lasso even though I centered the response?
It's my understanding that if I center $y$, the intercept should be 0. However, when using glmnet, I get a non-zero intercept doing this:
...
- 75
0
votes
0
answers
14
views
Normalized Transformations worse than Original Transformation
I am working on a set of hourly prices for two years and I want to transform these to decrease the variance. Since I have negative prices, log is not possible, so I am using an Asinh (area hyperbolic ...