Questions tagged [standardization]
Usually refers to "z-standardization" which is shifting and rescaling data to assure they have zero mean and unit variance. Other "standardizations" are possible, too.
835
questions
0
votes
1
answer
20
views
How to model a standardized index in a regression?
I have a standardized index, $x$ (variations in s.d.) and I want to regress my dependent variable, $y$ on it.
In my dataset, the index ranges from approximately -2 to 2, but there is no constraint.
...
- 347
0
votes
0
answers
21
views
Why do the standardized beta values and CIs of a glm poisson regression model not differ from the unstandardized ones (using report function)?
For a specific research question i fitted a generalized linear mixed model using a poisson link function due to the characteristics of my data.
For reporting purposes i used the report package and the ...
0
votes
0
answers
19
views
Normalization/standardization of time series data
I have energy consumption data where rows represent different users and columns are different measurements. I don't really understand, how and in which order i need to normalize and standardize the ...
- 1
0
votes
1
answer
39
views
Data Standardization in DBSCAN Clustering
"If the data has a much different range of measurement values, data standardization is carried out so that the distance calculation becomes effective. For example, if the income variable is ...
- 3
0
votes
0
answers
68
views
Do I need to standardise variables when computing an index of interest?
Suppose I have to create an "index of interest" for the products of an e-commerce. By index of interest I mean a $parameter$ which, based on some variables, tells me which products are the ...
- 37
5
votes
1
answer
103
views
Correcting for selection bias with standardisation/g-computation
Two sets of methods for correcting for selection bias are g-computation (standardisation) and inverse probability of censoring weighting (IPCW). I'm having a difficult time understanding how to apply ...
- 1,162
0
votes
0
answers
16
views
How to do standardizing of time series data for regression tasks?
I'm using an LSTM-based model to train a regression model. I standardize the data (i.e., zero mean and unit standard deviation) (train, valid and test have different means and STDs). The model works ...
- 1
0
votes
0
answers
31
views
Normalization of log-returns or normalization of cumulative log-returns
This questions seeks for discussion to find theoretical support for normalizing cumulative log-returns vs normalizing log-returns
By "normalizing" (also known as standarizing) I mean it in ...
- 21
0
votes
0
answers
14
views
VaR (value-at-risk) of a Normal Random Variable - Confused about Scaling in Derivation
In the book "Quantitative Risk Management: Concepts, Techniques and Tools - Revised Edition" (by McNeil, Frey, and Embrechts)", there is the following example (example 2.11):
Suppose ...
4
votes
2
answers
379
views
Hypotheses Testing - Correlation vs. Regression
I conducted a survey for my thesis and would like to begin with the hypotheses testing now. However, I am not sure what the correct process is - sorry if these are really basic questions!
Specifically,...
- 57
1
vote
1
answer
58
views
Why is the sample standard deviation used in the z-test?
The assumption of the z-test is that the population standard deviation $\sigma$ is known. With this in mind I tried to manually compute the p-value for a (one-tailed) z-test that the sample mean $\bar{...
- 509
0
votes
1
answer
36
views
Normalize vs Standarization with percentage data
I'm studying access patterns to a facility with clustering.
My variables are percentages. For example, for each user, I have the percentage of access 'in time' versus late, or the percentage of using ...
- 219
13
votes
4
answers
1k
views
I've always learned that data standardization is not necessary for OLS regression, but then recommended for neural networks. Intuitively, why is that?
I know for LASSO and elastic net regression, standardization is important, because coefficient penalization in regularization will be biased if the ranges of data are different.
Meanwhile, OLS ...
- 131
3
votes
3
answers
168
views
How to decorrelate $X$ and $X^3$?
We know that if $X$ is positive, then $X^2$ is highly positively correlated with $X$.
I've plotted an array of integer numbers from 100 and 110 with the following code:
...
- 53
0
votes
1
answer
55
views
How to convert six point likert scale to four point likert scale
I would like to convert the mean score of a sample that was administered a questionnaire using a six point likert scale into a mean that is comparable to the same questionnaire using a four point ...
- 1
1
vote
1
answer
17
views
Can I create effect sizes from published reports without access to the data? Thinking about standardizing unstandardized coefficients?
I'm reviewing and synthesizing 48 published articles on the same topic (association between x and y). Instead of describing findings based on statistical significance and power, I'd like to be able to ...
- 11
0
votes
0
answers
43
views
Standardizing level-1 predictors in multilevel models
There has been discussion about this topic (e.g. here, here and this recent question with no answers yet prompted me to ask this as I haven't found a clear answer to this question).
So, when fitting ...
- 1,086
2
votes
2
answers
46
views
Univariate vs. Multivariate Standardization
There are several common methods for scaling input features to machine learning models prior to training the model. The most popular methods seem to be standardization (centering by the mean and ...
- 88
1
vote
0
answers
15
views
Counts, percentages and scaling in clustering
I am working in a clustering.
I have a dataset with the accesses of a series of subjects to a facility (each row register the subject ID and the date/time of the access)
I can, for example, count the ...
- 219
0
votes
0
answers
34
views
Age-sex standardization over study period
I have 10 years of health (number of cases of health outcome, such as asthma cases) and population data, but am using the data cross sectionally because the exposure data are also cross-sectional (due ...
- 1
0
votes
0
answers
15
views
Feature Engineering with Standardized Data
I'm relatively new to feature engineering, but at a high-level what I understand that it does is takes various features in a dataset (perhaps two, perhaps more) and combines them in such a way using ...
- 21
4
votes
2
answers
793
views
Is calculating skewness necessary before using the z-score to find outliers?
For example, if I specify a z-value of 3, then I would look at both sides and know its position in the distribution (99.73%).
Would this change if I have a left or right skewed distribution? Would I ...
- 125
0
votes
0
answers
29
views
Is it OK calculating and compare the differences between standardized (z-log) means of different subsets of a dataset?
I have a large dataset of different species-level traits like leaf length, lifespan, height, etc that I am combining with a time-series occurrence dataset to see changes in mean trait values over time....
- 1
2
votes
1
answer
120
views
Why it is called "BatchNorm" not "Batch Standardize"?
Regarding the differences between "Normalization" and "Standardization," I found that:
Normalization: Is the process of making a dataset having a specified range, probably [0,1] ...
0
votes
0
answers
22
views
Pre-processing (standardization and exponential smoothing) in expanding window cross-validation for time series data
I am currently working on a time series forecasting task using an expanding window cross-validation approach. My dataset is created using the sliding window technique (Window: 4, Horizon: 1). During ...
- 1
1
vote
0
answers
62
views
Should standardizing a vector of data lead to a unit length vector? [closed]
I am reading "Introduction to Econophysics" by Stanley and Mantegna and I found the following in Chapter 13.
I can't understand why the 2-Norm of a vector standardized by subtracting the ...
- 11
0
votes
0
answers
28
views
Can I standardize a binary dependent variable? and what would be the interpretation of the regression coefficients?
I have 3 linear regressions with differently scaled dependent variables and the same independent varibales. I standardized the dependent variables for comparison reasons. The other regression includes ...
0
votes
0
answers
31
views
Why does scaling of one predictor influence the coefficient estimates of other predictors in ridge regression?
In Introduction to Statistical Learning it is written
The standard least squares coefficient estimates discussed in Chapter 3
are scale equivariant: multiplying $X_j$ by a constant $c$ simply leads ...
- 59
1
vote
1
answer
23
views
In the context of machine learning, at what stage is the normalization process implemented?
I have several data types and I want to use them as features for binary classification task.
The data is as follows:
1- Genes: I have several datasets containing bulk RNA-seq data. Some of these ...
- 479
1
vote
0
answers
41
views
Standardizing dependent variable to model random intercept and slope
I have a model for a small data set (30 participants) where I would like to specify random intercepts for participants and a random slope for their ages, but don't have enough observations per ...
- 11
0
votes
0
answers
54
views
Per Mel Spectrogram min-max normalization vs full training set min-max normalization for CNN classification of audio
I am watching a tutorial on using mel spectrograms to classify the audio's genre via CNN. My question is why apply local min-max normalization to each individual mel spectrogram? What I mean by local ...
0
votes
0
answers
33
views
How to normalize dataset with mixed data (continuous and count)?
I am trying to determine what procedure should I use to feature engineer the most descriptive possible dataset to predict a binary outcome. The dataset has variables that are count-valued with ...
- 253
0
votes
0
answers
16
views
Standardize agglomerative feature clustering across samples or features?
I know that typically, one has a feature matrix of n samples by m features. Let's say I have a matrix X in this format. If I was going to perform hierarchical clustering on the samples, I know I ...
- 1
2
votes
1
answer
49
views
How should I scale data that has been assembled from different data sources?
The data I'm woking with consists of 3 types of data:
1- binary features: those features are either 0 or 1. I have about 6 or 7 columns.
2- cells: the values here range from 0 to 0.8 at max. Here I ...
- 479
0
votes
0
answers
33
views
Z-Score to compare data with different time periods?
I have survey data for multiple countries, with respondents nested in countries. The survey was conducted in different waves, but not all countries participated in all waves. Thus, there are countries ...
0
votes
0
answers
15
views
Can I safely compare standardized coefficients between datasets with differents numbers of observations?
For simplicity, let's say that I regress Y on X using OLS in three different datasets, which all have different numbers of observations (1200, 1500 and 1900 respectively).
In all three datasets, I ...
- 81
0
votes
0
answers
26
views
Can I use standardised coefficients in ordinal logistic and Poisson regressions?
I ran some OLS regressions with standardised coefficients, but to test the robustness of my results, I also want to run an ordinal logistic regression, as well as a Poisson model, using count data.
My ...
- 81
1
vote
0
answers
166
views
Normalization or standardization in stock prediction
Currently I watched the videos (links below) that argues using the normalization (max-min scale) is the bad idea when it comes to the stock prediction. In the videos, the editor aruges that people ...
- 63
0
votes
0
answers
15
views
Navigating Differences in Scale: A multivariate approach with standardization or separate component analysis?
I am conducting a research study on sex differences in the physical limitations of patients with ankylosing spondylitis (AS), using the Bath Ankylosing Spondylitis Metrology Index (BASMI). The BASMI ...
- 93
1
vote
1
answer
37
views
Machine learning regression using centered and scaled data
Imagine I have a large dataset of many variables and many observations. I would like to create a regression model to predict the values of new data.
For the sake of ease, say I find a ridge regression ...
- 113
2
votes
1
answer
50
views
Can I reasonably estimate the population mean and standard deviation from a large sample all taken at a single percentile?
I am currently looking at a dataset of Fair Market Rents which are determined at different percentiles over the years - for example, nationally in 1983 they were all set at the 40th percentile, and in ...
- 21
1
vote
0
answers
19
views
Stardardization for Random Forest, SVM and Logistic Regression
I have a classification project and I want to compare three models: Random Forest, SVM and Logistic Regression.
Random Forest are tree based algorithms wheras, SVM is a distance based model and LR is ...
- 11
1
vote
0
answers
48
views
Standardized and unstandardized canonical correlation coefficients
What exactly are the standardized and unstandardized canonical correlation coefficients and what is the difference between them?
- 11
5
votes
2
answers
489
views
Can you combine two different metrics on different scales by Z-scoring?
A colleague wants to analyze two outcomes (X1, X2). They believe the two outcomes measure the same construct or something similar. They decide to Z-score both outcomes in separate dataset and combine ...
- 818
0
votes
0
answers
122
views
Bias towards categorical data when one-hot encoding and standardizing (for machine learning)
I have a dataset containing a fair amount of continuous and categorical variables. I one-hot encode these variables to be used in various machine learning algorithms.
Let's presume a variable has n ...
- 63
1
vote
0
answers
29
views
How to create a composite variable out of 3 non-correlated variables?
I am working on a model to account for flood risk and it is based on three variables:
Variable 1: drainage (float: 0 - 80)
Variable 2: estimated population (float: 0-2,000)
Variable 3: road network ...
- 11
2
votes
1
answer
302
views
Do we lose information when we normalize an image? [closed]
Before training a machine learning algorithms, it is advisable to perform feature scaling. Suppose we have a "toy" dataset where each image is composed of two pixels $x_0$ and $x_1$. Lets ...
- 452
1
vote
0
answers
28
views
Transformation to compare many distributions with different means and variances
I have daily sales (total volume in dollars) from 200 stores of the same franchise, over two years. I would like to identify any store with anomalies or special patterns, which could be the sign of a ...
- 111
0
votes
0
answers
36
views
Uses of data standardization without subtracting mean
From what I've seen, it is common practice in Deep Reinforcement Learning to standardize certain data. By standardization, I refer to the process of subtracting the mean and dividing by the standard ...
- 101
1
vote
0
answers
72
views
Standardization of out-of-sample data
I have a panel (N firms across 10 years) dataset on which I want to estimate and test a prediction model $f$:
\begin{equation}
y = f(x).
\end{equation}
Following common practice, I split my data into ...
- 1,029