Skip to main content

Questions tagged [standardization]

Usually refers to "z-standardization" which is shifting and rescaling data to assure they have zero mean and unit variance. Other "standardizations" are possible, too.

Filter by
Sorted by
Tagged with
1 vote
0 answers
10 views

What standard population to use in my epidemiological study?

I need to standardized my mortality rations. for the period 2000-2022 in one European country. However, I am not use what standard population to use? Is Sagi outdated? In addition, I find this ...
aleksandar_m's user avatar
0 votes
1 answer
43 views

why does LASSO regression return unstandardized coefficients [closed]

I have more general questions that does not refer to a coding issue. Why does LASSO regression require standardization of the predictors but return unstandardized coefficients (glmnet function - https:...
Simon's user avatar
  • 1
0 votes
0 answers
31 views

How to handle Data Normalization in case that a Logarithmic scale is required?

Let's say we wished to build a Regressor (e.g. a Support Vector Regressor) to predict the price of an asset, within a given time span from now on. However, what if the historical data we have ...
Juan Flautista De Torrepacheco's user avatar
5 votes
1 answer
118 views

Variables to not standardize in regression?

In context of regression, when does it not make sense to standardize a variable? I understand that in binary & categorical variables, the mean and standard deviation are meaningless. My ...
jbuddy_13's user avatar
  • 3,372
0 votes
0 answers
34 views

Simple Linear Regression, impact of standardizing data

Let's assume we have $(X_i, Y_i), 1 \le i \le n$ a series of $n$ observations. I want to explain $Y^T = (Y_1, \dots, Y_n)$ as a linear function of $X^T = (X_1, \dots, X_n)$. My model is: $$ Y = \...
jocelinbordet's user avatar
2 votes
0 answers
41 views

Standardization of summary statistic of group-linked values

Assume that you measure a summary statistic (e.g., arithmetic mean) in measurement windows of a fixed size along a long sequence of values and that these values are grouped into regions belonging to ...
Michael Gruenstaeudl's user avatar
1 vote
0 answers
54 views

How to use a multi-linear regression to forecast meaningful values

I have built a multi-linear regression model based two predictors $P_1$ and $P_2$ to predict $Q$: $$ q = A + Bx_1 + Cx_2 + Dx_1^2 + Ex_2^2 + Fx_1*x_2 $$ where $x_1$ and $x_2$ are the ...
user avatar
0 votes
0 answers
22 views

should the standardisation of numerical variables be carried out before or after the rebalancing technique of the target variable?

I am dealing with a classification task of a binary target variable (company failure prediction yes or no) for a university project. I was wondering, should the standardisation of numerical variables ...
Elena's user avatar
  • 1
0 votes
0 answers
3 views

Standardizing between time sensitive datasets

I have two datasets for property values in 2020 and 2024, in which I've created two separate models to predict property value. I want to compare the output of these models by using the mean absolute ...
Kevin's user avatar
  • 1
0 votes
0 answers
15 views

Target variable standardization for lasso regression

I am working with different models for a regression task. The range of my target variable is very small: I noticed a very bad performance of the lasso regression and elastic net model in comparison ...
Limmi's user avatar
  • 13
1 vote
2 answers
216 views

When should I standardize, before or after a regression?

I have a panel dataset and my dependent variable is the logit-transformed share of farm workers on long-term contracts. I am particularly interested in the effects of two variables, pastoral focus in ...
Mikhail's user avatar
  • 97
0 votes
0 answers
79 views

Is it bad not to standardize all features (regression)?

I'm working with a neural network with two hidden layers for a regression task. My output values for the training set vary from 0 to 2000 and for the test set from 0 to 600. My main problem is ...
stella's user avatar
  • 1
2 votes
1 answer
61 views

Standardizing necessary before Gower's distance?

I am unsure whether I should standardize my data material (numerical data with different scales) before performing Gower's distance calculation to obtain Gower's distance matrix. If I understood the ...
ExchangedVisual111's user avatar
0 votes
0 answers
13 views

Reporting results for time-invariant predictors in latent growth curve models

I am results for reporting a time invariant predictor of the intercept and slope factors within a latent growth model, adjusted for covariates such as age and sex. My dependent variables and ...
Aepkr's user avatar
  • 309
0 votes
0 answers
22 views

How understand the logic underneath standardization methods?

I have a dataset of very different data and I want to make a multivariate analysis. I am a biology student so I'm very poor in statistics and so I am willing to study this fascinating yet important ...
Jan's user avatar
  • 1
0 votes
0 answers
22 views

Standardization vs Normalization of data

In the above example I am using 3 methods for distance calculation 1.Calculating distances of actual data 2.Standardizing the data and calculating the distances 3.Using min-max scaler and then ...
Hari Krishna's user avatar
4 votes
1 answer
218 views

Standardisation and its effect on regression and the MSE

I am working on a linear regression problem using sklearn's diabetes dataset and coding things from scratch and while evaluating the performance of the model, I have come to question how to move ...
InvestingScientist's user avatar
0 votes
0 answers
29 views

Help interpreting normalized HMM (or otherwise) results

I have run a hidden markov model with five variables on very different scales. Because of this I normalized the input data beforehand using Carets preprocessing: ...
Jcarroll's user avatar
0 votes
0 answers
12 views

Determining power under the assumption that Z_0(test statistic) has the variance of 1

My apology if there already are similar questions. On statistical hypothesis testing; When you design the test for the estimate of binomially-distributed population's mean and are determining the ...
Bipod's user avatar
  • 1
3 votes
2 answers
98 views

Obtain one single growth curve from various replicates in a growth experiment

I am performing a yeast growth experiment measuring areas of growth across a time interval, therefore having area (%) as my dependent variable and time (min) as my independent variable. I have three ...
Paolo Vallejo Janeta's user avatar
1 vote
1 answer
79 views

Non-standardized vs Standardized Data Normality

I have a dataset with over 900 cases. I performed a Q-Q Plot on the non-standardized HRV data (dependent variable), and the points formed a wavy curve indicating it does not follow a normal ...
Mark S.'s user avatar
  • 41
0 votes
0 answers
27 views

Normalize age frequency data for PCA

I am working on a project to forecast house ownership rates. One dataset I have consists of number of people of each age from 1-99 per geographic area code. For example, 20 people aged 1, 59 people ...
burn_burn_55's user avatar
1 vote
1 answer
43 views

Mathematics behind standardizing the data points in machine learning algorithms (e.g., K-means clustering)

For K-means algorithm, among other methods using distance-based measurements to determine similarity between data points, why we have to standardize the data points with mean as 0 and standard ...
Sophia's user avatar
  • 121
1 vote
1 answer
87 views

Calculating standardised regression coefficients from GLMER model

I have three separate glmer models investigating the individual and household-level risk factors of malaria infection in three different spatial locations: 1) outside the forest, 2) at the forest ...
Trypanosoma's user avatar
0 votes
0 answers
78 views

Understanding the Normalization/Standardization of geospatial coordinates

I'm building a neural network to predict future [latitude,longitude,altitude], and am having trouble dealing with the features. I've reviewed the answers to the ...
LivelyECDSA's user avatar
0 votes
0 answers
26 views

Normalising series data of variable length

I have a model m which takes as input a starting array of n_input elements which are actually traces of mining processes and ...
Shivam Roy's user avatar
1 vote
1 answer
31 views

How to model a standardized index in a regression?

I have a standardized index, $x$ (variations in s.d.) and I want to regress my dependent variable, $y$ on it. In my dataset, the index ranges from approximately -2 to 2, but there is no constraint. ...
Oalvinegro's user avatar
0 votes
0 answers
27 views

Why do the standardized beta values and CIs of a glm poisson regression model not differ from the unstandardized ones (using report function)?

For a specific research question i fitted a generalized linear mixed model using a poisson link function due to the characteristics of my data. For reporting purposes i used the report package and the ...
LukasDphd's user avatar
0 votes
0 answers
33 views

Normalization/standardization of time series data

I have energy consumption data where rows represent different users and columns are different measurements. I don't really understand, how and in which order i need to normalize and standardize the ...
Moulmein's user avatar
0 votes
1 answer
114 views

Data Standardization in DBSCAN Clustering

"If the data has a much different range of measurement values, data standardization is carried out so that the distance calculation becomes effective. For example, if the income variable is ...
Anna's user avatar
  • 3
0 votes
0 answers
73 views

Do I need to standardise variables when computing an index of interest?

Suppose I have to create an "index of interest" for the products of an e-commerce. By index of interest I mean a $parameter$ which, based on some variables, tells me which products are the ...
TF7's user avatar
  • 37
5 votes
2 answers
206 views

Correcting for selection bias with standardisation/g-computation

Two sets of methods for correcting for selection bias are g-computation (standardisation) and inverse probability of censoring weighting (IPCW). I'm having a difficult time understanding how to apply ...
Lachlan's user avatar
  • 1,192
0 votes
0 answers
18 views

How to do standardizing of time series data for regression tasks?

I'm using an LSTM-based model to train a regression model. I standardize the data (i.e., zero mean and unit standard deviation) (train, valid and test have different means and STDs). The model works ...
A User's user avatar
  • 1
0 votes
0 answers
130 views

Normalization of log-returns or normalization of cumulative log-returns

This questions seeks for discussion to find theoretical support for normalizing cumulative log-returns vs normalizing log-returns By "normalizing" (also known as standarizing) I mean it in ...
Alfonso's user avatar
  • 21
0 votes
0 answers
17 views

VaR (value-at-risk) of a Normal Random Variable - Confused about Scaling in Derivation

In the book "Quantitative Risk Management: Concepts, Techniques and Tools - Revised Edition" (by McNeil, Frey, and Embrechts)", there is the following example (example 2.11): Suppose ...
randomvariable's user avatar
4 votes
2 answers
424 views

Hypotheses Testing - Correlation vs. Regression

I conducted a survey for my thesis and would like to begin with the hypotheses testing now. However, I am not sure what the correct process is - sorry if these are really basic questions! Specifically,...
ber1495's user avatar
  • 57
1 vote
1 answer
219 views

Why is the sample standard deviation used in the z-test?

The assumption of the z-test is that the population standard deviation $\sigma$ is known. With this in mind I tried to manually compute the p-value for a (one-tailed) z-test that the sample mean $\bar{...
monade's user avatar
  • 519
0 votes
1 answer
107 views

Normalize vs Standarization with percentage data

I'm studying access patterns to a facility with clustering. My variables are percentages. For example, for each user, I have the percentage of access 'in time' versus late, or the percentage of using ...
Kaikus's user avatar
  • 219
13 votes
4 answers
1k views

I've always learned that data standardization is not necessary for OLS regression, but then recommended for neural networks. Intuitively, why is that?

I know for LASSO and elastic net regression, standardization is important, because coefficient penalization in regularization will be biased if the ranges of data are different. Meanwhile, OLS ...
cisisc's user avatar
  • 131
3 votes
3 answers
214 views

How to decorrelate $X$ and $X^3$?

We know that if $X$ is positive, then $X^2$ is highly positively correlated with $X$. I've plotted an array of integer numbers from 100 and 110 with the following code: ...
ricber's user avatar
  • 53
0 votes
1 answer
156 views

How to convert six point likert scale to four point likert scale

I would like to convert the mean score of a sample that was administered a questionnaire using a six point likert scale into a mean that is comparable to the same questionnaire using a four point ...
Aksel's user avatar
  • 1
1 vote
1 answer
20 views

Can I create effect sizes from published reports without access to the data? Thinking about standardizing unstandardized coefficients?

I'm reviewing and synthesizing 48 published articles on the same topic (association between x and y). Instead of describing findings based on statistical significance and power, I'd like to be able to ...
medusa's user avatar
  • 11
0 votes
0 answers
87 views

Standardizing level-1 predictors in multilevel models

There has been discussion about this topic (e.g. here, here and this recent question with no answers yet prompted me to ask this as I haven't found a clear answer to this question). So, when fitting ...
Sointu's user avatar
  • 2,448
3 votes
2 answers
94 views

Univariate vs. Multivariate Standardization

There are several common methods for scaling input features to machine learning models prior to training the model. The most popular methods seem to be standardization (centering by the mean and ...
noNameTed's user avatar
  • 135
1 vote
0 answers
23 views

Counts, percentages and scaling in clustering

I am working in a clustering. I have a dataset with the accesses of a series of subjects to a facility (each row register the subject ID and the date/time of the access) I can, for example, count the ...
Kaikus's user avatar
  • 219
0 votes
0 answers
49 views

Age-sex standardization over study period

I have 10 years of health (number of cases of health outcome, such as asthma cases) and population data, but am using the data cross sectionally because the exposure data are also cross-sectional (due ...
enat_b's user avatar
  • 1
0 votes
0 answers
17 views

Feature Engineering with Standardized Data

I'm relatively new to feature engineering, but at a high-level what I understand that it does is takes various features in a dataset (perhaps two, perhaps more) and combines them in such a way using ...
Hau5's user avatar
  • 21
4 votes
2 answers
1k views

Is calculating skewness necessary before using the z-score to find outliers?

For example, if I specify a z-value of 3, then I would look at both sides and know its position in the distribution (99.73%). Would this change if I have a left or right skewed distribution? Would I ...
JAdel's user avatar
  • 125
0 votes
0 answers
30 views

Is it OK calculating and compare the differences between standardized (z-log) means of different subsets of a dataset?

I have a large dataset of different species-level traits like leaf length, lifespan, height, etc that I am combining with a time-series occurrence dataset to see changes in mean trait values over time....
Abermals's user avatar
2 votes
1 answer
192 views

Why it is called "BatchNorm" not "Batch Standardize"?

Regarding the differences between "Normalization" and "Standardization," I found that: Normalization: Is the process of making a dataset having a specified range, probably [0,1] ...
Abdallah WallyAllah's user avatar

1
2 3 4 5
17