Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [bias-correction]

The tag has no usage guidance.

1
vote
0answers
30 views

Does bias in regression coefficients affect the prediction?

Goal is to create ols model for out of sample prediction for log(wages). Theory say I could have a sample selection bias. So I choose the heckit method to correct for it. The correction term lambda (...
1
vote
1answer
28 views

Calculating bias of ML estimate of AR(1) coefficient

I am trying to develop adjustment factors for maximum-likelihood estimates of the auto-regression coefficient in an AR(1) process. By simulation I have discovered that the estimates are positively ...
3
votes
1answer
33 views

Should I subtract lower bound from Gamma distributed data before estimating distribution parameters?

I have some real world data that reflects waiting time in a system. As it's about waiting times I assume it's Gamma distributed and visual check (histogram overlaid by a fitted Gamma PDF) shows no ...
0
votes
1answer
14 views

Correction for measurement error

Let's suppose that the true model is: $$ y_t^* = x_t^* \beta + e_t^* $$ and suppose that data on $x_t^*$ is observed with error: $$ x_t = x_t^* + u_t $$ If we consider the regression $y_t^* = x_t \...
0
votes
0answers
14 views

Berkson versus classical measurement error: how to choose?

When discussing the consequences of mis-measured variables, an important distinction is to be made whether the measurement error is of the classical, or Berkson type: Classical: $x=a^C + b^Cx^{*}+e$, ...
0
votes
0answers
17 views

Endogenous subgroups in impact estimation

I am using RCT data to estimate the impact of a program. After doing the straightforward analysis, I decided to estimate the program impact by subgroups (treatment status*subgroup). The subgroups were ...
1
vote
1answer
92 views

Bootstrap based bias correction

Assume we have a probablistic model $f_{\theta}(x)$ and try to estimate the parameter $\theta$ based on data $x$ with some procedure that yields a biased estimator $$E[\hat{\theta}]=\theta + \eta,$$ ...
0
votes
0answers
37 views

upward/downward bias of negative variable

If I have a variable that, considering some omitted factor, should have fallen by a higher amount than when it is not there - would that be a downward bias? I.e. the decrease is not large enough, so ...
0
votes
0answers
35 views

Hierarchical clustering: Should I first normalize / correct for phenotypic data?

I've a set of N=100 samples, each sample having M=10 variables (100x10 matrix). These 10 variables (M_i) are responses to some drugs. In addition I've for each sample a list of phenotype data (...
1
vote
0answers
46 views

How to control/correct for response bias in survey or questionnaire data for Factor Analysis

I would like to apply Confirmatory Factor Analysis (CFA) to a Likert-type questionnaire data. It is supposed that this data is affected by response bias: some patients either overestimated or ...
0
votes
0answers
11 views

Log-linear BIAS adjustment

I have a loglinear model of: $log(\mu(S_{ij|gij}))=\alpha_0+\alpha_1g_{ij}$ where gij is distance, and Sij is connectivity There is a bias in the distribution of count values for the outcome ...
0
votes
1answer
27 views

How much bias am I risking by doing model goodness of fit comparison without accounting for clustering?

I am interested in testing whether an interaction term is statistically significant or not in a logistic regression. Data is large and observations are clustered by family and suffer from sparsity for ...
0
votes
0answers
60 views

Biassing priors to improve confusion matrix

I have a text classification problem to solve. I need to classify a given sample of text into one of two classes A or B. My training set has about 30% A and 70% B. This is my prior. Now, when I ...
0
votes
1answer
34 views

How to remove cofounding effect on a variable?

I'm working in a team that is collecting data by bicycle : We have biometric t-shirts that measure our ventilation rate. The problem is that during the last data collection, participants used masks to ...
1
vote
0answers
20 views

Logistic regression where there are unreported failures

Hello: Let's say you have a large number of reported results from a cooperative game. The data consists of a number of independent variables, such as number of players, choice of opposition, etc., ...
0
votes
1answer
49 views

Proper Imputation and bias-correction on degrading signal with Kalman Filtering?

A signal degrades in its quality. Some signals are far more robust to degradation while others are not. We will simulate degradation by randomly removing values from a function and then applying ...
0
votes
0answers
108 views

Trying to understand the boot function and bias in R

I have the following code with the assistance of package "boot", it is very simple so I can learn and you can teach me more effectively: ...
0
votes
1answer
73 views

Correction needed for an non-parametric Anova?

I'm trying to analyze the behavioral data of my research experiment and I'm a bit lost... o_O Briefly, subjects undergo 4 different conditions (of increasing complexity 1<2<3<4). At the end ...
1
vote
1answer
232 views

Understanding a derivation of bias correction for the Adam optimizer

I'm reading paper about the Adam optimizer and went up until the bias-correction section; in the paper they estimate the bias of the moving average of the squared gradient. These are the equations ...
0
votes
1answer
34 views

Choice of deflator in OLS

I am running a pooled OLS model and am not able to internalize change in coefficients/ significance due to change in deflator. The OLS specification is as follows: Market Value = Constant + A1*Profit ...
0
votes
1answer
12 views

Accounting for measurement Bias across manufacturers and multiple treatments

I've been trying to figure out the appropriate statistical approach for the following problem from work (simplified here): I've got 5 manufacturers of a drug and each manufacturer makes their own ...
1
vote
0answers
39 views

Choosing an appropriate statistical distance which punishes entropy

Problem Description: I have with me experimental statistics of a system and I wish to fit a theoretical model so that the computed statistics on the model fit the experimental ones. I am using an RBM ...
0
votes
0answers
16 views

Name of this methodology to correct for bias in parameter estimates

Assume that inferences are to be made on some parameter $\mu$ of a statistical model. A parameter estimate $\hat{\mu}$ has been obtained however it is known that $\hat{\mu}$ has significant bias of ...
1
vote
3answers
142 views

Bonfferroni correction of correlation p-values. What's my k?

I ran a series of Pearson correlations on my data and my supervisor told me to correct the p-values to account for the multiple comparisons I made. She told me to divide the p-value by the number of ...
2
votes
1answer
286 views

Debiasing after LASSO in R

I'm doing machine learning analysis (Lasso) in R. Lasso introduces bias, and I heard I can recover less biased (or unbiased) coefficient estimates by debiasing methods. Is there any R package that ...
0
votes
1answer
340 views

Top-coding and regression

I intend to conduct a multiple linear regression using large US Health Survey data. However, having looked at one of the variables relating to income, it seems that it has been top-coded at the 95th ...
1
vote
1answer
206 views

How to train a Bayesian network with Bernoulli switch variable?

I model my problem as a simple V-structured Bayesian network. There is an $outcome$ variable, the binary $switch$ variable, and some environment features $X$. All the variables are observed during ...
1
vote
1answer
2k views

Bias-corrected percentile confidence intervals

I'm trying to estimate bias-corrected percentile (BCP) confidence intervals in R on a vector from a simple for loop used for resampling. I am primarily looking for help implementing the calculation on ...
5
votes
0answers
52 views

Unbiased estimator for top-k bernoullis

Supposed I have $n$ coins and I'm interested in finding the $k < n$ coins which have the highest odds of coming up heads and I want to know $p(heads)$ for each of these $k$ coins. Assume that I'm ...
3
votes
2answers
75 views

Does adding audits produce less biased data?

My team made a survey about sexual behaviors among college students in China, and the result looks unreasonable1, so we suspect that many of the participants aren't serious when filling out the ...
1
vote
0answers
65 views

Debiasing confidence intervals by people exhibiting overconfidence

Assuming I want to de-bias a confidence interval that was estimated by someone who is overconfident (in the sense of overprecision) in his/her opinion. I do not know how overconfident (i.e. I do not ...
1
vote
1answer
154 views

Non-participation bias: weighting for the inverse of the sampling fraction

This and other studies, where only a proportion of people return a completed questionnaire, suffer from non-participation bias. I have previously just compared participants with non-participants to ...
1
vote
0answers
92 views

Correcting bias in non-equivalent group analysis with logistic regression

I have constructed two groups based on observational data: control (C) and treatment (T). T was exposed to a feature that C was not exposed to. My total number of observations is very high (> 300K). ...
2
votes
0answers
83 views

Survival Analysis - age censoring/bias in independent categorical variables

I suspect that this is a - if not trivial - common question that betrays me as a newbie. Anyhow, here goes... I have data that reflects behaviours of different demographic groups. There are 10 ...
0
votes
1answer
27 views

What does “correct for weather condition” mean?

Many studies show that there exists a positive linear correlation between AOD (Arial Optical Distortion) and PM2.5 (Particulate Matter <2.5mm), after correcting for weather variables such as wind ...
0
votes
0answers
26 views

Bias imbalance in learning algorithm

I have the following model: $$ q^\star = argmin\{\sum_{i=1}^n 𝓁(y_i q^T x_i)+ λ\|q\|^2_2\} $$ where each $x_i$ is a sample of a $p$ dimensional feature vector. As it is common to do, I have encoded ...
1
vote
0answers
123 views

Bias correction term for maximum likelihood estimation of mutual information from joint distributions

According to this webpage, the bias correction term when estimating $I(X;Y)$ for discrete random variables $X,Y$ is $\sim \textrm{df}(X,Y)/N$ where $\textrm{df}(X,Y)$ is the degrees of freedom of the ...
3
votes
1answer
131 views

Bias Correction for Estimator with known bias

To be brief, my question is what could we do when we have an estimator with a known bias. I want to estimate the parameter $\theta$ in a distribution model. I select an estimator $\hat\theta$ (e.g. ...
0
votes
1answer
53 views

Improve fit by trend adjustment

I have data of daily observations for 35 years and I have modeled data for the same period. The coefficient of determination ($R^2$) between them is zero (no correlation at all!). I want to correct ...
1
vote
0answers
14 views

What do you do when you discover a mistake in survey implementation partway through implementation?

I'm working on a survey that is already partway through the implementation phase. It is a web survey, and unfortunately, one of my colleagues noticed that there was an error in the skip logic, leading ...
1
vote
1answer
36 views

Correcting for season-length bias of Gini on win percentage

I made this plot to try and compare the competitiveness of the major US sports (NHL/NBA/MLB/NFL): Each point of a given color represents, for a given season, the Gini coefficient of the win ...
7
votes
1answer
2k views

Additive bias in xgboost (and its correction?)

I am taking part in a competition right now. I know it is my job to do that well, but maybe somebody wants to discuss my problem and its solution here as this could be helfull for others in their ...
1
vote
1answer
60 views

Is it appropriate to compare sports with different-length seasons using Gini?

I made this plot to try and compare the competitiveness of the major US sports (NHL/NBA/MLB/NFL): Each point of a given color represents, for a given season, the Gini coefficient of the win ...
0
votes
1answer
41 views

What statistical tool can be used to correct for differences in the amount of data an individual is evaluated on?

Let's say an individual gets a score (between 1 and 6) on different pieces of equipment in their department. For example, if I'm proficient at repairing a particular piece of equipment I will score ...
2
votes
0answers
248 views

Post-hoc correction of machine learning bias

I have been using a machine learning algorithm to predict a continuous variable, although am having an issue whereby whichever method I use, there is a systematic bias at low and high values of the ...
1
vote
0answers
44 views

How to correct for study bias in protein interaction data?

Interactions between proteins are crucial for the correct functioning of the living cell. That is why it is important to study protein interaction networks, detect hubs, leaves and network modules and ...
1
vote
0answers
143 views

Best estimate of among groups variance with unequal within groups variances

Goal I have about 100,000 sets of groups. For each set, I would like to measure its among groups variance in order to then make comparisons among sets. Description for each set In each set, I have $...
2
votes
2answers
1k views

Closed formula for D4 constant calculation? (Moving range chart constant)

I need to build a Moving Range Shewhart control chart given a series of observations. In short, I have to calculate the central line and the upper and lower limits as follows ...
0
votes
1answer
46 views

censoring as a result of survey screening question

I have a survey data where respondents were asked if they or an immediate family member had previously used a lawyer and, for those who said yes, they were then asked if they had ever sued. I want to ...
0
votes
0answers
188 views

Correcting specific non normal distribution

I have run a multilevel model (time series cross section data; xtmixed) and am checking if the assumptions hold. Given the non-normality of my residuals, and after having corrected potential linearity ...