Questions tagged [bias-correction]
The bias-correction tag has no usage guidance.
56
questions with no upvoted or accepted answers
5
votes
1
answer
88
views
Unbiased estimator for top-k bernoullis
Supposed I have $n$ coins and I'm interested in finding the $k < n$ coins which have the highest odds of coming up heads and I want to know $p(heads)$ for each of these $k$ coins.
Assume that I'm ...
- 239
4
votes
0
answers
206
views
Bias correcting penalized maximum likelihood / maximum a posteriori estimates
Suppose an estimator $\hat\theta_T$ is defined as the value of $\theta$ maximizing:
$$\sum_{t=1}^T{l(y_t|\theta)}+\mu_T g(\theta),$$
where $l(y_t|\theta)$ is the log-likelihood of observation $t$, $\...
- 535
3
votes
0
answers
64
views
Estimating the Absolute Difference in Mean Between Two Normal RVs
I've seen more general questions of this nature, but none that discuss specifically the problem setup that I'm interested in, which I believe is substantially simpler.
Suppose that we observe two ...
- 41
3
votes
0
answers
2k
views
How to back transform from log10 ~ log10 regression in order to predict?
I would like to understand the results of this paper (they supply all of their R code and raw data here). The idea is to regress home range on body mass for a range of taxa.
For one group of birds, ...
- 1,001
3
votes
1
answer
433
views
How to train a Bayesian network with Bernoulli switch variable?
I model my problem as a simple V-structured Bayesian network. There is an $outcome$ variable, the binary $switch$ variable, and some environment features $X$. All the variables are observed during ...
- 530
3
votes
0
answers
1k
views
Kalman Filter to correct model simulation bias
I am working with a large scale deterministic model, which attempts to simulate CO2 emissions in different regions. When compared to historic data, the model output suffers from systematic biases.
...
- 381
2
votes
0
answers
233
views
ADAM bias correction derivation
Where $\beta_2\in [0,1)$ and $g_i$ is the gradient at step i.
We approximate E[gi^2] with E[gt^2] by a correction term $\zeta$. I think this is because it is assumed that the gradients are bounded. Is ...
- 21
2
votes
0
answers
73
views
How to average across multiple regressions, correct for p-value and estimates?
In this data frame I have 2 replicates that were placed in low, mid, or high areas with recorded densities at x and y. Within a certain level e.g. rep 1 "mid" has x = 24 and y = NAs. ...
- 603
2
votes
0
answers
122
views
Correcting for reliability scores for Hedges g
I've received a revise and resubmit on my meta-analysis investigating the effects of a positive psychology intervention on depression and anxiety symptoms. I used Hedges' g as the effect size within a ...
- 85
2
votes
0
answers
109
views
Survival Analysis - age censoring/bias in independent categorical variables
I suspect that this is a - if not trivial - common question that betrays me as a newbie. Anyhow, here goes...
I have data that reflects behaviours of different demographic groups.
There are 10 ...
- 151
2
votes
0
answers
237
views
Bias correction term for maximum likelihood estimation of mutual information from joint distributions
According to this webpage, the bias correction term when estimating $I(X;Y)$ for discrete random variables $X,Y$ is $\sim \textrm{df}(X,Y)/N$ where $\textrm{df}(X,Y)$ is the degrees of freedom of the ...
- 1,059
2
votes
1
answer
73
views
Correcting for season-length bias of Gini on win percentage
I made this plot to try and compare the competitiveness of the major US sports (NHL/NBA/MLB/NFL):
Each point of a given color represents, for a given season, the Gini coefficient of the win ...
- 1,422
2
votes
0
answers
492
views
Post-hoc correction of machine learning bias
I have been using a machine learning algorithm to predict a continuous variable, although am having an issue whereby whichever method I use, there is a systematic bias at low and high values of the ...
- 738
2
votes
0
answers
225
views
Best estimate of among groups variance with unequal within groups variances
Goal
I have about 100,000 sets of groups. For each set, I would like to measure its among groups variance in order to then make comparisons among sets.
Description for each set
In each set, I have $...
- 5,112
2
votes
0
answers
708
views
Selection bias correction and a multinomial logit
I have a data set for a number of people making 2 decisions - where to live; and how many hours to work. For every observation with a non-zero amount of work, there is an observed wage.
I've assigned ...
- 21
2
votes
0
answers
98
views
Bias correction methods in G-side GLMMs
Are there any conventions for the use of bias correction methods in G-side generalized linear mixed models (GLMMs)?
For linear mixed models I usually use the Kenward and Roger adjustment, but it is ...
- 1,231
1
vote
0
answers
18
views
How can I take into account multiple individuals with multiple observations across a data?
I have a set of data as this:
ID V1 V2 V3
A 12 10 8
A 11 9 10
B 7 10 8
C 13 10 9
C 10 12 6
This dataset is from health data, where each individual takes ...
- 67
1
vote
0
answers
26
views
How to account for bias in experiment data when quantifying treatment efficacy?
Say my randomization wasn't very effective, so I have two groups, each with 100 individuals and the difference in success rates is 3% before the treatment has ever been administered.
I want to know ...
- 3,020
1
vote
0
answers
130
views
Implementing bias-adjustion for step3 latent profile analysis in R
I am identifying a latent profile model with the Mclust package in R. After identifying an optimal number of cluster I would like to identify possible covariates and distal outcomes via logistic/...
- 31
1
vote
1
answer
46
views
Suggestions on dealing with outliers when sample size is very small AND you must order the results
I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry.
For each category we throw out the high and low scores and average the remaining ...
- 11
1
vote
0
answers
101
views
Python implementation of standard Expectation maximization algorithm to estimate position bias in LTR
Is there any Python implementation that would allow to use logs data of clicks and their ranking, and return and estimate of relative position bias as discussed in section ...
- 11
1
vote
0
answers
91
views
Standard deviation in generated data with specified autocorrelation - correction factor?
I want to generate time-series data with specified AR(1) and standard deviation.
I am using arima.sim to generate univariate time series.
(Ultimately, I want to use ...
- 11
1
vote
0
answers
68
views
Biased logistic regression in pytorch
My model has decently high AUC=90%, but is biased, underestimating the probability $y=1$. This is systematic across some of the input features as well. How can I nudge the bias term, or otherwise ...
- 41
1
vote
0
answers
101
views
How to combine stratified sampling and avoid common driver in both training & test set - R Cross-Validation
I want to build a random forest or gradient boosting model on very heterogeneous data using cross-validation in R.
Due to data heterogeneity and dependency between the observation (the same subject ID ...
- 21
1
vote
0
answers
54
views
How can I understand ATT estimator for matching discrepancies in Causal Inference Mixtape?
While studying 'Causal Inference: Mixtape' by myself, something I didn't know happened.
link: https://mixtape.scunning.com/matching-and-subclassification.html#bias-correction
$\begin{align} \...
- 73
1
vote
0
answers
166
views
How to avoid underestimation in neural network model when loss targets contain min something positive and model output itself?
Apologies if this has been asked already, but I couldn't find anything about it.
Straight to the problem:
I want to train a neural network for regression and the target in the loss function is a min ...
- 11
1
vote
0
answers
24
views
Correcting for known imputation bias
I have a survey sample which includes income values by type of income for a significant number of high-income households. Some of the income data is measured, i.e. present in the responses of the ...
- 3,117
1
vote
0
answers
152
views
Sampling bias in multinomial logistic regression
I am interested in estimating a set of coefficients in a multinomial logistic model. However, I only observe a subsample of the true sample in which base category $A$ was chosen. I have no way of ...
- 11
1
vote
0
answers
212
views
Does bias in regression coefficients affect the prediction?
Goal is to create ols model for out of sample prediction for log(wages). Theory say I could have a sample selection bias. So I choose the heckit method to correct for it. The correction term lambda (...
1
vote
1
answer
67
views
Correction for measurement error
Let's suppose that the true model is:
$$ y_t^* = x_t^* \beta + e_t^* $$
and suppose that data on $x_t^*$ is observed with error:
$$ x_t = x_t^* + u_t $$
If we consider the regression $y_t^* = x_t \...
- 1,045
1
vote
0
answers
167
views
How to control/correct for response bias in survey or questionnaire data for Factor Analysis
I would like to apply Confirmatory Factor Analysis (CFA) to a Likert-type questionnaire data. It is supposed that this data is affected by response bias: some patients either overestimated or ...
- 11
1
vote
0
answers
34
views
Logistic regression where there are unreported failures
Hello: Let's say you have a large number of reported results from a cooperative game. The data consists of a number of independent variables, such as number of players, choice of opposition, etc., ...
1
vote
0
answers
46
views
Choosing an appropriate statistical distance which punishes entropy
Problem Description: I have with me experimental statistics of a system and I wish to fit a theoretical model so that the computed statistics on the model fit the experimental ones. I am using an RBM ...
- 111
1
vote
0
answers
113
views
Debiasing confidence intervals by people exhibiting overconfidence
Assuming I want to de-bias a confidence interval that was estimated by someone who is overconfident (in the sense of overprecision) in his/her opinion. I do not know how overconfident (i.e. I do not ...
- 33
1
vote
0
answers
130
views
Correcting bias in non-equivalent group analysis with logistic regression
I have constructed two groups based on observational data: control (C) and treatment (T). T was exposed to a feature that C was not exposed to. My total number of observations is very high (> 300K).
...
- 411
1
vote
0
answers
15
views
What do you do when you discover a mistake in survey implementation partway through implementation?
I'm working on a survey that is already partway through the implementation phase. It is a web survey, and unfortunately, one of my colleagues noticed that there was an error in the skip logic, leading ...
- 111
1
vote
1
answer
122
views
Is it appropriate to compare sports with different-length seasons using Gini?
I made this plot to try and compare the competitiveness of the major US sports (NHL/NBA/MLB/NFL):
Each point of a given color represents, for a given season, the Gini coefficient of the win ...
- 1,422
1
vote
0
answers
51
views
How to correct for study bias in protein interaction data?
Interactions between proteins are crucial for the correct functioning of the living cell. That is why it is important to study protein interaction networks, detect hubs, leaves and network modules and ...
- 61
0
votes
0
answers
49
views
Incorporating Inverse Probability Weights in phylogenetic imputation
In Cortes et al. “Sample Selection Bias Correction Theory.” In Algorithmic Learning Theory, 5254:38–53. Lecture Notes in Computer Science. 2008., they describe how inverse probability weights can be ...
- 228
0
votes
0
answers
19
views
Correcting for spillover bias in experiments?
Suppose I have an experiment designed where I'm interested in observing the effect of daily walking on self-reported joint pain in elderly adults. The population of interest are elderly adults who ...
- 3,020
0
votes
0
answers
38
views
Correction of biased probabilities
I have a classifier that outputs biased predictions, that I want to correct for in a mathematically sound way.
I define bias as the difference between the true class distribution and the average ...
- 211
0
votes
0
answers
33
views
Significance of parameter on cointegrating vector
I have been reading Section 6.2 (page 96) of this manual, where a procedure called "fully modified least squares" is discussed, and was developed on a paper of Phillips and Hansen (1990)
. ...
- 101
0
votes
0
answers
140
views
How to bias-correct proportions for a prop.test in R?
Let's say I ask 200 participants in a study what experience brought the most joy to their lives:
A, B, C, or D.
...
- 1
0
votes
1
answer
77
views
Reducing bias for estimate of population standard deviation
https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation#Other_distributions
"For non-normal distributions an approximate (up to O(n−1) terms) formula for the unbiased estimator of ...
0
votes
1
answer
244
views
Correcting for bias in GEE models with small cluster size
In GEE, several methods have been proposed for correcting for bias when the cluster size is small to moderate (<40). Some have proposed alternative variance estimators, e.g. Morel, Bokossa, and ...
- 36
0
votes
0
answers
431
views
upward/downward bias of negative variable
If I have a variable that, considering some omitted factor, should have fallen by a higher amount than when it is not there - would that be a downward bias? I.e. the decrease is not large enough, so ...
- 21
0
votes
0
answers
127
views
Hierarchical clustering: Should I first normalize / correct for phenotypic data?
I've a set of N=100 samples, each sample having M=10 variables (100x10 matrix). These 10 variables (M_i) are responses to some drugs. In addition I've for each sample a list of phenotype data (...
- 564
0
votes
0
answers
152
views
Biassing priors to improve confusion matrix
I have a text classification problem to solve. I need to classify a given sample of text into one of two classes A or B.
My training set has about 30% A and 70% B. This is my prior. Now, when I ...
- 101
0
votes
1
answer
70
views
How to remove cofounding effect on a variable?
I'm working in a team that is collecting data by bicycle : We have biometric t-shirts that measure our ventilation rate.
The problem is that during the last data collection, participants used masks to ...
- 407
0
votes
1
answer
90
views
Correction needed for an non-parametric Anova?
I'm trying to analyze the behavioral data of my research experiment and I'm a bit lost... o_O
Briefly, subjects undergo 4 different conditions (of increasing complexity 1<2<3<4).
At the end ...
