Skip to main content

Questions tagged [bias-correction]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
1 answer
48 views

What is the difference between using logistf and brglm2 when dealing with complete separation in a logistic regression?

I am trying to looking at how the three factors A (5 levels, a-e), B (2 levels, a and b) and C (2 levels, a and b) affect the likelihood of event Y (1 = occured, 0 = did not occur). I initially ran a ...
Insect_biologist's user avatar
7 votes
2 answers
130 views

What to show as error-bar if the bootstrap distribution is biased?

Say I have a sample, of finite size $N$, and I compute some statistic $\theta$ from it. I want to plot this sample estimate, $\hat{\theta}$, with an error-bar. To compute the error, I am using ...
Luismi98's user avatar
  • 170
0 votes
0 answers
15 views

Methods to level spatiotemporal data when simultaneous measurements of the same physical quantity are different

I have data of (simulated) measurements of the density content of ionized ozone in the atmosphere with three different satellites. Specifically, I have a unique set of observations x1,x2,x3,...xN for ...
requiemman's user avatar
0 votes
0 answers
29 views

Why does the jackknife reduce bias? [duplicate]

Given a sample $x = (x_1, \ldots, x_n)$, define $x_{(-i)}$ as the sample values excluding sample $x_i$. That is, $$ x_{(-i)} = (x_1, \ldots, x_{i-1}, x_{i+1}, \ldots x_n). $$ Now given estimator $T(x)$...
Adam Cataldo's user avatar
1 vote
1 answer
41 views

For a biased estimator, how does one call the point for which the expected value of the estimator is equal to the observed sample estimate? [closed]

Let $\hat{\theta}$ be a biased estimator whose bias depends on the true value $\theta_0$, such that $E[\hat\theta|\theta_0]= f(\theta_0)\neq \theta_0$. Let $t_{sample}$ be a sample realization of $\...
Matifou's user avatar
  • 3,094
1 vote
1 answer
56 views

Is it possible to use poststratification when some observations have missing values on the variables used as strata?

This is a theoretical question, so I don't have data to share. Let's say I know the percentage of men and women in my population of interest, as well as the distribution of occupations and age ...
Cavdi's user avatar
  • 13
0 votes
1 answer
147 views

How to avoid bias/avoid overfitting when choosing a machine learning model? [closed]

My typical workflow in the past, when creating machine learning models, has been to do the following: Decide on some candidate model families for the task at hand. Divide dataset into train and test ...
user avatar
1 vote
0 answers
19 views

How can I take into account multiple individuals with multiple observations across a data?

I have a set of data as this: ID V1 V2 V3 A 12 10 8 A 11 9 10 B 7 10 8 C 13 10 9 C 10 12 6 This dataset is from health data, where each individual takes ...
Jorge A's user avatar
  • 97
0 votes
0 answers
19 views

Correcting for spillover bias in experiments?

Suppose I have an experiment designed where I'm interested in observing the effect of daily walking on self-reported joint pain in elderly adults. The population of interest are elderly adults who ...
jbuddy_13's user avatar
  • 3,382
3 votes
1 answer
287 views

How to estimate bias-corrected variance of a half-normal distribution?

Wikipedia says that for a given numbers $\{x_i\}_{i=1}^{n}$ drawn from a half-normal distribution, the variance of that distribution can be estimated by sample variance $\hat\sigma^2 = \frac{1}{n} \...
Andrey L.'s user avatar
  • 133
0 votes
2 answers
387 views

How to address 'immortal time bias' using R - equivalent to Stata stset?

We have a dataset on cancer patients who have consented to join a study after their diagnosis, which could be months or even years later. After some follow-up, an event occurs. We can fit this data ...
zx8754's user avatar
  • 270
1 vote
0 answers
26 views

How to account for bias in experiment data when quantifying treatment efficacy?

Say my randomization wasn't very effective, so I have two groups, each with 100 individuals and the difference in success rates is 3% before the treatment has ever been administered. I want to know ...
jbuddy_13's user avatar
  • 3,382
1 vote
0 answers
151 views

Implementing bias-adjustion for step3 latent profile analysis in R [closed]

I am identifying a latent profile model with the Mclust package in R. After identifying an optimal number of cluster I would like to identify possible covariates and distal outcomes via logistic/...
David Janda's user avatar
1 vote
1 answer
49 views

Suggestions on dealing with outliers when sample size is very small AND you must order the results

I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry. For each category we throw out the high and low scores and average the remaining ...
Omar Paloma's user avatar
0 votes
0 answers
39 views

Correction of biased probabilities

I have a classifier that outputs biased predictions, that I want to correct for in a mathematically sound way. I define bias as the difference between the true class distribution and the average ...
Jondiedoop's user avatar
3 votes
0 answers
65 views

Estimating the Absolute Difference in Mean Between Two Normal RVs

I've seen more general questions of this nature, but none that discuss specifically the problem setup that I'm interested in, which I believe is substantially simpler. Suppose that we observe two ...
Eric Weine's user avatar
2 votes
1 answer
60 views

How to correct for sampling bias in one population when comparing against another

I have two populations that I'd like to compare across certain metrics. However, most members of population A did not respond to our request for data, and those respondents that did are not ...
mdrishan's user avatar
  • 207
1 vote
0 answers
102 views

Python implementation of standard Expectation maximization algorithm to estimate position bias in LTR

Is there any Python implementation that would allow to use logs data of clicks and their ranking, and return and estimate of relative position bias as discussed in section ...
Tony's user avatar
  • 11
2 votes
2 answers
125 views

Correcting for pre-experiment bias in proportions test?

Say I have an obstacle course, which not everyone completes, though, globally, most do. I hypothesize that the treatment, drinking Gatorade, will cause an increase in the obstacle course completion ...
jbuddy_13's user avatar
  • 3,382
1 vote
0 answers
95 views

Standard deviation in generated data with specified autocorrelation - correction factor?

I want to generate time-series data with specified AR(1) and standard deviation. I am using arima.sim to generate univariate time series. (Ultimately, I want to use ...
TakTsun's user avatar
  • 11
0 votes
0 answers
33 views

Significance of parameter on cointegrating vector

I have been reading Section 6.2 (page 96) of this manual, where a procedure called "fully modified least squares" is discussed, and was developed on a paper of Phillips and Hansen (1990) . ...
Barreto's user avatar
  • 101
5 votes
2 answers
197 views

Finite population correction for the variance

Just when I thought I was starting to understand Bessel's correction, I noticed that it is not valid when the sample size equals the population size and so likely not valid for sample sizes close to ...
Zaz's user avatar
  • 263
1 vote
0 answers
69 views

Biased logistic regression in pytorch

My model has decently high AUC=90%, but is biased, underestimating the probability $y=1$. This is systematic across some of the input features as well. How can I nudge the bias term, or otherwise ...
user623949's user avatar
2 votes
0 answers
251 views

ADAM bias correction derivation

Where $\beta_2\in [0,1)$ and $g_i$ is the gradient at step i. We approximate E[gi^2] with E[gt^2] by a correction term $\zeta$. I think this is because it is assumed that the gradients are bounded. Is ...
Le Ploit's user avatar
2 votes
1 answer
92 views

Covid data analysis question

This might be a dumb question but I'm doing a basic data analysis for a medical group. Previously, I did a project for them where we looked at patient outcomes in a major hospital (let's call it "...
smoalem's user avatar
  • 23
0 votes
0 answers
149 views

How to bias-correct proportions for a prop.test in R?

Let's say I ask 200 participants in a study what experience brought the most joy to their lives: A, B, C, or D. ...
format_0's user avatar
4 votes
2 answers
661 views

Bias Correction and Variance Estimation of Inverse Log-Linear Regression

My question is related to this one, but it has no answers or comments. So, I try to ask with a different way. I have a physical quantity $X > 0$ and a noisy sensor $Y$ which has the following ...
obareey's user avatar
  • 111
1 vote
1 answer
179 views

Reasons to prefer low bias with higher variance over the alternative (and vice versa)

I am trying to understand the bias-variance tradeoff in practice. I have read several related questions and answers, but still have a few questions: Assume we are estimating a structural equation ...
user321797's user avatar
0 votes
1 answer
4k views

Linear Regression of non-normally distributed data [duplicate]

I am trying to understand the relationship between royalties received (independent variable) and health expenditures (dependent variable) for each municipality through a linear regression. My ...
Igor's user avatar
  • 5
1 vote
0 answers
103 views

How to combine stratified sampling and avoid common driver in both training & test set - R Cross-Validation

I want to build a random forest or gradient boosting model on very heterogeneous data using cross-validation in R. Due to data heterogeneity and dependency between the observation (the same subject ID ...
John E.'s user avatar
  • 21
2 votes
2 answers
327 views

Multivariate sampling bias correction for data analysis

I have a dataset of properties for 20,000 samples and 200 of these samples have a categorical label (A, B or C); and I'm trying to assess the relationships between the properties and the labels. I am ...
A. Bollans's user avatar
1 vote
0 answers
54 views

How can I understand ATT estimator for matching discrepancies in Causal Inference Mixtape?

While studying 'Causal Inference: Mixtape' by myself, something I didn't know happened. link: https://mixtape.scunning.com/matching-and-subclassification.html#bias-correction $\begin{align} \...
vinsh_77's user avatar
2 votes
0 answers
75 views

How to average across multiple regressions, correct for p-value and estimates?

In this data frame I have 2 replicates that were placed in low, mid, or high areas with recorded densities at x and y. Within a certain level e.g. rep 1 "mid" has x = 24 and y = NAs. ...
slicer's user avatar
  • 613
0 votes
1 answer
314 views

ML Model shows systematic bias in predictions. Can or should this be corrected and how?

I am trying to build a machine learning model using Microsoft ML.NET to predict road surfacing life. I have a set of observed road surfacing lives with associated data such as traffic counts, number ...
Fritz45's user avatar
  • 221
4 votes
1 answer
823 views

Dealing with pre-test bias by repeated random sampling

I'm dealing with zero inflated data that has extreme volatility. I wish to randomly assign X observations/participants/subjects from the data, treat half of them with a new treatment ("target&...
Matan Shaked's user avatar
1 vote
0 answers
171 views

How to avoid underestimation in neural network model when loss targets contain min something positive and model output itself?

Apologies if this has been asked already, but I couldn't find anything about it. Straight to the problem: I want to train a neural network for regression and the target in the loss function is a min ...
Jane Aminato's user avatar
3 votes
1 answer
526 views

log-transformation bias correction "in reverse" when creating simulated dataset from regression predictions

I am developing 1000 simulation datasets where I add random noise to predictions from a regression model where the y-variable has been log-transformed to account for non-normality. I have notice that ...
ken's user avatar
  • 213
0 votes
1 answer
246 views

Finding UMVUE of function of poisson parameter

I am to estimate $\exp(-\lambda)\lambda^2/2$ from the distribution $Exp(\lambda) \sim \frac{e^{-\lambda}\lambda^x}{x!}$ I used the indicator function $W=\mathbb I_{2}(X_1)$ as an initial unbiased ...
smaillis's user avatar
  • 133
0 votes
0 answers
20 views

First difference in logs transformation produces biased results on back-transformation [duplicate]

I have a strongly trended series where the trend appears to be exponential and I believe the errors tend to be proportional to the current value. In order to convert it to a stationary series for ...
andrewH's user avatar
  • 3,157
3 votes
1 answer
2k views

Should one apply bias correction for the standard deviation, for small sample sizes, as a matter of course?

If one is dealing with small sample sizes, let's say $8-16$ observations per sample, and we are interested in estimates of the standard-deviation (let us also assume Gaussian statistics), is there a ...
user27119's user avatar
  • 328
2 votes
1 answer
53 views

How to decide if to use weights or not when estimating some $\mu$ of a population that has sub-populations with different $\mu_i$?

Setting and Notation Let's assume we have a population with (for example) two sub-population. Say, males and females. In the population they are split 50%-50%. We care about the population level ...
Tal Galili's user avatar
  • 21.8k
0 votes
1 answer
79 views

Reducing bias for estimate of population standard deviation

https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation#Other_distributions "For non-normal distributions an approximate (up to O(n−1) terms) formula for the unbiased estimator of ...
Mathstudent's user avatar
0 votes
0 answers
28 views

Can I correct a bias, or “skew” in my data using linear regression? [duplicate]

My supervisor has suggested that I correct a bias, or "skew" in my data using linear regression. Apparently the line of fit in the plot below should be horizontal (example attachted) Its a ...
user avatar
0 votes
1 answer
396 views

Nested CV vs CV with a holdout?

I just learned of nested cross-validation and wanted to understand how my current approach is worse/ok. Currently I would: Divide the data into a train/test set (80/20ish). Use k-fold cross-...
Josh's user avatar
  • 308
6 votes
3 answers
194 views

Sample standard deviation is a biased estimator: Details in calculating the bias of $s$

In this post Why is sample standard deviation a biased estimator of $\sigma$? the last step is shown as: $$\sigma\left(1-\sqrt\frac{2}{n-1}\frac{\Gamma\frac{n}{2}}{\Gamma\frac{n-1}{2}}\right) = \sigma\...
Darya's user avatar
  • 75
0 votes
1 answer
282 views

why is standard deviation a biased estimator [duplicate]

In this post: Why is sample standard deviation a biased estimator of $\sigma$?, I am having difficulty understanding some of the steps. We have : (a) $s^2=\frac{1}{n-1}\sum_{i=1}^{\infty}(x_i-x\bar)^2$...
Darya's user avatar
  • 75
1 vote
1 answer
413 views

Focal Predictions from a linear model: How to test for difference between factor levels (pairwise) instead of comparing errorbars?

Sorry for the cluelessness, I know this is a topic that arises often in different variations. Still, I couldn't find an answer for my situation. In my work, I sample people and try to generalize the ...
Emman's user avatar
  • 207
0 votes
1 answer
31 views

My Design defines the Hypothesis test?

a conceptual question appear in my mind and i need help, in pratical therms, whats defines the statistical test its my hypothesis and my experimental design ? Using this example, if i am interrested ...
Leanderson Silva's user avatar
11 votes
2 answers
2k views

Kernel density estimation and boundary bias

What sort of kernel density estimator does one use to avoid boundary bias? Consider the task of estimating the density $f_0(x)$ with bounded support and where the probability mass is not decreasing ...
Jesper for President's user avatar
4 votes
0 answers
213 views

Bias correcting penalized maximum likelihood / maximum a posteriori estimates

Suppose an estimator $\hat\theta_T$ is defined as the value of $\theta$ maximizing: $$\sum_{t=1}^T{l(y_t|\theta)}+\mu_T g(\theta),$$ where $l(y_t|\theta)$ is the log-likelihood of observation $t$, $\...
cfp's user avatar
  • 535