# How to measure uncertainty of a parameter when false positives exist?

The main goal of my research is to measure the percentage of brown dwarf stars in the Pleiades star forming cluster that are actually double stars (i.e. the brown dwarf star has a companion brown dwarf orbiting it).

To do this, I need to estimate what is the faintest brown dwarf star detectable in our observations using the Hubble Space Telescope Wide Field Camera 3.

My question is this:

Say you measure a parameter $X$ 100 times ($X_1, X_2,...,X_{100}$). There are only true positive detections and false positive detections.

1. 80 of those measurements have a 99% chance of being a true positive.

2. 20 measurements have a 50% chance of being a true positive (50% chance of a false positive).

If there was no false positives, the uncertainty $\sigma_X$ = standard deviation of all measurements.

How do you measure $\sigma_X$ when false positives exist as shown above?

To elaborate: each measurement of parameter $(X_1,X_2,...,X_{100})$ has an assigned chance of being a false positive $(P_{f1},P_{f2},...,P_{f100})$. So how to measure $\sigma_X$ is complicated by the existence of these false positives.

• Just to elaborate: each measurement of parameter ($X_1$,$X_2$,...,$X_{100}$) has an assigned chance of being a false positive ($P_{f1}$,$P_{f2}$,...,$P_{f100}$). So how to measure $\sigma_X$ is complicated by the existence of these false positives. – Eugenio Victor Garcia Feb 20 '14 at 20:38

In principle this is a classification problem. If you would know which observation is a true positive, you could just take these observations and estimate the mean and the variance for them. Doing so, you implicitly assume that the true value follows a normal (or more accurately T-student) distribution defined by the obtained mean $\overline{x}$ and variance $s$:

$$\mu\;\tilde{}\;N\left(\overline{x},s\right)$$

Because you are not certain about which observations are true positives, there are various scenarios which need consideration. If you would construct 100 scenarios then in 80 of them you would include an observation which you give 80% to be a true positive. The probability model gets more complicated:

$$p(\mu) = p\left(\mu\ |\, \text{subset of the }X_i\right)\times p\left(\text{subset of the }X_i\right)$$ where the first factor on the right hand side denotes the probability for a certain value $\mu$ to be the true value given the subset of observations. The second factor denotes the probability that a certain subset of the observations contains all the true positives and no false positives.

So how to get the standard deviation? You could write a computer program that samples values for $\mu$ from the above stated probability distribution. If you afterwards have a list of $\mu_1,\mu_2,\dots,\mu_N$, you can calculate mean and variance in the usual way: $$\mu = \frac{1}{N}\sum_{i=1}^N \mu_i \hspace{1em}\mbox{and}\hspace{1em} \sigma^2 = \frac{1}{N}\sum_{i=1}^{N}(\mu_i-\mu)^2$$ Additionally, just to make sure that the variance (or standard deviation) is a meaningful measure for the uncertainty, you could draw a histogram for the $\mu_1,\mu_2,\dots$ and compare it to a normal distribution characterized by $\mu$ and $\sigma$.

In order to get the $\mu_i$'s you can perform the sampling in two steps. Firstly, sample a subset of the $X_i$. The following pseudo code demonstrates how this could be done:

for each observation X_i do
P := probability for X_i to be a true positive
R := random number in [0,1] drawn from uniform distribution
if R smaller P
accept X_i as true positive
otherwise
reject X_i for the scenario


If you work with statistical programming language $R$ you could also simply use the sample function.

For the sampled subset of the $X_i$'s calculate a mean $\overline{x}$ and variance $s$ in the usual way. Use that to draw a sample data point $\mu_i$ from the normal (better T-student) distribution characterized by mean $\overline{x}$ and variance $s$.

Repeat these two steps until you have enough data points $\mu_i$ to get convergence for $\mu$ and $\sigma$.

• There is pretty clear systematic error in the $X_i$'s since (long story short) the point spread function models for HST Wide Field Camera 3 are not perfect. This may imply that the distribution is likely not a student's T. I do know this: If the sample is all false positives, then the distribution is clearly different from a distribution with false positives and true positives as we are dealing with here. Howabout doing the accept/reject as suggested to sample the $X_i$'s and then compute $\overline{x}$ and variance $s$ from it? Is knowledge of the underlying distribution required? – Eugenio Victor Garcia Feb 21 '14 at 21:59
• Also - I am entirely serious - if this method works I am more than willing to mention you by name in the acknowledgement section of resulting refereed publication :) – Eugenio Victor Garcia Feb 21 '14 at 22:03