Return to Question

I have a collection of $N$ different objects. An expert "looks" at the object $O_i$ and produces a prediction: binary label $y_i\in\{0,1\}$. The predicted binary label $y_i$ is observed (data) and is known to be noisy.

Now assume that there are $K$ different experts with a consensus on what it means to label "1" or "0". The $k$th expert produces an output label $y^k_i$ (data) for object $O_i$. Therefore, each object has $K$ observations.

Additionally, each object has observed real-valued features vector $x_i$ (data). Note that the feature vectors are not used by experts; features are only noisy observations that are believed to be useful for prediction given no experts!

My idea is to assign each expert a random variable $r^k$ such that it denotes the probability of flipping the noise-free/hidden $\hat y_i$.

The main goal is to estimate noise-free/hidden $\hat y_i$ label for each object. It is expected that feature vectors and multiple experts are complimentary and helpful to discover the hidden latent labels.

For example, there are 5 objects with hidden true labels: [1 0 0 1 1]. An expert's intrinsic (hidden) flipping probability is $r=0.1$. This means that the observed outputs for this experts is a vector of length 5 where each label is flipped independently with a probability of 10%.

It is unclear to me how to build a model that attempts to:

1. uncover hidden labels $\hat y_i$
2. infer the probabilities $r^k$

I started by fitting a logistic regression model but realized that I am fitting different quantity.

My package of choice is PyMC3 but I am open to other probabilistic programming packages such as Stan, JAGS.

Toy (made up) data:

Suppose there are 3 objects and 4 experts.

The feature vectors are given (4 dimensions):

$x_1 = [0.3, 0.4, 0.01, 0.9]$

$x_2 = [0.1, 0.9, 0.05, 0.6]$

$x_3 = [0.9, 0.1, 0.15, 0.3]$

One can train a binary classifier that takes object vectors and predicts binary label. The problem is that the feature vectors are known to contain fairly little information about the prediction task, thus the hope on crowd of experts.

The experts (independently) make their label predictions on the three objects (observed):

Expert 1: $y^1 = [1,1,1]$

Expert 2: $y^2 = [0,1,0]$

Expert 3: $y^3 = [1,1,0]$

Expert 4: $y^4 = [1,0,0]$

One can simply use majority vote to approximate the true label. The problem is that the expert predictions are noisy and experts have different expertise.

The data are object feature vectors $x_i$ and expert label vectors $y^k$.

The hidden/latent factors are: noise-free object labels $\hat y$ and expert flipping probabilities $r^k$.

I have a collection of $N$ different objects. An expert "looks" at the object $O_i$ and produces a prediction: binary label $y_i\in\{0,1\}$. The predicted binary label $y_i$ is observed (data) and is known to be noisy. Also, objects are of different complexity: an expert with poor "skill" will likely mislabel a work item.

Now assume that there are $K$ experts (each with a different "skill") with a consensus on what it means to label "1" or "0". The $k$th expert produces an output label $y^k_i$ (data) for object $O_i$. Therefore, each object has $K$ observations.

Additionally, each object has observed real-valued features vector $x_i$ (data). Note that the feature vectors are not used by experts; features are only noisy observations that are believed to be useful for prediction given no experts!

The main goal is to estimate noise-free/hidden $\hat y_i$ label for each object. It is expected that feature vectors and multiple experts are complimentary and helpful to discover the hidden latent labels.

For example, there are 5 objects with hidden true labels: [1 0 0 1 1]. An expert's intrinsic (hidden) flipping probability is $r=0.1$. This means that the observed outputs for this experts is a vector of length 5 where each label is flipped independently with a probability of 10%.

It is unclear to me how to build a model that attempts to:

1. uncover hidden labels $\hat y_i$
2. infer the $k$th worker "skill" (flipping probability?)

I started by fitting a logistic regression model but realized that I am fitting different quantity.

My package of choice is PyMC3 but I am open to other probabilistic programming packages such as Stan, JAGS.

Toy (made up) data:

Suppose there are 3 objects and 4 experts.

The feature vectors are given (4 dimensions):

$x_1 = [0.3, 0.4, 0.01, 0.9]$

$x_2 = [0.1, 0.9, 0.05, 0.6]$

$x_3 = [0.9, 0.1, 0.15, 0.3]$

One can train a binary classifier that takes object vectors and predicts binary label. The problem is that the feature vectors are known to contain fairly little information about the prediction task, thus the hope on crowd of experts.

The experts (independently) make their label predictions on the three objects (observed):

Expert 1: $y^1 = [1,1,1]$

Expert 2: $y^2 = [0,1,0]$

Expert 3: $y^3 = [1,1,0]$

Expert 4: $y^4 = [1,0,0]$

One can simply use majority vote to approximate the true label. The problem is that the expert predictions are noisy and experts have different expertise.

The data are object feature vectors $x_i$ and expert label vectors $y^k$.

The hidden/latent factors are: noise-free object labels $\hat y$ and expert flipping probabilities $r^k$.

I have a collection of $N$ different objects. An expert "looks" at the object $O_i$ and produces a prediction: binary label $y_i\in\{0,1\}$. The predicted binary label $y_i$ is observed (data) and is known to be noisy.

Now assume that there are $K$ different experts with a consensus on what it means to label "1" or "0". The $k$th expert produces an output label $y^k_i$ (data) for object $O_i$. Therefore, each object has $K$ observations.

Additionally, each object has observed real-valued features vector $x_i$ (data). Note that the feature vectors are not used by experts; features are only noisy observations that are believed to be useful for prediction given no experts!

My idea is to assign each expert a random variable $r^k$ such that it denotes the probability of flipping the noise-free/hidden $\hat y_i$.

The main goal is to estimate noise-free/hidden $\hat y_i$ label for each object. It is expected that feature vectors and multiple experts are complimentary and helpful to discover the hidden latent labels.

For example, there are 5 objects with hidden true labels: [1 0 0 1 1]. An expert's intrinsic (hidden) flipping probability is $r=0.1$. This means that the observed outputs for this experts is a vector of length 5 where each label is flipped independently with a probability of 10%.

It is unclear to me how to build a model that attempts to:

1. uncover hidden labels $\hat y_i$
2. infer the probabilities $r^k$

I started by fitting a logistic regression model but realized that I am fitting different quantity.

My package of choice is PyMC3 but I am open to other probabilistic programming packages such as Stan, JAGS.

Toy (made up) data:

Suppose there are 3 objects and 4 experts.

The feature vectors are given (4 dimensions):

$x_1 = [0.3, 0.4, 0.01, 0.9]$

$x_2 = [0.1, 0.9, 0.05, 0.6]$

$x_3 = [0.9, 0.1, 0.15, 0.3]$

One can train a binary classifier that takes object vectors and predicts binary label. The problem is that the feature vectors are known to contain fairly little information about the prediction task, thus the hope on crowd of experts.

The experts (independently) make their label predictions on the three objects:

Expert 1: $y^1 = [1,1,1]$

Expert 2: $y^2 = [0,1,0]$

Expert 3: $y^3 = [1,1,0]$

Expert 4: $y^4 = [1,0,0]$

One can simply use majority vote to approximate the true label. The problem is that the expert predictions are noisy and experts have different expertise.

The data are object feature vectors $x_i$ and expert label vectors $y^k$.

The hidden/latent factors are: noise-free object labels $\hat y$ and expert flipping probabilities $r^k$.

I have a collection of $N$ different objects. An expert "looks" at the object $O_i$ and produces a prediction: binary label $y_i\in\{0,1\}$. The predicted binary label $y_i$ is observed (data) and is known to be noisy.

Now assume that there are $K$ different experts with a consensus on what it means to label "1" or "0". The $k$th expert produces an output label $y^k_i$ (data) for object $O_i$. Therefore, each object has $K$ observations.

Additionally, each object has observed real-valued features vector $x_i$ (data). Note that the feature vectors are not used by experts; features are only noisy observations that are believed to be useful for prediction given no experts!

My idea is to assign each expert a random variable $r^k$ such that it denotes the probability of flipping the noise-free/hidden $\hat y_i$.

The main goal is to estimate noise-free/hidden $\hat y_i$ label for each object. It is expected that feature vectors and multiple experts are complimentary and helpful to discover the hidden latent labels.

For example, there are 5 objects with hidden true labels: [1 0 0 1 1]. An expert's intrinsic (hidden) flipping probability is $r=0.1$. This means that the observed outputs for this experts is a vector of length 5 where each label is flipped independently with a probability of 10%.

It is unclear to me how to build a model that attempts to:

1. uncover hidden labels $\hat y_i$
2. infer the probabilities $r^k$

I started by fitting a logistic regression model but realized that I am fitting different quantity.

My package of choice is PyMC3 but I am open to other probabilistic programming packages such as Stan, JAGS.

Toy (made up) data:

Suppose there are 3 objects and 4 experts.

The feature vectors are given (4 dimensions):

$x_1 = [0.3, 0.4, 0.01, 0.9]$

$x_2 = [0.1, 0.9, 0.05, 0.6]$

$x_3 = [0.9, 0.1, 0.15, 0.3]$

One can train a binary classifier that takes object vectors and predicts binary label. The problem is that the feature vectors are known to contain fairly little information about the prediction task, thus the hope on crowd of experts.

The experts (independently) make their label predictions on the three objects (observed):

Expert 1: $y^1 = [1,1,1]$

Expert 2: $y^2 = [0,1,0]$

Expert 3: $y^3 = [1,1,0]$

Expert 4: $y^4 = [1,0,0]$

One can simply use majority vote to approximate the true label. The problem is that the expert predictions are noisy and experts have different expertise.

The data are object feature vectors $x_i$ and expert label vectors $y^k$.

The hidden/latent factors are: noise-free object labels $\hat y$ and expert flipping probabilities $r^k$.

I have a collection of $N$ different objects. An expert "looks" at the object $O_i$ and produces a prediction: binary label $y_i\in\{0,1\}$. The predicted binary label $y_i$ is observed (data) and is known to be noisy.

Now assume that there are $K$ different experts with a consensus on what it means to label "1" or "0". The $k$th expert produces an output label $y^k_i$ (data) for object $O_i$. Therefore, each object has $K$ observations.

Additionally, each object has observed real-valued features vector $x_i$ (data). Note that the feature vectors are not used by experts; features are only noisy observations that are believed to be useful for prediction given no experts!

My idea is to assign each expert a random variable $r^k$ such that it denotes the probability of flipping the noise-free/hidden $\hat y_i$.

The main goal is to estimate noise-free/hidden $\hat y_i$ label for each object. It is expected that feature vectors and multiple experts are complimentary and helpful to discover the hidden latent labels.

For example, there are 5 objects with hidden true labels: [1 0 0 1 1]. An expert's intrinsic (hidden) flipping probability is $r=0.1$. This means that the observed outputs for this experts is a vector of length 5 where each label is flipped independently with a probability of 10%.

It is unclear to me how to build a model that attempts to:

1. uncover hidden labels $\hat y_i$
2. infer the probabilities $r^k$

I started by fitting a logistic regression model but realized that I am fitting different quantity.

My package of choice is PyMC3 but I am open to other probabilistic programming packages such as Stan, JAGS.

Toy (made up) data:

Suppose there are 3 objects and 4 experts.

The feature vectors are given (4 dimensions):

$x_1 = [0.3, 0.4, 0.01, 0.9]$

$x_2 = [0.1, 0.9, 0.05, 0.6]$

$x_3 = [0.9, 0.1, 0.15, 0.3]$

The experts make their label predictions on the three objects:

Expert 1: $y^1 = [1,1,1]$

Expert 2: $y^2 = [0,1,0]$

Expert 3: $y^3 = [1,1,0]$

Expert 4: $y^4 = [1,0,0]$

The data are object feature vectors $x_i$ and expert label vectors $y^k$.

The hidden/latent factors are: noise-free object labels $\hat y$ and expert flipping probabilities $r^k$.

I have a collection of $N$ different objects. An expert "looks" at the object $O_i$ and produces a prediction: binary label $y_i\in\{0,1\}$. The predicted binary label $y_i$ is observed (data) and is known to be noisy.

Now assume that there are $K$ different experts with a consensus on what it means to label "1" or "0". The $k$th expert produces an output label $y^k_i$ (data) for object $O_i$. Therefore, each object has $K$ observations.

Additionally, each object has observed real-valued features vector $x_i$ (data). Note that the feature vectors are not used by experts; features are only noisy observations that are believed to be useful for prediction given no experts!

My idea is to assign each expert a random variable $r^k$ such that it denotes the probability of flipping the noise-free/hidden $\hat y_i$.

The main goal is to estimate noise-free/hidden $\hat y_i$ label for each object. It is expected that feature vectors and multiple experts are complimentary and helpful to discover the hidden latent labels.

For example, there are 5 objects with hidden true labels: [1 0 0 1 1]. An expert's intrinsic (hidden) flipping probability is $r=0.1$. This means that the observed outputs for this experts is a vector of length 5 where each label is flipped independently with a probability of 10%.

It is unclear to me how to build a model that attempts to:

1. uncover hidden labels $\hat y_i$
2. infer the probabilities $r^k$

I started by fitting a logistic regression model but realized that I am fitting different quantity.

My package of choice is PyMC3 but I am open to other probabilistic programming packages such as Stan, JAGS.

Toy (made up) data:

Suppose there are 3 objects and 4 experts.

The feature vectors are given (4 dimensions):

$x_1 = [0.3, 0.4, 0.01, 0.9]$

$x_2 = [0.1, 0.9, 0.05, 0.6]$

$x_3 = [0.9, 0.1, 0.15, 0.3]$

One can train a binary classifier that takes object vectors and predicts binary label. The problem is that the feature vectors are known to contain fairly little information about the prediction task, thus the hope on crowd of experts.

The experts (independently) make their label predictions on the three objects:

Expert 1: $y^1 = [1,1,1]$

Expert 2: $y^2 = [0,1,0]$

Expert 3: $y^3 = [1,1,0]$

Expert 4: $y^4 = [1,0,0]$

One can simply use majority vote to approximate the true label. The problem is that the expert predictions are noisy and experts have different expertise.

The data are object feature vectors $x_i$ and expert label vectors $y^k$.

The hidden/latent factors are: noise-free object labels $\hat y$ and expert flipping probabilities $r^k$.