0
$\begingroup$

The TL;DR of the dataset is this:

The data set I'm working with is a set of votes. Experts in a field vote for outcome A or outcome B and I'm running a logistic regression to get a feel for how the intensity of the majority vote relates to the actual probability of the outcome.

For example, if in Case 1, 51% of experts think that a given outcome is more likely and in Case 2, 86% of experts think a given outcome is more likely, I would expect the real probability of the outcome in Case 2 to be higher than Case 1.

In this dataset I also have the historical accuracy of each expert. I think it's fair to say that if a given expert has been say 80% accurate over a given time frame and another expert has been say 60% accurate over that same time frame then the higher performing expert deserves more voting power.

When I perform my logistic regression, can I split my group of voters into three performance tiers, say average (>.50 - .66), good (>.66 - .82), and great (>.82), and use the votes majority in those tiers as the features for my logistic regression? The idea being that my solver would then optimize the coefficients for each group of experts and effectively, if not literally, find the optimal relative vote weight for each performance group.

For example the solver may find that the average group receives a coefficient of 0.2, the good group receives one of 0.4, and the great group receives one of 0.8. That to me says the expert picker's vote is worth 4 times that of an average picker.

I'm pretty sure that I can use features in this way and I'm pretty sure that it will produce logical weights based on performance - but I've been pretty sure and wrong before. What, if any, issues might I have with this approach?

If it helps I threw together a simplified data example.

+--------------------------------------------------------------+
|              Original Single Variable Approach               |
+-----------------------------------------+--------------------+
| Historical Accuracy                     | Says X will happen |
+-----------------------------------------+--------------------+
| 0.51                                    | Outcome B          |
+-----------------------------------------+--------------------+
| 0.53                                    | Outcome B          |
+-----------------------------------------+--------------------+
| 0.61                                    | Outcome B          |
+-----------------------------------------+--------------------+
| 0.67                                    | Outcome B          |
+-----------------------------------------+--------------------+
| 0.69                                    | Outcome B          |
+-----------------------------------------+--------------------+
| 0.81                                    | Outcome B          |
+-----------------------------------------+--------------------+
| 0.83                                    | Outcome A          |
+-----------------------------------------+--------------------+
| 0.84                                    | Outcome A          |
+-----------------------------------------+--------------------+
| 0.85                                    | Outcome A          |
+-----------------------------------------+--------------------+
|                                         |                    |
+-----------------------------------------+--------------------+
| Produces a single variable instance of: | .66 for Outcome A  |
+-----------------------------------------+--------------------+
+-----------------------------------------------------------------------+
|                       Three   Variable Approach                       |
+---------+----------------------------------------+--------------------+
| Group   | Historical Accuracy                    | Says X will happen |
+---------+----------------------------------------+--------------------+
| Average | 0.51                                   | Outcome B          |
+---------+----------------------------------------+--------------------+
| Average | 0.53                                   | Outcome B          |
+---------+----------------------------------------+--------------------+
| Average | 0.61                                   | Outcome B          |
+---------+----------------------------------------+--------------------+
| Great   | 0.67                                   | Outcome B          |
+---------+----------------------------------------+--------------------+
| Great   | 0.69                                   | Outcome B          |
+---------+----------------------------------------+--------------------+
| Great   | 0.81                                   | Outcome B          |
+---------+----------------------------------------+--------------------+
| Good    | 0.83                                   | Outcome A          |
+---------+----------------------------------------+--------------------+
| Good    | 0.84                                   | Outcome A          |
+---------+----------------------------------------+--------------------+
| Good    | 0.85                                   | Outcome A          |
+---------+----------------------------------------+--------------------+
|         |                                        |                    |
+---------+----------------------------------------+--------------------+
|         | Produces a three variable instance of: | 1 for Outcome B    |
+---------+----------------------------------------+--------------------+
|         |                                        | 1 for Outcome B    |
+---------+----------------------------------------+--------------------+
|         |                                        | 1 for Outcome A    |
+---------+----------------------------------------+--------------------+
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.