Optimal blind poll construction

Question

This is a question strongly related to Cauchy "characters".

I'm constructing a 4 question canvassing questionnaire that will tell the likely voter being contacted which of the presidential candidates most closely matches them. The advantage of this approach for a dark horse presidential candidate is obvious, presuming, of course, that most of the likely voters match him. I do have the verbal interest in this from the state-level executive director for such a presidential candidate.

One might be entitled to think that this work has been done umpteen times by the thousands of political science PhDs and/or major polling organizations -- at least using the General Social Survey data if nothing else -- and one would be wrong. Moreover, we don't have much time to deploy.

Ideally the questionnaire construction would result in a kind of decision tree where the door-to-door canvassing volunteer could have a mobile device app providing the next question to ask based on the answers to prior questions.

Also ideally, the construction process, itself, would minimize the likely-voter contact as this drives the expense. Using GSS data zeros out that cost and would be optimal if we could get access to the raw GSS data, but we can't. We have to do a survey to gather the data for construction of the 4-deep tree of questions.

On the result side, as a practical compromise, I've proposed falling back to finding just 4 questions rather than finding a 4-deep tree of questions.

On the construction side, as a practical compromise, I've proposed a prize-fund backing a tournament where:

1. The contestants each submit 4 questions.
2. The submitted questionnaires are paired up for the contests.
3. Campaign volunteers each get a pair of contested questionnaires, and get ten likely voters to completely answer all 8 questions.
4. The winning questionnaire of a contest is the one whose author can, from the answers to his own questionnaire, best-guess the answers to the opponent's questionnaire.
5. Award prizes after the log2(N) contests have selected a winner of that tournament.
6. Publish the rankings of the questionnaires and, by permission, their authors.

As resources permit, this tournament is iterated for multiple rounds.

We really have to place weight on the value of up-front volunteer time, so minimizing the construction labor is crucial.

I know this is fairly far from a pure mathematics question but I've brought it as close as I can to some kind of weighted figure of merit involving high-value labor in the construction phase and the expected accuracy of the resulting 4 questions answered by likely voters contacted during blind poll canvassing by lower value labor.

The question: About how far from optimal would be the proposed practical compromise of the 4-question questionnaire (constructed as described) from the ideal 4-deep decision tree of questions constructed from an infinite number of samples during the construction phase?

A secondary question: Is there a better way to make use of the same up-front volunteer time?

Answer 1

First: Why can't you get the raw data from the GSS? It's easily available. Fail that, you can work with ANES or with the US sample of the World Value Survey. Or raw exit poll data. If you need academic access to get the files, contact me.

Second: The poly-sci way to do this is to run the Ideal or OC to construct a d-dimensional "Issue-space", figure out where the candidates are in issue space(Pretty easy, either by interpreting item parameters of the "Supports Candidate X" question or just by looking at the coordinates of candidate supporters, and then find which 4 questions are maximally informative with regards to issue space.

I actually just finished working through a similar problem.

Answer 2

EDIT in response to last comments.

Here is my suggestion for how to run the contest.

1. The contest holder should decide on a list of "test questions". The 4-item questionnaires will be scored on how well they allow the guesser to guess the voter's responses to these "test questions". These test questions will be made public, and there will be a call for submissions for 4-test questionnaires. There will also be a call for participants to compete in the "guessing" contest. No participant is allowed to compete in both contests.
2. The contest holder decides on a list of the (e.g.) 10 most promising questionnaires.
3. The questionnaires are randomly assigned to volunteers. The volunteers interview potential voters.
4. The survey that a voter completes consists of (i.) the complete list of "test questions" (ii.) plus one of the competing 4-item questionnaires.
5. The completed surveys are assembled. A test is created for the guessers to complete. Each test question corresponds to an survey that a voter completed. The test question gives the voter's responses to the 4-item questionnaire. The guessers attempt to guess the voter's responses to the "test questions" based on that information.
6. Compute a "guesser score" based on how well each guesser did overall, and compute a "questionnaire score" by taking a weighted average for each questionnaire weighted by the guesser score.