1
$\begingroup$

This is a question strongly related to Cauchy "characters".

I'm constructing a 4 question canvassing questionnaire that will tell the likely voter being contacted which of the presidential candidates most closely matches them. The advantage of this approach for a dark horse presidential candidate is obvious, presuming, of course, that most of the likely voters match him. I do have the verbal interest in this from the state-level executive director for such a presidential candidate.

One might be entitled to think that this work has been done umpteen times by the thousands of political science PhDs and/or major polling organizations -- at least using the General Social Survey data if nothing else -- and one would be wrong. Moreover, we don't have much time to deploy.

Ideally the questionnaire construction would result in a kind of decision tree where the door-to-door canvassing volunteer could have a mobile device app providing the next question to ask based on the answers to prior questions.

Also ideally, the construction process, itself, would minimize the likely-voter contact as this drives the expense. Using GSS data zeros out that cost and would be optimal if we could get access to the raw GSS data, but we can't. We have to do a survey to gather the data for construction of the 4-deep tree of questions.

On the result side, as a practical compromise, I've proposed falling back to finding just 4 questions rather than finding a 4-deep tree of questions.

On the construction side, as a practical compromise, I've proposed a prize-fund backing a tournament where:

  1. The contestants each submit 4 questions.
  2. The submitted questionnaires are paired up for the contests.
  3. Campaign volunteers each get a pair of contested questionnaires, and get ten likely voters to completely answer all 8 questions.
  4. The winning questionnaire of a contest is the one whose author can, from the answers to his own questionnaire, best-guess the answers to the opponent's questionnaire.
  5. Award prizes after the log2(N) contests have selected a winner of that tournament.
  6. Publish the rankings of the questionnaires and, by permission, their authors.

As resources permit, this tournament is iterated for multiple rounds.

We really have to place weight on the value of up-front volunteer time, so minimizing the construction labor is crucial.

I know this is fairly far from a pure mathematics question but I've brought it as close as I can to some kind of weighted figure of merit involving high-value labor in the construction phase and the expected accuracy of the resulting 4 questions answered by likely voters contacted during blind poll canvassing by lower value labor.

The question: About how far from optimal would be the proposed practical compromise of the 4-question questionnaire (constructed as described) from the ideal 4-deep decision tree of questions constructed from an infinite number of samples during the construction phase?

A secondary question: Is there a better way to make use of the same up-front volunteer time?

$\endgroup$
3
  • $\begingroup$ I read the whole post, but I can't work out what your question is. $\endgroup$ May 25 '11 at 15:08
  • $\begingroup$ I've tried adding clarifying wrap-up at the end. Does it help you? $\endgroup$
    – Anonymous
    May 25 '11 at 15:37
  • 1
    $\begingroup$ I've discovered the appropriate term is "Information Gain": en.wikipedia.org/wiki/Information_gain_in_decision_trees $\endgroup$ Jul 31 '11 at 18:39
1
$\begingroup$

EDIT in response to last comments.

Here is my suggestion for how to run the contest.

  1. The contest holder should decide on a list of "test questions". The 4-item questionnaires will be scored on how well they allow the guesser to guess the voter's responses to these "test questions". These test questions will be made public, and there will be a call for submissions for 4-test questionnaires. There will also be a call for participants to compete in the "guessing" contest. No participant is allowed to compete in both contests.
  2. The contest holder decides on a list of the (e.g.) 10 most promising questionnaires.
  3. The questionnaires are randomly assigned to volunteers. The volunteers interview potential voters.
  4. The survey that a voter completes consists of (i.) the complete list of "test questions" (ii.) plus one of the competing 4-item questionnaires.
  5. The completed surveys are assembled. A test is created for the guessers to complete. Each test question corresponds to an survey that a voter completed. The test question gives the voter's responses to the 4-item questionnaire. The guessers attempt to guess the voter's responses to the "test questions" based on that information.
  6. Compute a "guesser score" based on how well each guesser did overall, and compute a "questionnaire score" by taking a weighted average for each questionnaire weighted by the guesser score.
$\endgroup$
7
  • $\begingroup$ Answer to a deleted comment: If you have 4 mutually independent questions, each with a 50/50 split in the population, that is the best you can do, and it will be at least as good as any 4-deep question tree (with binary responses.) This is actually mathematically trivial since any decision tree can be encoded as a (complicated) equivalent 4-item questionnaire. Example: Question 1: Do you support A? T/F. Question 2: (Do you support B AND did you answer T in question 1) OR (Do you support C AND did you answer F in question 1) etc. $\endgroup$ May 25 '11 at 18:50
  • $\begingroup$ So if I understand your suggestion, you would give the survey volunteers a random assortment of paired questionnaires to run past likely voters. Let's say 10 random pairs each. Then for each 8-question pairing answered, an open field of guessers would be presented with answers to one of the pair of questionnaires to see how well they could guess the answers to the other questionnaire. $\endgroup$ May 25 '11 at 18:59
  • $\begingroup$ Nope, I suggest scrap the pairing. Say you have 10 competing questionnaires. Collect 10 interviews for each questionnaire (each respondent only answers one questionnaire). Then give each guesser a private randomized test in which they fill in missing responses for each of the 100 interviews collected. This gives you more data to work with. $\endgroup$ May 25 '11 at 19:12
  • $\begingroup$ The surveyed voter would have to be presented with more than just the 4 questions from the one questionnaire otherwise there would be no data that could be missing from that presented to the guesser. So let's say that rather than a pair of questionnaires the voter is presented with 8 questions, 4 of which are randomly selected from the population of questions in all the questionnaires, and the other 4 are from a single questionnaire. The guessers would then have to guess the answers to the 4 randomly drawn questions based on the answers to the questionnaire's 4 questions. Is that right? $\endgroup$ May 25 '11 at 19:58
  • $\begingroup$ No, a portion of responses are blanked out from each survey. For example, the guesser is told the voter's answer to questions 1 and 3 and has to guess the voter's answer to questions 2 and 4. $\endgroup$ May 25 '11 at 20:01
1
$\begingroup$

First: Why can't you get the raw data from the GSS? It's easily available. Fail that, you can work with ANES or with the US sample of the World Value Survey. Or raw exit poll data. If you need academic access to get the files, contact me.

Second: The poly-sci way to do this is to run the Ideal or OC to construct a d-dimensional "Issue-space", figure out where the candidates are in issue space(Pretty easy, either by interpreting item parameters of the "Supports Candidate X" question or just by looking at the coordinates of candidate supporters, and then find which 4 questions are maximally informative with regards to issue space.

I actually just finished working through a similar problem.

$\endgroup$
7
  • $\begingroup$ What do you mean "Ideal or OC"? Constructing the issue space is, of course, the foundation and that is precisely where it is obvious to the most casual observer that professional pollsters like Zogby are quite simply out to lunch in their blind polls. The tournament style of questionnaire submission deals with that by not allowing any one viewpoint's biases to determine the range of issues. Concerns about relevance are dealt with by virtue of the fact that an irrelevant questionnaire will provide its author no information with which to predict the other questions. At most it can tie at 0. $\endgroup$ May 30 '11 at 6:21
  • $\begingroup$ Constructing "Issue-space" is a well solved problem in poly-sci, it's usually done via fancy variations of PCA and/or item response analysis. The field is generally called "Roll call analysis", because it's primarily applied to legislator's voting records. But it works with polls too! Provided you transform the questions. "Ideal" and "OC" are very easy to use R packages that will go ahead and estimate it for you. $\endgroup$
    – DavidShor
    May 30 '11 at 9:37
  • $\begingroup$ Generally though, I'm not sure what you mean about providing no information. I've worked with survey releases, primarily from Pew and PPP. All the questions are generally extremely correlated with each other. $\endgroup$
    – DavidShor
    May 30 '11 at 9:40
  • $\begingroup$ The PCA (variations) must be given input. Its input bias I'm concerned with. $\endgroup$ May 30 '11 at 18:04
  • $\begingroup$ By "no information" I'm referring to the "relevance" problem pointed out by Charles Y. Zheng below. Asking questions like "What is the third letter of your last name?" "What is the 4th digit of your phone number?" etc. Likewise, if the questionnaire is relevant and optimal for a given population, there should be no most-likely answer to any question with or without knowledge of the irrelevant questionnaire. $\endgroup$ May 30 '11 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.