0
$\begingroup$

The setting (see the dataset at the end of the question)

The setting of the problem is this:

  • I ask many multiple choice questions;
  • only 1 out of the 3 available answers is correct;
  • I ask the same question in two different forms (an EASY form and a COMPLEX form) (i.e. "what color is Hulk?" and "what is the color of the Hulk epidermis surface layer?");
  • The correct answer DOES NOT change (i.e. Hulk is always green);
  • The form of the available answers DOES NOT change, only the form of the question does;
  • Respondents to the questions in the easy form are N, respondents to questions in complex form are n (and n is always less than N)(please assume that there is not any kind of bias, it is not the topic of this question);

My goal is to predict which is the REAL CORRECT answer (TARGET variable in the dataset) based on what respondents choose and in order to do that: (see the dataset at the end of the question)

  1. I empirically noticed that the most selected answer in the EASY form tends to be the correct one (Var A of the dataset)
  2. I empirically noticed that the most selected answer in the COMPLEX form tends to be the correct one (Var B of the dataset)
  3. I empirically noticed that the answer that undergoes the minor reduction in response rate when shifting from EASY to COMPLEX form tends to be the correct one (Var E of the dataset)

Since point 3, is the heart of the question I will be more clear with an example: let's say that we have two possible answers: Answer 1 (A1) and Answer 2 (A2) to the same question (only one is correct), say that A1 has been selected 100 times in the easy form (remember the question is asked either in an easy or complex form, answers stay the same) and 90 times in the complex form, while A2 has been selected 200 times in the easy form while only 40 in the complex form.

Table to sum up:

        _____________________
        |   number of time   |
        |     selected       |
        –––––––––––––––––––––
        |question|question   |
        |in EASY |in COMPLEX |
        |form    |form       |
|–––––––|––––––––|–––––––––––|
|answer |   100  |    90     |
|  1    |        |           |
|–––––––|––––––––|–––––––––––|
|–––––––|––––––––|–––––––––––|
|answer |   200  |    40     |
|  2    |        |           |
|–––––––––––––––––––––––––––––

It is clear that the shift from easy to complex form, has hit way much more A2, and this fact is crucial for me, since indeed the correct answer is A1, exactly for this reason. Mathematically the reduction of A1 is (90-100)/100=-10% While the reduction of A2 is (40-200)/200=-80%

It means that 80% less people selected A2 when the question was asked in a complex form.

Now here comes the question(s) I have for you guys: (see the dataset at the end of the question)

  1. Can I use the Var E (the reduction in the selection) even if it is, of course, correlated with variable A and B since it is a non-linear transformation of these? The fact is that I still do want to include in my model both Var A and Var B, since they are informative as well (I probably will do a decision tree model that will also help me understand and rank which is the most relevant variable, in other terms the one with a greater coefficient) The fact is that the first thing I read on statistics book are statement like: "never never never use correlated variables" and stuff like this, and so I'm a bit terrified now
  2. As you can see from the dataset down there, I have a problem, because some questions receive way much more answers than other. See for example question "wuv" received 9,000,000 answers in the easy form while question "xyz" only 2,000. Now, the dataset I uploaded here, is fake and with example data, but this really happens in my dataset and so I would like to solve this problem. My fear, is not about a Construct Validity Problem or neither a selection bias problem because the number of respondent to each question is totally random (because some time I submit the question to 1000 people, sometimes to 100000) (and assume this is the only way of doing this experiment please). My fear is that it is pointless to include in a model continuous variables (such as Var A and Var B) that are so diverse. For example I do not have this problem with Var E, because it is a percentage value, so always from 0% to 100%. How can i solve this problem? How can I standardise Var A and Var B in way such to eliminate the huge difference in response between question "wuv" and "xyz"?

The dataset (to see column of Var E move the table to the left)

                  TARGET       var A          var B           var C        Var D            Var E
_________________________________________________________________________________________________________|
|answer|question|   is  |n of people that|n of people that||n of respons|n of respons|(varB-varA)/varA   |
|      |   id   |correct|choose it in the|choose it in the||in EASY form|in COMPLEX  |{the % reduction of|
|      |        | (Y/N) |EASY form       |COMPLEX form    ||            |form        | respondent from   |
|      |        |       |                |                ||            |            | EASY to COMPLEX}  |
|——————————————————————————————————————————————————————————————————————————————————————––––––––––––––––––|
|————————————————————————————————————————————————————————————————————–––––––––––———————––––––––––––––––––|
|  1   |  xyz   |   Y   |       1000     |      500       ||    2000    |   850      |   -0.5 (or -50%)  |
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|  2   |  xyz   |   N   |       800      |      300       ||    2000    |   850      |      -0.625       |
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|  3   |  xyz   |   N   |       200      |      50        ||    2000    |   850      |      -0,75        |
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|  1   |  abc   |   N   |       6000     |      800       ||   10000    |   1400     |   -0.8666666      |
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|  2   |  abc   |   Y   |       3000     |      500       ||   10000    |   1400     |   -0.8333333      |
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|  3   |  abc   |   N   |       1000     |      100       ||   10000    |   1400     |      -0.9         |
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|  1   |  wuv   |   N   |    1000000     |    300000      ||  9000000   |   800000   |       -0.7        |
|—————————————————————————————————————————————–——————————————————————––––––———————–––––––––––––––––––––––|
|  2   |  wuv   |   N   |    6000000     |    400000      ||  9000000   |   800000   |    -0.93333       |
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|  3   |  wuv   |   Y   |    2000000     |    100000      ||  9000000   |   800000   |       -0.5        |
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
|————————————————————————————————————————————————————————————————————––––––———————–––––––––––––––––––––––|
| ...  |  ...   |  ...  |       ...      |      ...       ||    ...     |   ...      |                   |

Relevant Facts about the variables in the dataset (how they have been calculated)

Var A < Var B
Var C < Var D

Var C = (varA|answer1) + (varA|answer2) + (varA|answer3)
Example: Var C for the question "xyz" is 1000 + 800 + 200 = 2000

Var D = (varB|answer1)+(varB|answer2)+(varB|answer3)
Example: Var D for the question "xyz" is 500 + 300 + 50 = 850

Example of VAR E for answer 1 of question "xyz": (500-1000)/1000 = -0.5
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.