# Does this qualify as an experiment?

I have to do an AP Statistics project, and I must conduct an experiment. I am having trouble as to whether or not my idea would qualify as an experiment:

To find out if students really know contemporary politics, we decided to design an experiment. We will provide subjects from group 1 with a false question and subjects from group 2 with a true question.

Question 1 (False): "Do you support Democratic Candidate Bernie Sanders' plans to lower taxes on American Citizens?"

Question 2 (True): "Do you support Democratic Candidate Bernie Sanders" plan to increase taxes on American citizens?"

We think most students don't really know their politics. The idea here is to give group 1 an obvious and agreeable question, even though the substance of the question is bluntly false and untrue. This will serve as the treatment of the experiment.

Our second question is a standard and neutral question and will serve as the constant in this experiment for us to measure against, because it is a true statement. If this even qualifies as an experiment, please help with the following questions regarding inference and analysis:

• Parameter: ??????? Would it be "the proportion of students who understand and support Bernie Sanders' plan to increase taxes"? In this case, the "Proportion of students who answer "Yes" to question 2.???

• What are the Null and Alternative Hypotheses?

• What Test would I use?

• Wouldn't it make more sense to give students several multiple choice questions to test their political knowledge? I don't think your experimental setup really has a null/alternative hypothesis. For the test I suggest, the null hypotheses might be "students score better on a multiple choice political exam than they would from random guessing", and the alternative hypothesis might be "students do no better than random guessing on a political exam".
– user1566
Feb 22 '16 at 20:15
• What Test should I use? Thanks Feb 23 '16 at 15:56

First, you'll need to refine your scientific hypothesis and introduce a statistical hypothesis. Currently, your scientific hypothesis is "we think most students don't know their politics", and you have not yet specified a statistical hypothesis.

A scientific hypothesis is a statement about the state of the world that requires verification. All scientific hypotheses have three characteristics in common:

1) They are intelligent, informed guesses about some phenomena.

2) They can be reduced to an if-then statement (e.g. "if Bob exercises, then he'll lose weight).

3) Their truth or falsity can be determined by observation or experimentation.

A statistical hypothesis is a statement about one or more parameters of a population. For example, μ < 80 is a statistical hypothesis; it states that the population mean is less than 80 (e.g. the final grade average of AP Statistics students in High School X is less than 80.)

Importantly, your operationalization of the scientific hypothesis must reflect the scientific hypothesis itself. Currently, you have four possible outcomes:

YES to Q1; interpretation: students don't know contemporary politics

NO to Q1; interpretation: students know contemporary politics

YES to Q2; interpretation: students know contemporary politics

NO to Q2; interpretation: students don't know contemporary politics

Clearly, if someone answers YES to Q1, it does not mean that the person does not know contemporary politics; conversely, it might mean the individual is well-versed in politics, and supports the idea of increased taxation with the understanding that, in return, the government would spend that money on public goods and services.

The point here is not to debate nuisances of tax policy, but rather to show that, if you want to make inferences from data, it's important to set up questions that faithfully tap into the scientific hypothesis.

If you want to measure a person's knowledge of contemporary politics, or for that matter, measure anything at all, chances are someone else has already done so, and you'd be best served by using tried-and-true methods. For example, you might use a sample of questions from the US Naturalization Exam given to immigrants. This test explicitly gauges immigrants' knowledge about United States history and politics.

Here are some suggestions for you moving forward:

Think about how you might revise your current hypothesis to an if-then statement. Then, introduce a statistical hypothesis that includes a null and an alternative. The test you use will depend upon the number and nature of your dependent variable (s), as well as the number and nature of your independent variable(s). I assume that your class has covered a number of statistical tests.

Here is an example from Kirk (2013) in evaluating a scientific hypothesis that should help you think about experimental design:

Step 1: State the Scientific Hypothesis

On average, college students who are active in student government at Big Ten universities have higher IQs than college students who are not involved in student government.

Step 2: State the Statistical Hypothesis

Kirk assumes that the mean IQ of non-involved college students, H-null, is 115. Thus, the null and alternatives are:

Null hypothesis: μ ≤ 115

Alternative hypothesis: μ > 115.

As written, the null hypothesis states that the mean IQ of active students is equal to or less than the mean IQ of college students who are not involved in student government (which is known to be 115). Alternatively, the mean IQ of active students is higher than the mean IQ of non-involved students.

Step 3. Specify the Test Statistic

Given that we want to compare a sample mean to a known population mean, we use a one-sample t-test. Other important assumptions include that the population of IQ scores we are sampling from are approximately normal, and that our sample is random.

Step 4. Specify the sample size and the sampling distribution

n = 60 and a t-distribution with n-1 (60) degrees of freedom.

Step 5. Specify the level of significance:

α (alpha) = 0.05

Step 6. Obtain a random sample of size n, compute t, and make a decision.

Decision rule:

Reject the null hypothesis if t falls in the upper 5% of the sampling distribution of t; otherwise, do not reject the null hypothesis.