Skip to main content
Cross Validated

Return to Answer

added 13 characters in body
Source Link
Matthew Gunn
Matthew Gunn
  • 23k
  • 1
  • 62
  • 95

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, you have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. In some sense everybody got treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In the example in your question, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. WholeCourses, series of courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, you have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. In some sense everybody got treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In the example in your question, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Whole courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, you have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. In some sense everybody got treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In the example in your question, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Courses, series of courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

deleted 37 characters in body
Source Link
Matthew Gunn
Matthew Gunn
  • 23k
  • 1
  • 62
  • 95

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, you have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. Everybody gotIn some kind ofsense everybody got treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In thisthe example in your question, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Whole courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, you have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. Everybody got some kind of treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In this example, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Whole courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, you have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. In some sense everybody got treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In the example in your question, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Whole courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

deleted 37 characters in body
Source Link
Matthew Gunn
Matthew Gunn
  • 23k
  • 1
  • 62
  • 95

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, it isyou have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias. Randomly assignment of treatment works quite well in a science laboratory, but it is much more difficult to accomplish in the social setting.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. Everybody got some kind of treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In this example, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Whole courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, it is selection bias. Randomly assignment of treatment works quite well in a science laboratory, but it is much more difficult to accomplish in the social setting.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. Everybody got some kind of treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In this example, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Whole courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

The reasoning in the first text is not flawed. It is correct.

Theory of random assignment

If treatment is randomly assigned, any unobservables in the treatment group will be balanced with unobservables in the control group. The same effects from unobservables in the treatment group will also be present in the control group! The difference in effect size between the two groups will be an estimate of the treatment effect.

The key concept here is that you don't need to control for everything to produce consistent estimates of an effect. You need treatment to be orthogonal of confounding, unobservable variables.

Can spurious things happen with small populations?

With small groups, you may, by chance have an unbalanced assignment of unobservables. Randomly pick a group of 10 people and you might have 7 guys and 3 girls.

But if you randomly pick a group of 10,000, the split will be close to 50-50 (or whatever the split is in the population.) With larger populations, it becomes less and less likely that your control group differs from your treatment group by chance.

This may be a problem for small studies, but as they get larger, this is less of a concern.

The bigger problem?

In the social setting, you have many problems where random assignment may diverge from a mathematical ideal. For example, there may be selection bias.

Example: 100 kids are randomly accepted to a preschool program. Another 100 kids are randomly denied. It looks like random assignment of treatment (and a control group), but what if parents of the 100 kids that are denied INSTEAD find alternatives to the preschool program?

Denial of treatment causes kids to get unobserved, supplemental education! And unobservables aren't equally balanced between treatment and control. Everybody got some kind of treatment! Your experiment then isn't comparing apples to no apples, it's comparing one kind of apple to another...

In this example, I would think assigning people to work out may be quite hard! Does the treatment group actually work out? How do you prevent the control group from not working out?

Summary

Do additional experiments help? Yes. They may use better techniques, have different error etc... More knowledge is better.

Can we have an unrepresentative control and treatment group by chance? Yes. But with true random assignment, it becomes increasingly unlikely as $n$ increases. Flipping coins, 47 or fewer heads in a sample of 100 is quite likely. 4700 or fewer heads in a sample of 10000 is close to impossible.

Should I trust my intuition on probability? Probably not. People (me included) have awful intuition about probability. It's hard.

So all science is wonderful? No! There are so many ways things can be wrong. Assumptions of the statistics can be horribly violated in so many different. Whole courses are devoted to experimental methods for good reason! In practice, random assignment doesn't work as nicely with people as it does in the laboratory because people are clever, smart and can respond in ways you didn't even imagine!

added 186 characters in body
Source Link
Matthew Gunn
Matthew Gunn
  • 23k
  • 1
  • 62
  • 95
added 797 characters in body
Source Link
Matthew Gunn
Matthew Gunn
  • 23k
  • 1
  • 62
  • 95
added 797 characters in body
Source Link
Matthew Gunn
Matthew Gunn
  • 23k
  • 1
  • 62
  • 95
Source Link
Matthew Gunn
Matthew Gunn
  • 23k
  • 1
  • 62
  • 95