Question about Multiple Comparisons I was taught to control for multiple comparisons, i.e. when I do more than one test at some significance level, alpha, to lower alpha as given by some choice of a multiple comparisons procedure. Anyone that can answer my question knows what I am talking about. My question is do I account for multiple comparisons:
per section of a paper - so that each part, or section, of a paper has a level of alpha.
per paper - so that the whole paper has a level of alpha
per dataset - so that the whole analysis of the dataset has a level of alpha
per research question- which aims to answer a question and may incorporate multiple datasets and analyses of them, so that the answer to the question has a level of alpha, and may constituent multiple papers, or a single paper with addendums.
or per individual - my whole lifetime - so that I have a level of alpha (this is clearly a joke, although to be able to say that I have a significance level of alpha would be both hilarious and impressive.)
Or some thing else I haven't thought of, or it seems to you I misunderstand something.
 A: The answer will depend on who you ask, the experimental design, the analysis plan, and potentially even the journal you are publishing in. Frequentists and Bayesian have different philosophical beliefs about multiple testing. Observational studies are more at risk of data-driven bias than controlled experiments or trials. Different journals use different statistical guidelines that affect reporting.  With that said, let's consider two common scenarios:


*

*Differential gene expression screens usually screen 20,000 or more variables (genes) for differential expression between two conditions. This means 20,000 hypotheses are usually tested. These screens are often used to generate hypotheses about potential genes of interest. Because the intent is hypothesis generation, we know multiple comparisons will lower our power and miss some genes, but we are OK with this because there are 20,000+ genes and we want to maintain an approximate error rate so we can determine how much money we should expect to spend on true/false positives and prioritize candidates for further research. For settings with an extreme number of tests, I think most would agree some sort of multiple comparisons (or bayesian shrinkage) is needed. 

*Randomized controlled trials (RCTs) usually test a set pre-registered questions that were developed before scientists had access to the data. RCTs usually test around 5 primary hypotheses, a number far from the 20,000 hypotheses tested in gene expression screens. Here the issue becomes more blurry; some statisticians such as Frank Harrell argue that pre-specified questions shouldn't be adjusted for multiple comparisons while the NEJM's Statistical Guidelines advocate

When comparing outcomes in two or more groups in confirmatory analyses, investigators should use the testing procedures specified in the protocol and [statistical analysis plans] to control overall type I error — for example, Bonferroni adjustments or prespecified hierarchical procedures.

These two scenarios represent different sides of the analysis spectrum. Gene expression screens are exploratory in nature, while RCTs are confirmatory. Studies that are exploratory and test a large number of hypotheses should probably adjust for multiple comparisons because the goal is usually to prioritize hypotheses for further investigation. Confirmatory studies with pre-registered questions are less prone to data-driven bias so it is less clear when to adjust for multiple comparisons in a confirmatory setting. This doesn't mean pre-define 1000 hypotheses for your confirmatory study, but instead think twice about if your experiment with two pre-registered, unrelated hypotheses needs multiple comparisons adjustment.
In terms of what level (per section, paper, dataset etc) should you adjust for multiple comparisons, most people adjust for a set of related hypothesis tests pertaining to a single analysis. This analysis could encompass multiple datasets and may or may not take up multiple sections of the paper. Think about what parts of your paper are exploratory vs confirmatory and then think about whether hypotheses were pre-specified or data driven and how many tests were conducted. It's always helpful to talk to a statistician about your specific research to get better insight into these issues. 
Here are a few articles I suggest reading for further insight into this topic:


*

*No Adjustments Are Needed For Multiple Comparisons Kenneth Rothman; Epidemiology

*Multiplicity Considerations in the Design and Analysis of Clinical Trials Cook and Farewell; Journal of the Royal Statistical Society

*Frequentist versus Bayesian approaches to multiple testing Sjölander and Vansteelandt; European Journal of Epidemiology

A: Your question is best answered by asking: What kind of error do you want to be unlikely? This also gives some hint on which adjustment procedure to use. 
Let's start with your joke example. Imagine a politician claiming in his election campaign: "Vote for me as I never state an erroneous claim in my life." Clearly this guy should control for the  probability that any of his claims until election day is wrong. (Opening quite an interesting statistical question on how to adjust for multiplicity if the number of tests is unknown in advance.).
Now imagine a dose finding study: Imagine experimental units that get a toxic agent in different doses each group. At which dose do toxic symptoms start to show in a way that a physiological parameter of the experimental units exceeds its safe range? It may be possible that no toxic symptoms show at all as the maximum dose is still safe. Then we don't get to know the toxic dose, so we stay with the null hypothesis. 
The opposite will be that at least one dose is toxic and the lowest of these doses is the answer to the research question. So you choose a multiplicity adjustment procedure that may incorporate the natural ordering of the doses and its conditional probabilities: Given a low dose is toxic, a higher dose will be toxic as well.
Another more equality related example: Imagine a survey on political party preference and consumer buying habits. Your sample is divided in supporters of more than two political parties. You ask: The supporter of which party would 
prefer a new product X over Y? There you don't have an ordering like in the dose finding example. But again, it may happen that among all subgroups product X is not preferred and finding at least one subgroup that seems to prefer X would be an error. So you have to adjust for the opposite: Supporters of at least one party prefer X over Y.
However, if the survey's intention is just to find at which party convention to advertise a product (now that's my joke) with a given budget, multiplicity correction is not necessary. You spend the budget anyway. The question is rather which subgroup would ignore the ad anyway. 
How to adjust in gene screening? You have individuals separated into two groups by a certain phenotype. Now you are pretty sure that this is caused by different genotypes but you want to find significant genes. In this case, the question "is there any gene?" is of no importance: You are already pretty sure. In this case multiplicity adjustment should make sure that among the significant genes, only a small proportion is erroneuously found. This leads to an adjustment procedure for false discovery rate (FDR) which is more powerful than the family-wise error rate (FWER) used in the dose finding example.
I hope this helps getting some orientation on how to choose multiplicity adjustment. I think it is not so helpful to classify the research questions and decide upon that nominal classification. Rather think of the "sachverhalt", the related facts, and the intention of the analysis. What outcome would be an error depending on what? 
