How large should $M$ be? Here is an interesting question I came across:

Suppose you want to design a multiple choice test with $M$ questions. Each question has $4$ alternatives, out of which only one alternative is correct. How large will you take $M$ such that you can distinguish between a good student and a bad student?

This is the question, and that's it. An answer has to be given using Statistics only using this information.
I would like to see how one can logically approach a solution. Considering Glen_b's advice and whuber's answer, I am quite convinced that there is not a single correct answer to this question. I also feel that this question was essentially devised to check the application skills of the student. (By the way, this is not something I created, it was asked in an admissions interview of a university.)
Something I thought of and not at all sure about:

It does not depend on $M$. It just depends on how much I want a bad student to score. I will assume a bad student randomly guesses each answer, so he has a success probability of $1/4$ and failure probability of $3/4$. Suppose I do not want him to score more than $Mp$ marks, where $0<p<1$ (assuming each question carries $1$ mark and there is no negative marking).
Let $B$ be the number of correct answers given by a bad student. Then clearly, $B\sim Binomian(M,1/4)$.
On an average, I do not want a bad student to score more than $Mp$ marks. Therefore, $Mp>E(B)=M/4$ implying $p>1/4$. So I need to design my cutoff marks, not my number of questions.

Does this seem plausible?
Well, here's the thing: this was not the intended solution! And I do not know what the intended solution is. I would like to understand how such kinds of questions can be tackled using Statistics only. Using estimation, hypothesis testing, etc.
 A: This reply aims to illustrate the art of statistical modeling and the kinds of thought processes that go into it.  It does not purport to develop a unique answer, because (as it will show) no such answer is possible.  It will be content with uncovering important questions to ask in response: questions that will help make the original question more realistic, more specific, and more meaningful.  In short, this reply exemplifies the reasoning that occurs whenever one engages with any question posted here on Cross Validated!
Basic premises
Let's begin with some of the obvious points.  See whether there is reasonable room to disagree with the following interpretations:

*

*Each question will be graded with a numerical score.


*The grading system will not vary from question to question.


*The grading system will reward correct responses and penalize incorrect responses.


*The overall test score will be the sum of the individual question scores.


*"Distinguishing" among students will be based solely on their individual test scores, one per student.


*You are not asked to distinguish students within larger groups: you will be presented only with two test scores, each based on a test of $M$ questions.
Modeling
Now let's start modeling.  We need to understand why test scores can vary between students and even why one student might achieve different scores on different versions of the test.  Many models are possible.  When choosing among them, especially in such abstract or general circumstances, we want to favor generally applicable, simple models that require a minimum of assumptions. We would need to engage in the following considerations.

*

*We might inquire whether the same test will be administered to the students, as is usually the case in a classroom, or a different test for each student, as often occurs in standardized testing situations.  This will determine whether the two scores are independent (or if not, exactly how they might be dependent).


*It will be difficult to proceed if the test is "adaptive" in the sense of selecting subsequent questions based on a student's responses to earlier questions.  We might be obliged to assume the test questions are independently selected from a pool of all such questions.  (If this assumption is untenable, we would need details of the adaptive algorithm.)


*There is potentially a large number of test questions that could be asked, of which each test is a sample.  Because we have no details of what the test measures, and most tests have potentially very large numbers of questions to draw from, we may consider this "pool" of possible questions to be practically infinite.  This assumption simplifies subsequent analysis because we will not have to estimate or consider the size of the pool, about which we know nothing.


*Because we know nothing about the individual test, we should assume it is constructed according to commonly understood conventions of fairness: it should be representative of the entire pool of questions and not "stacked" to measure specialized areas of knowledge.  About the only way to make any progress in analyzing this situation is to assume the selection of questions behaves exactly like a random sample of the pool, for otherwise we would have to inquire about the details of how questions actually have been selected.


*A "good" student is one whose score on a hypothetical test consisting of all possible questions is relatively high, whereas a "bad" student is one whose score on that test is relatively low.  If we don't define "good" and "bad" in this way, then we have to entertain the possibility that the test does not measure what it purports to--but that isn't something we can analyze with the limited information available.


*During any given administration of the test, a student's performances on the individual questions are independent.  This assumption is made to avoid having to model variation in a student's "goodness," about which we have no information at all.


*The performances of the two students are independent.  In particular, one student is not copying from the other.  This assumption captures a commonly understood aspect of test-taking: namely, it is individual and not collaborative.
Analysis
At this point we may have enough information (achieved through the necessary simplifying assumptions) to begin a penetrating analysis.  What immediately becomes apparent is that we know nothing about the difficulty of the questions in the test pool.  For instance, these questions might be extremely difficult, so that no student could expect to improve on random guessing.  Or possibly these questions are extremely easy, so that even the dullest student will likely answer them all correctly.  In such extreme cases it is (intuitively) apparent it will be impossible to distuinguish between "good" and "bad" students using this test: $M$ can be arbitrarily large.
If we would like to proceed constructively, then, we must assume the question pool actually is useful for discriminating knowledge: some questions will be hard, but most good students will be able to answer them correctly, while some will be easy--yet some students will answer them incorrectly.
We still can't get anywhere, because we have to consider possible test-taking strategies.  When multiple-choice questions are awarded one point for a correct answer and zero points for any other kind of answer (or no answer), then guessing the answers to difficult questions is rewarded.  When incorrect answers are penalized, guessing can become a poor strategy.  It is evident from this description that test takers must have some degree of self awareness in order to carry out any rational guessing strategy: they have to have some accurate sense of what they know and do not know!
Gaps and Missing Information
At this juncture we have formulated the problem well enough to have identified critical gaps in the information, gaps where it would be foolhardy to make further assumptions.  These include

*

*We need to know the grading scheme, in particular whether guessing is rewarded or penalized.


*We need to know the degree to which students might be able to rule out obviously wrong answer choices: that is, their degree of self-awareness.


*We might need to know something about the difficulty distribution of the question pool.


*We need to understand whether and how the selections of questions on the test might be interdependent.
The answers to these unknowns will determine how to answer the question.  It could rest on a simple Binomial analysis, but it could also require a different analysis of potential guessing patterns or potential difficulty distributions.
Pushing Towards an Answer
Having some experience in pedagogy and assessment of students, I suspect what the questioner was looking for was not a full answer.  They would instead have been most interested in a thought process that would uncover the unstated assumptions needed in order to made progress towards an answer.  There was no "intended solution."
If they were to push for a full answer--which they might do in order to ascertain your skills at actually using a model--then the trick is to make assumptions that are both realistic and simplify the analysis.  One such set is to suppose that

*

*Each student definitely knows the answer to a proportion $p$ of the questions in the pool and otherwise (a) has no clue yet (b) is aware they have no clue concerning the correct answer.


*Guesses are rewarded: thus, the answer to a question a student does not know is a random choice with a $1/4$ chance of being correct.


*Different tests are administered to the students, assuring independence of the scores.
With these assumptions, the score on a test of $M$ questions is the result of a chain of random processes:

*

*The $M$ questions are sampled from an (infinite) pool.


*Each question is answered where the answer is known and otherwise guessed at.
The distribution of answerable questions in (1) is Binomial$(M,p)$.  Conditional on this number $X$, the distribution of questions that are correctly guessed is Binomial$(M-X, 1/4)$.  Thus the distribution of the score is the sum of these.  Evidently this is a distribution entirely determined by $M$ and $p$.  Indeed, a single randomly chosen question will be correctly answered with chance $p + (1-p)/4$, whence the score has a Binomial$(M, p + (1-p)/4)$ distribution.
One Possible Formulation
With these (grossly) simplifying assumptions, the problem has been formulated as the following:

Given independent random variables $X\sim \text{Binomial}(M, p+(1-p)/4)$ and $Y\sim \text{Binomial}(M, q+(1-q)/4)$,

*

*Construct a test of the hypothesis $p=q$ against the alternative $p\ne q$.

*Compute the power of this test as a function of $p$ and $q$.  The power (obviously) will depend on $M$.

*For a desired power $\beta$ and set $\mathcal{A}$ of specified values of $p$ and $q$ (representing possible pairs of "good" and "bad" students), find the smallest $M$ that achieves a power of at least $\beta$ for all pairs in $\mathcal{A}$.


After answering (2), you might elect to measure the difference between good and bad students (the "effect size") in some way that makes answering $3$ mathematically easy.
There are other formulations, depending on the simplifying assumptions.  We have gone far enough, though, to demonstrate that a solution $M$ can be found because the test power will depend on $M$ in any reasonable situation.
