Skip to main content
Cross Validated

Return to Answer

edited body
Source Link
Ben
Ben
  • 133k
  • 7
  • 255
  • 588

A priori sample size calculations are only sensible if there is a fixed cost to sampling

This is an excellent question, and it raises some common-sense issues that are commonly misunderstood in sampling theory. When we are considering sample size questions, the first thing we need to ask ourselves is what (if any) impediment, inconvenience, cost, etc., there is to just collecting all the data there is on the matter of interest. If there is no cost to this (construed broadly as including time, money, etc.) then we might as well go out and collect all possible data on the matter and then we don't need statistical inference procedures at all. Consequently, the idea of using a limited sample size that is less than what is possible only arises as a sensible procedure when there is some cost to sampling.

Once we have identified the costs of sampling (again, construed broadly to include more than just money), we need to decide if there are any fixed costs that accrue when we conduct a sampling "excursion", or not. If there are no fixed costs then there is no need to limit our data collection by any a priori calculation --- we can just collect some data, decide if it is enough, collect some more, decide if it is enough, and so on. So in this case, we can indeed todo something like what you describe; sampling without any a prioiri sample size calculation and then stopping when we have enough data for the inferences we want to make. (We need to be a bit careful with this method, because certain kinds of "stopping rules" combined with particular statistical procedures can bias analysis, but assuming we are cognisant of this, we should be able to formulate an appropriate "stopping rule" and stop when we have the data we need.) Various statistical control processes operate in exactly this way --- they collect data as it is generated and update inferences with each new data point.

If there are fixed costs to sampling, but the fixed costs are quite low relative to the unit costs, it might still be useful to sample over more than one "excursion". In such a case we might make an initial a priori sample size calculation for an initial "pilot" set of data, then use this data to make an updated sample size calculation for the next set of data, and so on. In this case there might be a number of data collection excursions, but each will involve a sample size calculation. This is also somewhat similar to what you describe. Finally, if the fixed costs of sampling are quite substantial relative to the unit costs, then the "traditional" method of using only one sampling "excursion" is probably going to be optimal. In this case the sample size is fully determined by a single a priori calculation.

So, in terms of where your reasoning breaks down, it is only in the possibly exaggerated claim that "[t]raditionally, sample size determination is done as part of the design phase." This is only true if there are substantial fixed costs to sampling that make it desirable to sample "in one go" and thereby necessitate an a priori computation of the entire sample size. Your observation that it can be sensible to collect data "as you go" is perfectly legitimate (under appropriate circumstances), and it is actually an underappreciated principle of sampling theory.

A priori sample size calculations are only sensible if there is a fixed cost to sampling

This is an excellent question, and it raises some common-sense issues that are commonly misunderstood in sampling theory. When we are considering sample size questions, the first thing we need to ask ourselves is what (if any) impediment, inconvenience, cost, etc., there is to just collecting all the data there is on the matter of interest. If there is no cost to this (construed broadly as including time, money, etc.) then we might as well go out and collect all possible data on the matter and then we don't need statistical inference procedures at all. Consequently, the idea of using a limited sample size that is less than what is possible only arises as a sensible procedure when there is some cost to sampling.

Once we have identified the costs of sampling (again, construed broadly to include more than just money), we need to decide if there are any fixed costs that accrue when we conduct a sampling "excursion", or not. If there are no fixed costs then there is no need to limit our data collection by any a priori calculation --- we can just collect some data, decide if it is enough, collect some more, decide if it is enough, and so on. So in this case, we can indeed to something like what you describe; sampling without any a prioiri sample size calculation and then stopping when we have enough data for the inferences we want to make. (We need to be a bit careful with this method, because certain kinds of "stopping rules" combined with particular statistical procedures can bias analysis, but assuming we are cognisant of this, we should be able to formulate an appropriate "stopping rule" and stop when we have the data we need.) Various statistical control processes operate in exactly this way --- they collect data as it is generated and update inferences with each new data point.

If there are fixed costs to sampling, but the fixed costs are quite low relative to the unit costs, it might still be useful to sample over more than one "excursion". In such a case we might make an initial a priori sample size calculation for an initial "pilot" set of data, then use this data to make an updated sample size calculation for the next set of data, and so on. In this case there might be a number of data collection excursions, but each will involve a sample size calculation. This is also somewhat similar to what you describe. Finally, if the fixed costs of sampling are quite substantial relative to the unit costs, then the "traditional" method of using only one sampling "excursion" is probably going to be optimal. In this case the sample size is fully determined by a single a priori calculation.

So, in terms of where your reasoning breaks down, it is only in the possibly exaggerated claim that "[t]raditionally, sample size determination is done as part of the design phase." This is only true if there are substantial fixed costs to sampling that make it desirable to sample "in one go" and thereby necessitate an a priori computation of the entire sample size. Your observation that it can be sensible to collect data "as you go" is perfectly legitimate (under appropriate circumstances), and it is actually an underappreciated principle of sampling theory.

A priori sample size calculations are only sensible if there is a fixed cost to sampling

This is an excellent question, and it raises some common-sense issues that are commonly misunderstood in sampling theory. When we are considering sample size questions, the first thing we need to ask ourselves is what (if any) impediment, inconvenience, cost, etc., there is to just collecting all the data there is on the matter of interest. If there is no cost to this (construed broadly as including time, money, etc.) then we might as well go out and collect all possible data on the matter and then we don't need statistical inference procedures at all. Consequently, the idea of using a limited sample size that is less than what is possible only arises as a sensible procedure when there is some cost to sampling.

Once we have identified the costs of sampling (again, construed broadly to include more than just money), we need to decide if there are any fixed costs that accrue when we conduct a sampling "excursion", or not. If there are no fixed costs then there is no need to limit our data collection by any a priori calculation --- we can just collect some data, decide if it is enough, collect some more, decide if it is enough, and so on. So in this case, we can indeed do something like what you describe; sampling without any a prioiri sample size calculation and then stopping when we have enough data for the inferences we want to make. (We need to be a bit careful with this method, because certain kinds of "stopping rules" combined with particular statistical procedures can bias analysis, but assuming we are cognisant of this, we should be able to formulate an appropriate "stopping rule" and stop when we have the data we need.) Various statistical control processes operate in exactly this way --- they collect data as it is generated and update inferences with each new data point.

If there are fixed costs to sampling, but the fixed costs are quite low relative to the unit costs, it might still be useful to sample over more than one "excursion". In such a case we might make an initial a priori sample size calculation for an initial "pilot" set of data, then use this data to make an updated sample size calculation for the next set of data, and so on. In this case there might be a number of data collection excursions, but each will involve a sample size calculation. This is also somewhat similar to what you describe. Finally, if the fixed costs of sampling are quite substantial relative to the unit costs, then the "traditional" method of using only one sampling "excursion" is probably going to be optimal. In this case the sample size is fully determined by a single a priori calculation.

So, in terms of where your reasoning breaks down, it is only in the possibly exaggerated claim that "[t]raditionally, sample size determination is done as part of the design phase." This is only true if there are substantial fixed costs to sampling that make it desirable to sample "in one go" and thereby necessitate an a priori computation of the entire sample size. Your observation that it can be sensible to collect data "as you go" is perfectly legitimate (under appropriate circumstances), and it is actually an underappreciated principle of sampling theory.

Bounty Ended with 25 reputation awarded by CommunityBot
Source Link
Ben
Ben
  • 133k
  • 7
  • 255
  • 588

A priori sample size calculations are only sensible if there is a fixed cost to sampling

This is an excellent question, and it raises some common-sense issues that are commonly misunderstood in sampling theory. When we are considering sample size questions, the first thing we need to ask ourselves is what (if any) impediment, inconvenience, cost, etc., there is to just collecting all the data there is on the matter of interest. If there is no cost to this (construed broadly as including time, money, etc.) then we might as well go out and collect all possible data on the matter and then we don't need statistical inference procedures at all. Consequently, the idea of using a limited sample size that is less than what is possible only arises as a sensible procedure when there is some cost to sampling.

Once we have identified the costs of sampling (again, construed broadly to include more than just money), we need to decide if there are any fixed costs that accrue when we conduct a sampling "excursion", or not. If there are no fixed costs then there is no need to limit our data collection by any a priori calculation --- we can just collect some data, decide if it is enough, collect some more, decide if it is enough, and so on. So in this case, we can indeed to something like what you describe; sampling without any a prioiri sample size calculation and then stopping when we have enough data for the inferences we want to make. (We need to be a bit careful with this method, because certain kinds of "stopping rules" combined with particular statistical procedures can bias analysis, but assuming we are cognisant of this, we should be able to formulate an appropriate "stopping rule" and stop when we have the data we need.) Various statistical control processes operate in exactly this way --- they collect data as it is generated and update inferences with each new data point.

If there are fixed costs to sampling, but the fixed costs are quite low relative to the unit costs, it might still be useful to sample over more than one "excursion". In such a case we might make an initial a priori sample size calculation for an initial "pilot" set of data, then use this data to make an updated sample size calculation for the next set of data, and so on. In this case there might be a number of data collection excursions, but each will involve a sample size calculation. This is also somewhat similar to what you describe. Finally, if the fixed costs of sampling are quite substantial relative to the unit costs, then the "traditional" method of using only one sampling "excursion" is probably going to be optimal. In this case the sample size is fully determined by a single a priori calculation.

So, in terms of where your reasoning breaks down, it is only in the possibly exaggerated claim that "[t]raditionally, sample size determination is done as part of the design phase." This is only true if there are substantial fixed costs to sampling that make it desirable to sample "in one go" and thereby necessitate an a priori computation of the entire sample size. Your observation that it can be sensible to collect data "as you go" is perfectly legitimate (under appropriate circumstances), and it is actually an underappreciated principle of sampling theory.