Sample Size Calculation: what you need to know
The sample size in a quantitative study refers to the number of subjects drawn from the target population to participate in the study. At the start of a research study, sample size (aka power calculation) answers the question: “How many participants or subjects do I need to include in this study”?
Although sample size determination plays a crucial role in reproducible quantitative research, few researchers know how to determine sample size. In this blog post, I will briefly review the key considerations for sample size estimation.
Given that the primary purpose of any research study that is based on a sample of the population is to be able to generalize the results to the larger population, it is important to determine the ideal sample size. This way, you ensure that the results obtained from the study can be generalized with a high degree of confidence back to the larger population.
The key factors that influence sample size estimation include, the sampling method adopted for sample selection, the research study design, and the statistical analysis.
Sampling method refer to how the researcher decides to select a subset of research participants from the sampling frame. There are two main sampling methods, which are: probability and nonprobability sampling methods. Probability sampling utilizes a random selection process in which every subject has an equal nonzero chance of being selected. Example probability sampling methods include simple random sampling, stratified sampling, systematic sampling, and cluster sampling. Conversely, in nonprobability sampling subjects are selected based on subjective judgment rather than random selection, and not all subjects have an equal nonzero chance of being selected. Example nonprobability sampling methods include convenience sampling, quota sampling, snowball sampling and purposive sampling. In relation to the purpose of sampling-based research which is the ability to generalize study results to the larger population from which the sample was drawn, probability sampling methods have the advantage of higher generalizability, greater representativeness of the population and lower response bias. For these reasons, it is appropriate to select an adequate sample size for research that adopts probability sampling methods. Conversely, estimating sample size when using nonprobability sampling methods will not be relevant since the convenient sampling approach adopted are not likely to generate results that will allow for making statistical inference to the larger population. To reduce the problem of non-generalizability associated with nonprobability sampling methods, researchers should strive to include as many subjects as possible from the different subgroups and demographics available in the larger population.
Research study designs are broadly divided into two categories – observational and experimental. An important distinction between these two categories is that with observational study designs, the researcher does not impose any intervention and observes only to assess the current conditions.
Descriptive studies are the simplest forms of observational studies and are generally designed to describe the distribution of one or more variables, without regard to any causal or other hypothesis. Examples include case reports, case series and cross-sectional studies. In descriptive studies, the main parameter of interest could be a categorical outcome (such as proportion or prevalence) or a continuous outcome (such as the mean). The steps for estimating sample size in such descriptive statics are as follows: First determine the confidence level and confidence interval (aka margin of random error). If the outcome variable of interest is the mean, proceed to determine the standard deviation. However, if the outcome variable of interest is a proportion in the population of interest, proceed to determine the proportion. How do we interpret these sample size concepts when applied to descriptive studies in practice.
If you set a 95% confidence level for the outcome of interest for example, that would indicate that the outcome of interest (e.g., the sample mean or proportion) will not differ by more than a certain value from the true population mean/proportion in 95% of the sample if it is repeatedly drawn from the same population. The confidence interval (aka margin of error) is a measure of the precision of an estimate, that is how certain you can be that the statistics you calculated for the sample are close to what they would be if it were possible to measure the same statistics in the larger population from which the sample was drawn. The smaller the specified margin of error the larger the precision of our estimates and the larger the ideal sample size. Assuming we specify a 5% confidence interval, for a study that seeks to determine the prevalence of underaged drinking in a sample of community residents. If the prevalence of underaged drinking in our sample is 20%, this would mean that for the larger population community residents, we estimate underaged drinking to be between 15% and 25% (allowing for 5% margin of error on both sides).
In experimental study designs, the researcher conducts an intervention and records the results. When the researcher is interested in comparing two groups, usually in the form of an experimental/intervention and control group, it is recommended that the study participants be split equally between the two groups – a consideration that does not apply to observational studies (prospective vs. retrospective). This approach ensures maximum power for the given sample size. In this situation, the minimum sample size for each group will have to be determined based on the statistical test used.
The other very important consideration for experimental studies is the need to determine the effect size. The effect size is defined as the minimum effect an intervention must have in order to be considered practically significant. This has been considered one of the most challenging steps in sample size estimation. For one it’s hard to determine an acceptable effect size. For example, is the effect size small, medium or large? A couple of solutions have been suggested in situations where effect sizes cannot be determined. One suggestion recommends small, medium, and large effect sizes. Another recommendation is the use of a range of standardized or unit-free effect sizes.
The statistical analysis to be conducted is one of the most important considerations in sample size estimation. The statistical analysis in turn is determined by the study objective, the study design, the research question(s) and primary research outcome. In correlation and regression analysis for example, the researcher seeks to examine the relationship between a dependent/response variable and one or more independent variables. Using an insufficient sample size will make the results not generalizable to the larger population. For regression analysis, the number of predictors (independent variables) is important for estimating sample size. Larger sample sizes are required for higher numbers of predictors. Another important factor in sample size estimation for regression analysis is the R-squared, which is defined as the proportion of variance in the response that is explained collectively by the independent variables.
It should be mentioned though that in a lot of cases, it is not practical to determine the ideal sample size for regression analysis. This is because oftentimes there may be more than one multiple regression model. For this reason, some researchers have relied on “rules-of-thumb” to estimate sample sizes. One common rule-of-thumb is to ensure a minimum of 10 observations per variable.
It is worth mentioning that in most cases the population size is not needed as input in sample size calculations. However, if the population is limited and the size of the finite population can be obtained, this can be included in the sample size calculation and the equation for sample size calculation should be adjusted for the population size.
In addition to all the key considerations highlighted above, researchers will also have to account for the reality that not every subject in the estimated sample size will participate in the study. To account for dropout or non-responsiveness, researchers would normally recruit more subjects so that even when some subjects’ dropout or are non-responsive, they are still able to achieve the planned sample size for the study.
One last point I should quickly mention is that, while there are formulas for estimating sample size, you do not have to calculate sample size manually. Several open source and commercial proprietary software are available for sample size estimation. The open-source tools include OpenEpi, G*Power, POWER, PS, and GLIMMPSE.
In conclusion, estimating sample size is a key aspect of quantitative research. It is challenging and sometimes relies on assumptions that may not always hold true. Not to mention that there are several factors that need to be considered in sample size estimation. However, using the ideal sample size will produce research results that are reproducible, generalizable and of more practical significance. The process should always involve some serious consideration about the purpose of the research, the population of interest, and how confident you want to be with your study results.