Considering the same example of Corona Virus vaccines, assume you carried out a Hypothesis Test with 20 samples and got a P-value of 0.057. So close to beat that 0.05 threshold! Wouldn’t be amazing if you added a few more data points and test again and get a low P-value of <0.05 ? No! This, again, would be P-hacking.
We need to be certain of the size of sample BEFORE we carry out the Hypothesis Test. We cannot make changes to it later to suit our purpose. And this best sample size is achieved using Power Analysis. A Power Analysis determines the sample size that will ensure a high probability of you CORRECTLY rejecting a null hypothesis.
Larger the sample size, more generalized the mean will be, and closer it will be to the population mean. So higher will the probability to correctly reject the null.
For Power Analysis, you need to set a Power value, say 0.8, denoting 80% power, an alpha and the effect size (which estimates the overlap between the distributions of 2 groups). Plugging these values into the formula we get the estimated sample size that will result in 80% probability of correctly rejecting the null!
More is the overlap, lesser will be the confidence for a correct test, so higher will be the sample size value.
#statistics #datascience