Statistical Power Analysis in Machine Learning?

Power analysis is directly related to tests of hypotheses. While conducting tests of hypotheses, the researcher can commit two types of errors: [Type I error and Type II errorStatistical power mainly deals with Type II errors.

It should be noted by the researcher that the larger the size of the sample, the easier it is for the researcher to achieve the 0.05 level of significance. If the sample is too small, however, then the investigator might commit a Type II error due to insufficient power.

Power analysis is normally conducted before the data collection. The main purpose underlying power analysis is to help the researcher to determine the smallest sample size that is suitable to detect the effect of a given test at the desired level of significance. The reason for applying power analysis is that, ideally, the investigator desires a smaller sample because larger samples are often costlier than smaller samples. Smaller samples also optimize the [significance testing.