We must have an “evidence strong enough” to be able to reject your null hypothesis. This is controlled by the Significance Level.
The Significance Level, also denoted as alpha (α), is a measure of this strength of the evidence that must be present in your experiment before you can reject the null hypothesis and conclude that the effect is statistically significant. And the value of significance level has to be predefined before conducting your experiment.
But what value? A common value of 0.05 or 5% will mean that you are 95% sure that your experiment is correctly done, meaning a 95% Confident Level that there weren’t any errors in your experiment.
Lower the significance value, higher is the confidence. A value of 0.01 will mean you are 99% sure. This value cannot be changed once you start your experiment. For instance, if you set α = 0.01 (99% CI), but couldn’t prove it wrong. And the results showed that you could have proved it wrong if the level was set lower as 95%, you cannot do that!
But what to compare this significance value with? And you can never be 100% certain and there can be errors, chances being 5% or 1%. More on these later!
#datascience #statistics #machinelearning