TU-L0022 - Statistical Research Methods D, Lecture, 2.11.2021-6.4.2022

We also have a second strategy for statistical inference.  The problem with p-values is that, p-value only answers the question, is it likely or is it plausible that there is no effect in the population? But p-value doesn't really provide us with any direct evidence of the uncertainty of the estimate of the effect. And therefore we have another strategy called confidence intervals. So a confidence interval is an interval of two endpoints constructed around the estimate.

So here's an example of a confidence interval, the estimate is 2.02 and we have one endpoint that is below the estimate, which is 2.01 and then another endpoint that is above the estimate that is 2.03. So that's an interval that says something about the precision of the estimates. Formally the confidence interval is defined as an interval that is calculated in a way that, if we repeat the sample over and over many times, then the true population value will fall within that interval, 95% of the replications. So 5 times out of 100 replications, the population value would be outside the interval. So from this inference, we can kind of infer that, the population value is maybe somewhere within the interval. We can't say that it's precisely there in any formal way but we can kind of infer that maybe it's there. So these intervals can also be used the same way as a null hypothesis significance test, so you can compare whether zero is included in the interval. So interval here is 2.01 and 2.03, zero is not within the interval and therefore we say that it's unlikely that the population value would be 0.

To understand the confidence interval better, it's useful to understand it in the framework of the previous examples. So we had the difference between men-led companies and women-led companies that occurs by chance only, sometimes we get a large difference, sometimes we get a small difference. And these confidence intervals are intervals that are constructed around an estimate. So let's say an estimate is here, then the interval could be here. So this interval would not contain the population value, the population value of zero is above the interval. These two intervals would contain the population value, and in this last interval, constructed around this estimate, the population value falls below the interval. If the confidence interval is valid then these intervals, that include the population value, which in this case is 0, but it could be something else as well, will be 95% of the case. So 5 times out of 20, we get either this one or that one, so the population value is outside the interval. We can kind of informally infer that, maybe the population value is somewhere within the interval, but we can't say it precisely.

How these confidence intervals are calculated? One way, particularly common way, is to use a normal approximation. So the idea of a normal approximation confidence interval is that, we construct the interval so that it's the estimate minus 1.96, which is two standard deviations or 95 % of the normal distribution, times the standard error. So then the upper interval is the estimate plus 1.96 times the standard error. Why we multiply with 1.96 is that way we get 95 % of the normal distribution within the interval. So the confidence interval estimate, we can just do a little bit of math, and we can say that this is equivalent to comparing the estimate divided by standard error to 1.96, which is the t test basically. So there is an equivalence between t test or z test, to be more precise, which is a comparison against normal distribution, instead of student's t distribution. So if we just compare, whether the confidence interval includes a 0 or not, that is exactly the same thing as calculating a p-value and comparing it against 0.05. So doing a confidence interval doesn't make us any smarter, it's just the same thing in a slightly more complex way. If we can't assume that the estimates are normal then these two approaches are not the same, and there are techniques for calculating confidence intervals for that scenario as well. But the important thing to know about confidence intervals is that they are pretty useless if you just check whether 0 is in the interval.

There is a nice quote in an article by Cortina, who attributes the quote to Thompson that, if we were to be as rigid with confidence intervals as we are with the p-values, taking in the 0.05 as the gold standard, then we would just be stupid on another metric. So doing confidence intervals without interpreting, what the endpoints mean and just checking, whether zero is within the interval doesn't really make any sense whatsoever. The problem with confidence intervals and p-values is that both are commonly misinterpreted. So the p-value is, the probability of obtaining the opposite result if the null hypothesis is correct. It is not the probability that the null hypothesis is correct. I will explain that more in another video. So if you guessed a dice that is thrown correctly, it doesn't mean that you're clairvoyant. You could guess a dice randomly. The confidence interval is an interval that will contain the population value with the frequency of given confidence level.  It is not the probability that the population value is contained within a particular interval, that's a different thing. Understanding, why these two are not the same, is a bit more complicated, so I will not cover that but we'll take a look at this in more detail.