Confidence Intervals

From Wiki: In statistics, a confidence interval (CI) is a type of interval estimate, computed from the statistics of the observed data, that might contain the true value of an unknown population parameter. The CI has an associated confidence level that, loosely speaking, quantifies the level of confidence that the parameter lies in the interval.

More strictly speaking, the CI represents the frequency (i.e. the proportion) of possible confidence intervals that contain the true value of the unknown population parameter. If confidence intervals are constructed for a given confidence level from an infinite number of independent sample statistics, the proportion of those intervals that contain the true value of the parameter will be equal to the confidence level.

Menzi Chinn highlights what a Confidence Intervals is not:

The specific 95 % confidence interval presented by a study has a 95 % chance of containing the true effect size. No!

A reported confidence interval is a range between two numbers. The frequency with which an observed interval (e.g., 0.72–2.88) contains the true effect is either 100 % if the true effect is within the interval or 0 % if not; the 95 % [confidence interval] refers only to how often 95 % confidence intervals computed from very many studies would contain the true size, if all the assumptions used to compute the intervals were correct. [A polite way of saying, if your statistical model is garbage, your confidence interval is garbage, too. Garbage in Garbage Out.] It is possible to compute an interval that can be interpreted as having 95 % probability of containing the true value; nonetheless, such computations require not only the assumptions used to compute the confidence interval, but also further assumptions about the size of effects in the model. These further assumptions are summarized in what is called a prior distribution, and the resulting intervals are usually called Bayesian posterior (or credible) intervals to distinguish them from confidence intervals. Source: Greenland et al. (2016).