TU-L0022 - Statistical Research Methods D, Lecture, 2.11.2021-6.4.2022

Reliability is one way that the randomness can influence your study results, but it is not the only way, and therefore it is important to understand how different sources of error are related and how they differ. Reliability and sampling error are two main sources of random errors, so reliability is concerned about measures of individuals how much your individual measures vary from one measurement location to another. Sampling error, on the other hand, is another kind of process that produces random differences to your study results. So sampling error refers to the composition of the sample and how it varies from one random sample to another. So the idea here is that a reliability is measures of the same individuals how much they vary from one study to another if we do the same measurement procedures.

Sampling error refers to how much different repeated random samples from the same population influenced the study results. The definition for reliability is how much measure of a trait of an individual varies over measurement occasions for sampling error, how much statistical estimate varies over repeated samples because different samples contain different individuals. So this again, is a bit different. Causes of reliability and sampling error are also different. Reliability is related to the quality of your measurement instrument, it's not related to sample size, and sampling error, on the other hand, is related to sample size and population heterogeneity only if you have a random sample. Consequences of these two sources of error are also different. So reliability. If you have unreliable measures, then statistical associations using those unreliable measures will be inconsistent and biased unless you explicitly take reliability into account in your statistical models, which can be a bit complicated to do.

Sampling error, on the other hand, simply influences the efficiency of your estimates. If you have a high sampling error, then your results could still be unbiased. They are just less precise. So these two sources of random error have different, very different consequences. Sampling error is easy to reduce by increasing sample size. To reduce unreliability, you have to improve your measurement practices. Let's take a look at the bathroom scale example. So reliability here is or sampling error is whether these real weights here, these people here are an accurate presentation of the population. So is this a representative sample? That's what the sampling error is about. If we happen to take shorter people than on average in the population by chance our sample, then we'll have a larger sampling error than what we would have. If our people in the sample are closer to the population mean, then reliability is whether this real weights and the bathroom scale readings actually agree. So, assuming complete perfect validity, there would be no other measurement there except reliability. So these are about two different quantities. Is our sample representative that's the sampling error? Are our measures representative of these actual values? How closely they match? That's the question of reliability assuming perfect validity. We can also take a look at this through the hierarchy of reliability and validity.

So reliability is whether you have the same sample, you measure it again, do you get the same result? Sampling error goes to the statistical conclusion validity and it's about if you have a new sample from the same population, do you get the same result? If you don't get the same result from a new sample, then either you have a reliability problem or you have a sampling error issue. If you have a different result from the same sample, then it's a reliability issue. Then finally we have the external validity which is about new population. So if our results don't generalize if we repeat the study with the new population and we don't get the same result, then if we have ruled out sampling error, we have ruled out unreliability, then it's an issue of external validity or generalizability. So these are different sources of random error or why your results could be different from one study to another and it's important to understand what the differences are because they are sometimes confused.