Sample Selection

The validity of being able to use a sample in lieu the entire population to draw conclusion about the population rests on the assumption that the sample is just like the population regarding the quantity of interest, only smaller in size. If this turns out not to be true, then you cannot be sure that what you find to be true of the sample is also true of the population.

There are two types of samples that are commonly used that often turn out not to be representative of the population: convenience samples, and self-selected samples. Generalizations about the populations made based on sampling from these two types of populations are often faulty.

Convenience samples are groups to which analysts have easy access. They might be entities with which analysts have prior relationships, entities that happen to be in close contact to analysts at the right time, or entities that are particularly visible. Convenience sample are used because they are easy to access, not necessarily because they constitute a cross-section of the population of interest. Any time convenience samples are used to draw conclusions about some population, you should be wary of the validity of the conclusions.

Self-selected samples are composed of entities that volunteer to be part of a study. Volunteers often come forward because they have their own agenda to promote, rather than to simply provide information for the study at issue. People will volunteer, for example, if they have particularly positive or particularly negative feelings or experiences with the issue at hand and they want others to know about them. Again, you should be wary of conclusions drawn about a population based on samples composed of self-selected entitles.

In the Dartmouth study, UCLA was chosen as a sample hospital to be used as the basis for measuring waste precisely because the hospital has a reputation for being aggressive in providing care for patients. At the same time, the Mayo Clinic was chosen as a point of comparison for UCLA precisely because it is known for being conscientious about providing cost-effective care. As such, any comparisons should have been expected to provide large differences. Neither hospital is representative of the average hospital in the US, so any conclusions drawn from a comparison of what goes on at the two cannot be used as a valid estimate of measures at the national level.