INSIGHTS BLOG > Best Practices in Medicine Should Rely on Providers' Knowledge, Skills and Experience, Not Just Studies

In 2005, a physician-scientist research pioneer, John Ioannides, published what has come to be a widely circulated paper, “Why Most Published Research Findings Are False.” The replication crisis we’re having in science embodies the concern voiced by Mr. Ioannides. Yet, despite much evidence that so many studies are not valid, scientific professionals continue to rely almost exclusively on study results when deciding on best practices.

So many studies are flawed. As a simple example, please take my survey by answering the following question:

Over the past 12 months, how many times have you visited a doctor?

Take as much time as you need to answer the question…

Got your answer?

Okay, now let me ask you a few questions about the number of visits you just “reported” for my study.

First, to answer my question, did you just think back in your mind, or did you actually check your records? Most people will probably come up with a “good estimate” based on what they can quickly recall. Relatively few people will make the effort to reference records to help them come up with a more accurate estimate.

As for what we remember, researchers continue to discover new ways in which our memories paint an inaccurate portrait of “the truth” (to the extent the truth exists). For example, the telescoping effect is a common cognitive bias affecting our memory, where we tend “to displace recent events backward in time and remote events forward in time, so that recent events appear more remote, and remote events, more recent.” The telescoping effect is just one of many different cognitive biases – Wikipedia lists 42 different cognitive biases that affect our memories – any one of which may cause your reported number of visits to the doctor over the past 12 months to be more or less than the “true” number.

Second, what types of providers did you include in your estimate? Did you include any visits to a dentist, nurse practitioner, therapist, optician, optometrist, pharmacist, herbalist, or other allopathic provider? Different people will have different interpretations of what’s included in the category “doctor.”

Third, if you visited the same “doctor” (however you define it) more than once (perhaps for a follow-up), did you count that as one visit or two? Some people will count multiple visits to the same doctor as a single visit – that is, what they consider “a visit” is more figurative – whereas others will interpret that phrase more literally.

So then what’s the “actual” number of visits to the doctor I’m looking for? How do I define “doctor”? How do I define a “visit”? It depends: what am I trying to measure?

Now that I’ve posed some of the problems with the information I would have collected, how much faith would you put in any findings I would have reported based on analyses of such data? Probably not much, and rightfully so.

Perhaps, that’s a little simplistic. Clinical studies are generally more precise than my little survey study, aren’t they? Perhaps, but not necessarily. Many clinical trials study patient-reported outcomes, such as depression, anxiety, or fatigue. These types of reported outcomes are subjective. Clinical studies are also subject to bias from many different possible sources, most notably selection bias.

Well, then, that’s why we use randomized control trials (RCTs). RCTs are studies that are considered to be the gold standard for establishing “evidence-based” healthcare practices. RCTs control for all the problems potentially associated with other clinical trials, don’t they? Not necessarily. While randomizing a sample and providing a control group eliminate some potential forms of bias, other errors and forms of bias may still be present. In other words, even RCTs may not generate “the truth.”

Take, for example, the case of medical cannabis. The effects of a dose of cannabis on a particular individual will vary, depending on

• Characteristics of the individual, such as genetics, metabolism, and age
• Characteristics of the individual’s history with cannabis, such as history of use and tolerance
• Characteristics of the particular sample of cannabis consumed, including
  • Origin of sample: whole plant extract, isolate, synthesized compound, etc.
  • Form of use: flower, tincture, edible, etc.
  • Profile of contents in sample: profile of cannabinoids, ratio of cannabinoids, profile of terpenes, etc.
  • Size of dose: 5 mg, 25 mg, etc.
• Setting in which the cannabis was consumed: isolated in a sterile lab room, relaxing with others in a comfortable room, etc.

In any RCT, the form, profile, and dose of cannabis is generally standardized across patients. It follows from the bullet points above that the effects of that standardized dose will vary widely across different users. So then it’s quite possible that the standardized dosing will not correspond to the dosing specifications that would be effective for different patient populations. As such, RCTs could easily suggest that cannabis was effective, if the form, dose, and setting matched those appropriate for the group of patients studied. On the other hand, the very same RCT could easily suggest just the opposite. The study could very well determine that cannabis is not helpful for the patients studied, if the form, dose, or setting used were not appropriate for the particular group of patients examined.

One can probably find flaws in just about any study. Of course, having a flaw doesn’t necessarily invalidate a study. However, we must take study findings in their proper perspective. We cannot simply accept the findings of any RCT because it’s the gold standard or reject the findings of another study for not being an RCT. Medicine is not about “hard science.” It’s about people. People are complex. They don’t fit nicely into well-defined boxes.

Yes, studies can provide us with information. However, we shouldn’t depend solely on studies to find “the truth,” that is, what’s best in a particular situation or for a particular patient. Rather, we should depend on experts – such as doctors – to do what they have been trained to do: rely on knowledge, skills, and experience to deliver personalized medicine, to help patients find the best treatments that meet their particular needs.