• Overview
• Survey Example
• What to Look For in a Survey 

As information becomes more readily available, surveys of all kinds increasingly inform us about the world we live in, that is, about what people believe, what they’ve experienced, and how they feel about all kinds of social, political, professional, economic, medical, and other issues. 

Often times, reported outcomes from different surveys contradict one another. In these cases, or when it comes to any survey for that matter, how do we know whether or not the results are truly accurate and reliable?

Having a basic understanding of how surveys are conducted will go a long way in helping you determine which results are more likely, and which less likely, to be accurate. 

Overview

There are two ways of gathering information about a large group of people, that is, a population. The first way is to question each person in the group, that is, to conduct a census. The other way is to select a sample from the larger population, conduct a survey of the sample, then generalized the information obtained from that sample to the rest of the population. Conducting a survey is generally used because it's less resource intensive (in terms of both time and money). 

However, the problem with using a survey rather than conducting a census is that the information gathered from the survey may not be valid, that is, it may not accurately represent the beliefs or experiences of the larger population from which the sample was drawn, and/or it may not be reliable, that is, it may not generate consistent information when repeated over time. Figure 1 illustrates the difference between the accuracy and reliability of a survey.

Figure 1: Accuracy vs. Reliability

As it turns out, it’s not so easy to design and conduct a survey that generates accurate and reliable information about the larger population to which the results of the survey are applied. The only way to have any confidence that a survey is providing accurate and reliable information is to examine the underlying methodology used to generate the survey results. Unfortunately, I have discovered that in most cases, the methodologies used are not readily available, and in the few cases when they are available, they do not inspire confidence in the accuracy of the reported results. 

Survey Example

Presentation of Information

Before moving on to a discussion of what to look for in a well-conducted survey, let’s start with an example of an actual survey. Nancy Clanton recently published an article in The Atlanta Journal-Constitution titled, “What motivates travel nurses? Mostly money, survey finds.” In the first few sentences of her article, Ms. Clanton reports,^[1]

Travel nurses have seen a decline in pay this year, but money is still the main motivator for many of these health care providers.

According to Nomad Health’s Job Satisfaction Index, which surveyed travel nurses last month, 76% of respondents said they repeatedly seek these positions because of money and the ability to earn enough to meet financial goals.

Although these health care professionals were raking in up $10,000 a week during the pandemic’s peak, that dropped to an average of about $3,000 this year.

Several sentences later, the Ms. Clanton notes,

Almost as important was freedom and flexibility, with 67% listing it as a motivator.

With a body of less than 250 words, the article is short, where the first half focuses on the money issue and the rest describes the other factors that survey respondents reported affecting their decision to stay on the job. The article did not include any description of travel nurses other than their motivations as reported in the survey. The article also failed to include any description of the underlying methodology used to conduct the survey, but it did provide a direct link to the survey itself (the hotlink in the cited text).

I accessed the original survey as well as more information on travel nurses to help provide some context for the snippet of information presented in the article.

Betterteam provides a job description for travel nurses:^[2]

Travel nurses are healthcare workers who travel to patients and assist at healthcare facilities. They perform typical nursing duties and provide care in residential areas, at schools, and other organizations. Travel nurses also assist when hospitals are short-staffed.

Cristal Mackay published an article on the Aya Healthcare website titled, “Digging into the Data: Travel Nurse Demographics,”^[3] which provided more information about travel nurses. Ms. Mackay cites data from the Bureau of Labor Statistics (BLS) indicating that in May 2021, there were 66,790 temp nurses out of a total 3,047,530 registered nurses (RNs). That is, only 2% of all RNs are travel nurses. Why so few? Ms. Mackay indicates, 

Why do travel nurses only constitute a small percentage of registered nurses? One leading cause is that it requires a lifestyle that is not a good fit for everyone. Travel nursing requires individuals to leave their homes and the comforts of their social networks and family behind.

The article further indicates that travel nurses tend to be much younger than the average RN (35 years old vs. 52) and less likely to be married (30% vs. 71%). Also, Ms. Mackay notes that other sources suggest that millennials (the younger nurses’ generation) tend to desire more work-life balance, that is, flexibility in their jobs. Since travel nursing entails working gigs, it provides flexibility (time off) that millennials say they want. Finally, Ms. Mackay notes that travel nurses tend to be more diverse (less Caucasian and less female) than the general RN population.

Moving on to the original survey conducted by Nomad Health,^[4] the survey indicates

The contingent healthcare staffing industry is known to offer higher pay than permanent staff positions and our survey results demonstrate that “money” is a top motivator to start traveling (84%). After “money,” “freedom and flexibility” are a very close second at 71%.

…

Top-5 motivators for seeking a travel position (after first travel assignment)

• • Money, earning enough to meet financial goals: 76%
  • Freedom and flexibility: 67%
  • Sense of adventure: 32%
  • Work life balance: 32%
  • Ability to focus on patient, not the politics: 26%

Finally, Nomad Health includes a brief description of the methodology used to conduct the survey, 

Nomad Health collected 117 total responses in an incentivized, unmoderated five-minute survey fielded between June 21-29, 2023.

The 117 survey respondents – who were paid to participate in the survey – constitute 0.2% of all (66,790) travel nurses.

Discussion

So then if we put all the pieces of this puzzle together, what we have is the following.

• Travel nurses are nurses who travel to different locations to provide temporary nursing services when and where needed.
• Travel nurses are a tiny subset (2%) of all RNs (see Figure 2) who tend to be younger, less likely to be married, and who (by definition) are not averse to travel.
• A survey conducted of travel nurses paid a sample that included 0.2% of all travel nurses (see Figure 3) to provide their opinions about their job satisfaction and “motivations for seeking a travel position.”
• The vast majority of survey respondents indicated that they were attracted to the job for its high pay and also for the freedom and flexibility the job affords.

Figure 2: Travel Nurses as a Percentage of All Registered Nurses

Figure 3: Percentage of Travel Nurses Surveyed

When I initially read the original article, it suggested to me that travel nurses in particular (but perhaps all nurses more generally?) were mostly motivated by money when it came to performing their job. After my initial gut reaction, my next more rational thought was to ask myself two primary questions: 

• Do the sampled travel nurses accurately capture the motivations of all travel nurses?
• Are the motivations of travel nurses (theoretically captured in the survey) likely to be similar to those of the typical RN (not captured in the study)?

After examining the information collectively from the four sources cited above, my conclusions are the following.

First, most people decline to participate in surveys unless they have a specific motivation to do so, namely, they feel strongly for or against the issue under investigation, they’re being compensated (i.e., paid) to participate, and/or they feel pressure to participate (e.g., by a boss or professor). In any of these cases, the information provided by these respondents is unlikely to represent the opinions of the vast majority of people whose opinions were not captured. 

Second, the portion of travel nurses whose opinions were captured in the study is a miniscule portion of all travel nurses. This begs the question: what motivated that 0.2% of participants to respond? The money they were paid to participate probably played a role. If these individuals are indeed motivated by money, then it should come as no surprise that they indicated their primary reason for being a travel nurse – a subset of all nurses that’s known for its higher compensation – is for the money. But does that mean all travel nurses are primarilymotivated by money? 

Let’s consider the other motivations cited in the survey. Other than money, survey respondents are also motivated by freedom and flexibility, sense of adventure, and work-life balance. Given the fact that travel nurses tend to be younger and less committed than other nurses, and that they chose a profession that involves travel, it makes sense that travel nurses, like flight attendants, would be highly motivated by the adventure associated with travel. Travel nursing is also a gig job in the sense that it involves stints, not shifts. This type of employment allows for more freedom and flexibility than standard salaried or shift jobs. I would guess that these other factors may play at least as large a role in motivating the average travel nurse as money. 

And third, since travel nurses constitute such a tiny portion of all nursing jobs, it’s quite likely that the motivations of travel nurses may not at all represent the motivations of the average RN. 

To summarize the situation, the initial article on the motivations of travel nurses indicated that travel nurses are first and foremost motivated by money. I found this shocking and distasteful, since my general impression is that nurses are not primarily “in it for the money.” However, after taking a closer look at the broader context of the situation, it became clear that the article and survey are both high misleading and should simply be ignored. 

What to Look for in a Survey

Now that we have reviewed a specific example, the general case of what to look for in a survey should be easier to understand.

A survey is conducted by drawing a sample from a larger population, then gathering information from the sample, which is then processed and used to generalize to the larger population.

A Well-Defined Population

A good survey must have a well-defined population that is being characterized. If the population is not well-defined, then you can’t select a representative sample.

A Randomly Selected Sample

To have confidence that a sample will accurately represent a larger population, the sample must be randomly selected from that population. Many surveyors rely on convenience samples, such as social or professional acquaintances (professors are notorious for using their students as participants in their studies). However, convenience and other nonrandom samples are more likely to omit representation for some parts of the population, leading to outcomes that don’t accurately capture the populationas a whole that's being studied.

A High Response Rate

The response rate captures the portion of the sample that was selected to be included in the survey that actually ends up providing useful information for the survey. It doesn’t do you any good to have a randomly selected sample if most of the people selected for inclusion decline to participate. 

Generally speaking, higher response rates (from randomly selected samples) are more likely to provide responses that are representative of the population. As previously mentioned, most people decline to participate in surveys unless they have a specific motivation for doing so, which generally leads to biased responses. Higher response rates provide greater assurances that survey responses reflect opinions or experiences that are more closely matched to those of the population.

At the same time, having a low response rate doesn’t automatically doom a study (e.g., small samples can sometimes be reweighted to compensate for non-response), while having a high response rate doesn’t guarantee accuracy (it’s also important that the survey questions elicit accurate and complete information from respondents – more below).

Survey response rates were generally decreasing between the 1950s (when response rates exceeded 90% for surveys published in journals)^[5] and the early 2000s (when rates fell below 50%). However, new techniques were developed thereafter that enabled survey conductors to increase response rates, leading to higher response rates more recently, in the mid 60% – low 70% range.^[6]

A note on internet surveys: Most surveys today are conducted online. In many cases, all members of a specific cohort (e.g., members of a social or professional group) are asked to respond, and information is collected from whomever ends up participating. Internet surveys generally don’t report response rates, simply the number of respondents. However, without a response rate, or any information characterizing the sample, the responses are unlikely to be representative, that is, they are unlikely to indicate anything other than what the respondents themselves believe or have experienced (again, the people who choose to respond usually have a reason for doing so). A more credible survey conductor will compare characteristics of the sample to those of the population, but even in those cases, responses may be biased, so results should be taken with a grain of salt.

Questions and Techniques Designed to Elicit Accurate Responses

It’s essential for surveys to be designed well so as to elicit accurate and complete information from respondents. For example, 

• Terms should be well-defined
• Questions should be carefully worded to be clear and unbiased
• Menu options should be complete, that is, not omit possible responses (e.g., “I don’t know” or “not applicable”)
• Questions should be designed to be quickly and easily answered
• Respondents should not feel threatened or uncomfortable about answering honestly 
• Questions should capture all relevant and potentially confounding information

Information Collation and Processing

Finally, it’s important to understand how information received from respondents is collated and processed. For example, 

• Are responses validated with participants to ensure responses are accurately captured? 
• If responses look wacky, are they included anyway?
• Are any checks for internal consistency conducted? 
• If a respondent doesn’t answer all the questions, is the partial information used, or is the survey discarded? The use of partial responses could bias the information, since the sample makeup is potentially different for each question. 

Outcomes Reported in Context

In a world in which sensationalism sells and most people don’t seem to make it past the headlines, it’s common for survey outcomes be reported without the appropriate context (the travel nurse example is a case in point). But now that we see how difficult it is to generate accurate outcomes, and how easy it is to be misled, it’s important to be sure to investigate methodologies used and verify information is provided in context before relying on reported outcomes.

References 

^[1] Clanton N (2023, Jul 27). What motivates travel nurses? Mostly money, survey finds. The Atlantic Journal Constitution.https://www.ajc.com/pulse/what-motivates-travel-nurses-mostly-money-survey-finds/6D6EAC6FBFE7PPHIKFDX4NHP5A/

^[2] Travel Nurse Job Description. Betterteam. https://www.betterteam.com/travel-nurse-job-description

^[3] Mackay C (2022, May 20). Digging into the Data: Travel Nurse Demographics. Aya Healthcare. https://www.ayahealthcare.com/blog/digging-into-the-data-travel-nurse-demographics/

^[4] Nomad Health Job Satisfaction Index. Nomad Health. https://nomadhealth.com/resources/travel-clinician-index

^[5] Morton S et al (2012). In the 21st Century, what is an acceptable response rate? Australian And New Zealand Journal of Public Health. https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1753-6405.2012.00854.x

^[6] Holtom B et al (2022). Survey response rates: Trends and a validity assessment framework. Human Relations.https://journals.sagepub.com/doi/epub/10.1177/00187267211070769