by Kevin Schofield

This weekend’s reading is a document outlining the results of a recent poll commissioned by the Downtown Seattle Association (DSA) and the Seattle Metropolitan Chamber of Commerce. Both organizations are business associations that count many downtown-based businesses as members, and both work to influence the city’s political outcomes on behalf of their members. This survey – and the fact that they are publishing the results – appears to be an attempt to push the complex topic of public safety in a certain direction during an important election year – evidenced by the fact that they chose to ask “likely voters. “

DSA and the Chamber hired a local political polling firm, EMC, to conduct the survey. They showed it in mid-January, and conducted interviews over the phone and online. In total, they collected 500 survey responses, giving them a margin of error of plus or minus 4.4 percentage points: When looking at the total group’s responses to a given question, the “real” result for all likely voters in Seattle could be up. to 4.4 points higher, or 4.4 points lower.

They found that voters remain pessimistic about the direction of the city — 60% think it’s on the wrong track, and just 36% think it’s headed in the right direction — but those numbers have improved significantly from over a year ago. Mayor Harrell is viewed favorably by two-thirds of voters, Seattle police officers by slightly fewer (around 62%), and the Seattle Police Department (SPD) by just over half.

Homelessness is by far the biggest public safety concern among voters, cited by 52% of voters, followed by crime (23%) and drugs (19%).

Over 60% of respondents supported hiring new police officers, increasing funding for the SPD, and using signing bonuses and other incentives to increase hiring. Just over half believed that the SPD has made “significant progress with the police reform”. At the same time, there was overwhelming support for the creation of a new civilian public security force to respond to lower-priority nonviolent calls.

We should always treat survey results with skepticism until we’ve had a chance to question them; it’s easy to design a survey to encourage specific responses and implement it in ways that skew the response in the direction those paying for the survey want to see. That doesn’t necessarily mean that all surveys, or this one specifically, are biased; but it is important for us to look for signs that perhaps the results should not be trusted so much. To that end, let’s pull this one apart.

“Likely Voters”

The fact that EMC surveyed “likely voters” means several things. First, as mentioned above, they are trying to send a public message about what voters are thinking when they go to the polls this year. Second, “likely voters” are a different demographic than the general Seattle population, so we need to be clear up front that these poll results don’t necessarily tell us how all of Seattle thinks about public safety. The pool of likely voters tends to skew older and whiter than Seattle as a whole, an important demographic difference in a city that is younger and more ethnically diverse than most American cities.

‘Mixed Mode’ Surveys and ‘Response Bias’

The fact that the survey was done “mixed mode” – using both telephone and online interviews – is mostly a good thing, as is that they reached out to both landlines and mobile phones. As recently as five years ago, many polls called only landlines, eliminating the views of a large and growing young population who had only cell phones; now any poll that only surveyed landlines will be rejected immediately. Online polls often reach a different population than those doing telephone surveys, but studies have also shown that people tend to be more outspoken and express stronger views in online polls than they would when talking to a human interviewer – so including online opinion polls not only change the audience, but also the kind of response you get. We also need to be aware of “response bias”: Those who will take the time to answer a survey tend to be those with stronger opinions, so surveys almost always overestimate the strength of people’s beliefs.

Number of respondents

Having a total of 500 survey responses is at the low end of the acceptable range for statistical purposes. As of last November, there were 482,789 registered voters in Seattle, of whom 333,912 cast a ballot in the November 2022 election, and 267,414 in the November 2021 election. If about 250,000 Seattle voters are “likely” to vote in November, then examined EMC 1 in 500 of them.

It is always better to have more responses, but it takes more time and costs more money to collect them, so organizations that commission a survey often aim for the minimum number that will give them a statistically significant result. However, this not only increases the margin of error, but also makes response bias a bigger problem: the fewer respondents, the easier it is for a handful of people with strong opinions to skew the results.

Demographics

This is especially problematic if we try to look at piecemeal results for specific demographics: age, race/ethnicity, political party affiliation, and so on. For some of them, the sheer numbers are so small that the results are meaningless. For example, only 16 respondents listed their gender as non-binary; there are not enough people for us to draw any statistical conclusions about how non-binary Seattleites would answer the questions. But not distinguishing them can also be problematic: the EMC survey divides race/ethnicity into just two groups: white or BIPOC. But we know from many other polls that Asian, black, and Latino Seattleites may have different views on public safety and policy issues based on their lived experiences, and group them together (probably because the individual numbers are too small—only 100 people are in the BIPOC category) masks these important differences.

What is a “Cross-Tab”?

And this brings us to one of the most important questions to ask whenever someone gives you the poll results: “Can I see the crosstabs?” A “crosstab” is a table of the survey results broken down for each question, compared to the demographic question results. This allows us to double-check that survey respondents match the population as a whole, but it also allows us to see how different demographic groups answered specific questions (assuming the total number of people in that demographic group is large enough to be statistically meaningful). Plus, it lets us see what the pollster chose not to include in their report: statistics that might contradict a narrative they wanted to tell. You can see the cross-tabulations for the DAS/Chamber vote here.

Looking at the gender and age statistics, it appears that survey respondents generally match the population of “likely voters,” who we know are older and whiter than Seattle’s population as a whole. Of the survey respondents, 71% were white, and 48% were 50 or older; 55% were Democrats, 4% were Republicans and 5% were Socialists.

An anomaly in the data

One odd thing that emerges from the cross-tabulations, however, is that 45 people, nearly 10% of survey respondents, refused to provide race/ethnicity—a high number for a poll like this. Among the 45, half of them also refused to state their gender. We could dismiss this as a fluke, except that this group of 45 expressed extremely strong and consistently pro-police views on all of the related surveys, as well as the strongest views that the city is less safe than two years ago and that the city is still on the wrong track. They were also somewhat less concerned about homelessness as a public safety issue and much more concerned about drugs and the lack of prosecution of alleged criminals. Almost 60% still supported a civilian public security department, although a quarter strongly opposed it; 22% said they were Democrats, 31% Independents, 4% Republicans, none were Socialists, and a whopping 44% refused to give their political affiliation at all. All that said, 45 people is a small group in a relatively small poll: we can’t tell if this is response bias, sampling error, or something else. But the idea that maybe 10% of Seattle’s likely voters would give this kind of highly consistent “outlier” response means something — even if we don’t yet know what. And it was certainly a large enough group and strong enough response to affect the overall results.

All in all, there is nothing in the survey composition or cross-tabulations to suggest that we should ignore or dismiss the results – although the small survey size argues against paying much attention beyond the top-level numbers. We must also remind ourselves that this is a political poll, aimed at likely voters and intended to influence officials and candidates who are concerned about how voters will respond in this year’s election; it does not represent the general view of all Seattle athletes (nor does it claim to). There was also a survey about public safety, and while the DSA and the Chamber claim public safety is the most important issue for voters, this survey did not ask that question. We must be careful about assumptions that public safety will necessarily be the motivating factor in voters’ decisions in November.

Seattle Public Safety: Likely Voters in November 2023, January 2023
Cross tables for surveys

Kevin Schofield is a freelance writer and publisher Seattle Paper Trail. Previously he worked for Microsoft, published Seattle City Council Insighthosted “Seattle News, Views and Brews” podcast, and raised two daughters as a single father. He serves on the board of the Woodland Park Zoo, where he also volunteers.

📸 Featured image by VideoFlow/Shutterstock.com.

Before you move on to the next story …
The South Seattle Emerald is brought to you by Rainmakers. Rainmakers give recurring gifts at any amount. With over 1,000 Rainmakers, the Emerald is truly community-driven local media. Help us keep BIPOC-led media free and accessible. 
 
If just half of our readers signed up to give $6 a month, we wouldn't have to fundraise for the rest of the year. Small amounts make a difference. 
 
We cannot do this work without you. Become a Rainmaker today!