The UK’s Office for National Statistics used to knock on doors and ask people if they were employed, as a traditional way of compiling the country’s labor market data.

But when Covid-19 forced social distancing in 2020, door-knocking was out. At one point during the pandemic, the survey was reliant on how many phone numbers it could find for a representative sample — and also on people actually answering their mobile phones when an unknown number called. Only a quarter of Britons say they answer such calls, with most wary of scams. Fewer that half of UK households have a landline, and those numbers can be difficult to find.

Response rates plunged. By 2023, the data had become so patchy the ONS was forced to suspend its unemployment reading as it desperately sought to repair a key release that helps inform government policy, influences interest rates and drives billions of dollars of investment decisions. An ensuing crisis of confidence in Britain’s data ended with the resignation of its national statistician, Ian Diamond, last month, following fierce criticism from politicians and senior central bankers.

Getting households to pick up the phone is just one of many obstacles facing the world’s stats geeks, including government agencies tasked with collecting numbers on everything from joblessness to agricultural output. Just as cellphone users swipe away unwanted callers, people screen visitors with video doorbells and an increasing number live in apartment blocks impenetrable to data collectors, compounding a collapse in survey responses. The pandemic exacerbated most of these trends.

That’s left statisticians from Australia to Britain and the US scrambling to update collection methods stuck in the 20th century without introducing new 21st century biases. The solution that agencies and pollsters, from the ONS to the Bureau of Labor Statistics in the US, have zeroed in on: online surveys.

These polls are less costly than telephone or face-to-face surveys, more convenient for respondents, and statistics agencies are using AI to target non-responders. Yet as agencies are finding, the big analogue-to-digital switch in economic data is throwing up new problems for the indicators global investors and policymakers rely on.

“There’s no magic mode,” said Steve MacFeely, chief statistician at the Organisation for Economic Cooperation and Development. “They’re well aware of the risk and are doing it because they have no other choice. It’s the grim reality of official statistics today.”

Barrage of spam

Problems with the new wave of online surveys include fresh biases in the data, confusion over questions that a computer cannot address, gaining trust amid a barrage of spam and fraud in people’s inboxes, and the fact that households may be just as likely to ignore an email with an online survey as a phone call.

“What you gain in coverage/response, you possibly lose in measurement quality or content features,” said Eric Harrison, deputy director of the European Social Survey.

Problems with the UK’s labor market figures serve as a cautionary tale.

Britain’s efforts to create a new online-led labor market survey came after the response rate to its household survey collapsed to below 15%, forcing the ONS to temporarily withdraw its key labor market estimates in late 2023. BOE rate-setters complain it is making it far more difficult to set interest rates, even with the reinstated figures.

A full transition to an online questionnaire in the UK may not happen until 2027 after teething problems in testing. Early results showed the new online survey was producing significantly different readings from the existing survey that relies on telephone calls and knocking on doors, people familiar with the situation said.

One potential issue is that respondents aren’t able to ask an interviewer to clarify a question they don’t understand. The ONS is looking at an AI chatbot that could help.

“Moving the survey online has been challenging due to the survey length and complexity meaning many households didn’t finish it,” an ONS spokesperson said. “We’ve tested a shorter version that’s giving improved and more complete responses and this new version will go live in the field next month.”

Another key risk from online surveys is that it creates new unknown biases that statisticians have to counteract. In the UK, the existing labor force survey struggled to get enough young workers to answer. For an online questionnaire, it could be the opposite problem.

“If you were to do a face-to-face LFS survey, and you were to do a telephone or an online labor force survey, you’d get different results from all three,” said MacFeely from the OECD.

In the US, Joanne Hsu, director of consumer surveys at the University of Michigan, believes that being behind the computer screen may also skew how people respond to economic surveys.

The Michigan survey’s closely watched figures on economic sentiment and inflation expectations in the US fully transitioned to a web-only mode in 2024 after seven years of testing the new responses alongside its existing phone-based method. While it helped to double the survey’s reach to nearly 1,500 responses, Hsu noticed that respondents are more likely to give extreme answers on the web — especially for inflation expectations, which recently surged to the highest in three decades.

“People might think twice about giving any sort of extreme value to an interviewer,” she said.

While the US has seen a smaller drop off in responses to many of its official surveys, the rate for the Current Population Survey underpinning labor statistics has fallen by almost 20 percentage points in a decade to 68%. For the Job Openings and Labor Turnover Survey or JOLTS report, a business survey, response rates have tumbled to almost 30%.

Former US Bureau of Labor Statistics commissioner William Beach said that US data could soon be in the same state as Britain’s moribund labor statistics without action to modernize its surveys.

Labor force surveys “are dying, they’re decaying, they’re in very serious trouble,” Beach recently said on Bloomberg’s Odd Lots podcast. “Unless we modernize that survey, we will see a time when we will be like the British, unable to publish portions of it that just don’t have sufficient sample for statistical release.”

Response rates to the Current Population Survey are also under pressure as President Donald Trump’s crackdown on immigration may be discouraging foreign-born workers from participating in the survey. Trump’s election has compounded challenges as his job cuts across the federal workforce put greater pressure on funding and staffing.

The BLS and Census Bureau have been trying for years to mitigate the preexisting challenges by introducing a web-response mode for households to their CPS — which produces the unemployment and participation rates, among other statistics, in the monthly jobs report. Interviews currently are mostly conducted over the phone or in person.

The agencies are barely hanging onto the CPS’s sample size, which the BLS said last year was at risk of getting cut due to budget constraints. Advocacy groups successfully lobbied Congress for additional funds, which keeps the status quo for now, but still lacks necessary additional investment for the web transition. BLS and Census say the internet option will boost response rates, or at least stabilize them.

For statisticians facing pressures on multiple fronts, online surveys seem to be the least-worst option, rather than the cure-all.

Canada’s national statistics body has run an online survey for its labor data since 2015, alongside face-to-face and telephone interviews. It has seen a smaller decline in responses for its mandatory labor market survey than the plunges seen elsewhere and relies on targeting households with a bespoke strategy and contacting them ahead of time to keep response rates high.

“I think in the context of that gradual decline, the electronic questionnaire is an attractive option for busy couples or for busy families,” said Vincent Dale, director general of labor market, education and socioeconomic well-being at Statistics Canada.

Dale said slowly declining response rates in Canada was down “to things like trusting government, changing lifestyles meaning two earner families so less people at home during the day, [and] use of cell phones makes it more difficult to contact people.”

The Australian Bureau of Statistics is also taking a multi-prong approach after noting declining response rates in both business and household surveys. The ABS is modernizing its data acquisition processes to make it easier for people to respond quickly, including mobile-friendly surveys, and investing in “alternatives to traditional surveys.”

“Without continued modernization of data acquisition, including contemporary online surveys and increased use of administrative data, we would see rising costs and falling response rates with a corresponding drop in quality of survey estimates,” an ABS spokesperson said. “Lower quality data becomes a barrier to making well-informed decisions.”

--With assistance from Irina Anghel.