Why anecdotes aren't strong evidence when it comes to quitting smoking

In the early 1990s, I was the guest of the local health service in Broken Hill, New South Wales, during a national week promoting quitting smoking. I went on the local radio and the host invited ex-smokers to call in and talk about how they had quit.

Consistent with everything we know about the method most ex-smokers used at their final successful attempt to quit, many callers talked about quitting cold turkey (unassisted).

A series of papers about the unassisted quitting process by Andrea Smith and colleagues at the University of Sydney gives more details.

I recall the last caller wanting to tell the world that all the ways people had been discussing were all very well. But no one had mentioned the very best method of all. Could I guess it? There was an auspicious silence and our caller then extolled the importance of letting Jesus Christ into his life. Jesus could stop you smoking. Everyone should know this, he said.

Why miracles are a smoke screen

Across a 40 year career, I’ve seen countless testimonies supporting miracle smoking cures. These range from fairground hypnotists, acupuncture, herbal remedies, dipping your cigarettes in magic potions before you smoke them, paying someone to point a “laser” at special parts of your body while they extract $450 from your wallet, Alcoholics Anonymous-style smoking temptation story sharing, thinly disguised religious pitches from church-based health groups talking about “higher powers”, mantras to recite when tempted, and various offerings from the pharmaceutical industry.

The Cochrane Collaboration has systematically reviewed the evidence for 78 different interventions for quitting smoking.

The popularity of many quit methods is closely related to the marketing and promotional budgets of those standing to profit from their widespread use.

Quit smoking aids, including for the prescription drug Champix, have been heavily promoted.

Champix (varenicline) (a prescription drug) and nicotine replacement therapy (NRT) have had the longest, most lavishly supported innings, with NRT being advertised in prime-time media for many years.

But after some 30 plus years of NRT being promoted, its record is frankly underwhelming.

Buying NRT over the counter and trying to quit without additional professional support has a statistically significantly lower rate of success than trying to quit unassisted. Over-the-counter use of NRT promises about a 7% long-term success rate (in other words, a 93% failure rate).

With professional support, NRT fares better but very few smokers access such support, so the population impact is limited. For example, less than 4% of smokers ever call the Quitline.

How about e-cigarettes?

On July 6, submissions closed on a House of Representatives committee looking at the regulation of e-cigarettes, The 332 submissions were swamped by many individuals’ personal anecdotes explaining e-cigarettes have been a miracle.

People write passionately about having tried many other ways of stopping unsuccessfully. Some make compelling statements about their health rapidly improving. They want to spread their good news and encourage others to try to do what they have done. Their stories are very real: we’ve all met someone who knows someone who quit by vaping.

However, those who have tried and failed to quit using e-cigarettes are far less likely to be as enthusiastic and evangelical. Just as someone who tried to lose weight and failed is highly unlikely to want to take the time to write a political submission about their failure, so too is it unlikely a smoker who tried vaping, kept smoking and then discarded e-cigarette use, would bother to write.

And significantly, over one in four of Australians who smoke daily have either used or experimented with e-cigarettes and then abandoned them (see table 9).

Beware self-selection bias

Such positive personal testimonies represent self-selection bias about success and cannot be given credibility when it comes to making generalisations about the success or otherwise of any cessation method.

We would not count as strong evidence the heartfelt testimonies of those swearing by any given method.

This person swears ‘laser therapy’ has helped him quit smoking. But is this strong evidence?

For example, we’d have immediate questions about the person in the above video swearing by the success of “laser therapy”, when the Cochrane Collaboration has presumably found insufficient research about it to even publish a review. Claims about e-cigarettes need to be held to the same standard.

So, which evidence should we trust?

The strongest evidence about whether any given method for quitting smoking “works” comes from randomised controlled trials (RCTs) and from “real world” cohort studies where groups of smokers and ex-smokers are followed over time.

In contrast to the picture from testimonies, a 2017 meta-analysis of both of these types of studies for e-cigarettes (three RCTs and nine cohort studies) concluded:

There is very limited evidence regarding the impact of [e-cigarettes] on tobacco smoking cessation, reduction or adverse effects: data from RCTs are of low certainty and observational studies of very low certainty. The limitations of the cohort studies led us to a rating of very low-certainty evidence from which no credible inferences can be drawn.

Yet there are claims 6.1 million Europeans have quit by vaping. Such “big” numbers do not withstand scrutiny.

The 6.1 million number comes from a cross-sectional “snap-shot” survey where ex-smokers reported they used to smoke, then used e-cigarettes and now don’t smoke. Were it only that simple. This critique makes the key point that the survey questions would have allowed those who quit for only a short period to say they had stopped, when relapse is a major phenomenon and demands a longer-term view.

The critics also asked:

… how many of those who claim that they have stopped with the aid of e-cigarettes would have stopped anyway, and how many of those who used an e-cigarette but failed to stop would have stopped had they used another method?

How about smokers who quit (and relapse), often several times?

Researchers on a study Smoking in England, published a step-by-step estimation of the number of English smokers whose smoking cessation in 2014 could be attributed to e-cigarettes.

They took into account factors like an estimated 70% relapse back to smoking and the fact that e-cigarettes displace success rates that would have occurred via other methods (which fewer peoples use with the rise of e-cigarettes).

The group estimated 16,000 smokers quit permanently in a population of 8.46 million adult smokers. That’s about 0.19% shaved off England’s smoking population in just a year by e-cigarettes – just one in 529 smokers in a year quitting for good.

As the study leader Professor Robert West, also editor-in-chief of the journal Addiction, put it:

[This widespread use of e-cigarettes] raises an interesting question for us: If they were this game changer, if they were going to be – have this massive effect on everyone switching to e-cigarettes and stopping smoking we might have expected to see a bigger effect than we have seen so far which has actually been relatively small.

For perspective, in Australia where the prevalence of regular vaping remains marginal (only 1.5% of Australia’s daily smokers and 0.8% of ex-smokers use e-cigarettes daily – see table 9), smoking prevalence in those aged 14+ has declined over the 10 years between 2007-2016 (from 19.4% to 14.9%), an average of 0.45% a year. This decline reflects both smokers quitting and dying and reductions in uptake.

Smoking prevalence has indeed fallen fast in England in recent years while e-cigarette use has increased. But it is simplistic to assume this is the only explanation needed. As Robert West’s group has emphasised in this presentation (see slide 29):

The trajectories for smoking prevalence and quit attempts differ from that of prevalence of use of e-cigarettes.

In fact, the reduction has occurred concurrent with a comprehensive program to reduce smoking. During this time there has been a spectacular decline in tobacco affordability, with cigarettes being 27% less affordable in 2016 than in 2006. The decline in affordability tracks with the declining smoking rate almost exactly.