Monday, 10 October 2016
The serious trust deficiency afflicting medical advice and what to do about it
By Jerome Burne
Trust me I’m a doctor has become a knowing, cynical catch-phrase but the underlying truth is that we do need to trust our doctors, not only because trusted doctors exert a beneficial healing effect but also because we are entrusting them with something precious – our health.
Several recent events however suggest that we would do well to keep our cynical streak on high alert when visiting. Unquestioning trust can have dangerous consequences, as some patients taking statins have found along with those with ME/CFS.
A reasonable summary of a recent HIUK post about statins would be: ‘You can’t trust claims about statins because they are based on secret data’. While a summary of last week’s post might be: ‘You can’t trust claims about the benefits of psychological treatment for ME/CFS because they have been wildly exaggerated’. Not only that but the authorities involved fought hard and expensively to prevent the facts emerging.
I do think there may be a way out of this long-running conflict between patients and authorities which I will come to later but what happened initially in the wake of these two posts caused my trustometer to plunge even lower.
Impenetrable mathematical wizardry
First was a post on another blog which explained that a trial of the great and good Mediterranean diet had arrived at its impressive conclusion with the aid of the sort of impenetrable statistical workings that has given a shine to statins. I knew that a similar form of mathematical wizardry had created things called ‘collateralised debt obligations’ that set off the banking crash but I’d never actually seen one at work.
Now I have and this is what it looks like:
We calculated PAF [Population Attributable Fraction] based on the formula of rate difference: PAF = (I0 − Ii)/I0, equivalent to I0 − HR × dMDS/I0, where HR was estimated continuously with adjustment for potential confounders as aforementioned, I0 is observed incidence per 10,000 person-years, and Ii represents a hypothetical, ideal incidence if the population achieved high MDS (95th percentile) (dMDS=MDSideal – MDSobserved). The CI of the PAF was derived from bootstrapping to estimate HR and PAF iteratively (n resampling = 100, after confirming no difference in results between n = 100 and 1000).”
Remarkable isn’t it? This one was found in a large nutrition trial published last week extolling the benefits of the Mediterranean diet (the comment originates from the blog). I’m prepared to bet that no more than a handful of BMJ or Lancet readers could assess whether the answer it produced was right or wrong or how the conclusion was arrived at. You just have to take it on trust.
It might be described as a “statistical McGuffin” (A McGuffin is a term used in movie circles to describe something that propels the plot without having any further function.) Its purpose in the trial was to produce a result that would generate the impressive headline: ‘Mediterranean Diet could prevent 20,000 heart disease deaths per year.’
What you need to know about McGufIins
I’m grateful to the indefatigable Dr Zoe Harcombe, a hugely well-informed nutritionist with a clear grip on bio-medical statistics, for this explanation. She cheerfully admitted she didn’t understand the workings of this particular McGuffin, but that didn’t matter. She knew what its underlying assumptions were and what it was designed to do.
Its purpose was to translate the key finding of the research –that following the diet closely made you less likely to die from a heart attack – into the number of lives that would save in the UK as a whole.
The problem isn’t with the mathematical processes but, as with the financial statistical McGuffins, with the assumptions it was set to work on. Firstly, that the result of a study of a 24,000 people with an average age of 59 would be directly applicable to people of the average age in the UK which is 40. Secondly that 12.5% of the total number of heart attack deaths could be prevented with the diet. As Dr Harcombe explains there are good reasons why both are unfounded.
Without independent and informed experts like her, national dietary policy and drug guidelines can presented as objective and scientific when they are actually based on questionable assumptions.
Patients take unusual steps
But there is no need to reach for a McGuffin to give statistics a highly misleading spin. A quick change to the criteria for success can also do it effectively. This move was central to last week’s blog about the remarkable events in the world of ME/CFS where the issue of how best to treat patients is bitterly contested.
For those starting here the story centred round a standard clinical trial that, five years ago, had found that the official treatment –psychotherapy and graded exercise – was effective in improving many patients’ lives and helping a significant number to recover.
When it was published something unusual happened. Not only did a considerable number of patients not believe the results but they wrote letters and compiled reports setting out their reasons and demanding to see the original data. Last month five years of campaigning produced two totally unexpected results. The psychologists who ran the trial were forced to release the data and when it was analysed the patients concerns appeared justified.
What triggered a further drop in my trustometer was the response to this bombshell. Serious questions had been raised about the accuracy of the data underpinning a NICE approved treatment which affects around 200,000 patients. And nothing happened. No TV coverage, no headlines just an uneasy silence like a car crash in slow motion.
The curve ball here was that the re-analysis had been driven by the patients rather than the regulator. That’s unheard of, although there is no reason why the results shouldn’t be correct. In fact there is obviously a case for involving patients in any such re-assessment. They are certainly stakeholders in the business here and I describe a way it might happen below.
Nothing to do with me guv.
The BMJ covered this dramatic story in a misleadingly low-key way. It didn’t spell out the long campaign to get the data or convey how damning was the dismissal of the reasons given for not releasing the data. This was especially curious because the journal has been campaigning for the release of hidden statin data for at least two years. Surely a patient’s successful use of Freedom of Information could have been hailed as a significant victory and maybe a strategy to try?
The Guardian was the only national paper to cover it but it didn’t even attempt to set out the details or provide any context. Instead of dishing out the forensic treatment given to wealthy tax dodgers or regulation busting banks, it handed over half a page to the lead author of the disputed trial to explain how hard he worked to help patients – most likely true –but which he followed with a curious use of the passive tense:
‘There are claims of foul play,’ he wrote ‘with issues over freedom of information and sharing of trial data.’ The fact that he is one of the central figures in these issues doesn’t really come across. It would certainly leave the casual reader no wiser as to what had been happening.
What this limited and downbeat response highlighted was that once a treatment is out there approved and being used, it is virtually impossible to get any kind of reassessment until people start dying as with Vioxx. And that even evidence based critiques are unwelcome.
Cholesterol data fiddling goes back a long way
Resisting any kind of challenge and dismissing critics is also the way the current statin debate has been handled. What I hadn’t appreciated until I started reading a remarkable new book this week, was just how far back the fiddling of statin data and refusal to engage with critics goes.
It has one of those does-what-it-says-on-the-tin titles: ‘Fat And Cholesterol Don’t Cause Heart Attacks And Statins Are Not The Solution’ and contains contributions from 22 leading clinical and academic critics of the widespread use of these drugs.
It builds a detailed case for the way evidence has long been ignored or twisted, detailing, for example, the early shenanigans around the long running Framingham study investigating the links between diet and heart disease. One influential paper claimed that for every 1% reduction in cholesterol there was a 2% reduction in the risk of heart disease. This was seriously misleading. Years later when the study was re-examined it emerged that the study had actually found that for every 1% drop in cholesterol there was an 11% increase in coronary and other causes of death.
The book details other long, large scale trials such as MRFIT (Multiple Risk Factor Intervention Trial) and WHI (Women’s Health Initiative) both of which found that a drop in cholesterol had no effect on CHD. There’s a description of a randomised controlled trial of a cholesterol lowering drug called Cholestryramine, which claimed to have found the same 1% reduction leading to a 2% reduction in CHD events. Again, a subsequent review found no difference between the placebo and drug groups.
Critics warned off having a meeting
But the failed trials were not enough to counter the major PR offensives against cholesterol and the facts about the unreliable ones came out later. Persistent and informed critics were ruthlessly treated. One of these was George Mann, a professor of Medicine and Biochemistry who had done research in 1960 which failed to find a link between lower cholesterol and heart disease rates, which was never published at the time.
When he attempted to organise a conference to discuss flaws in the cholesterol hypothesis only 12 delegates turned up. The rest had been either fooled by false announcements saying it had been cancelled or warned their funding would be cancelled if they attended.
A big difference between then and now is the internet, which makes it far harder to stifle criticism as both ‘Statins Are Not The Solution’ and the ME/CFS re-analysis shows. The opportunities it throws up are leading to more formal involvement by cancer patients, for example, in their treatment under the mantra: ‘Nothing about us without us’. This could be expanded to other diseases.
Patient representatives have long had a place on committees run by companies and organisations but the latest idea, dubbed ‘evidence based advocacy’, is that they should be more involved, informed and proactive. The sorts of things being suggested include ‘conducting online surveys though software such as Survey Monkey, gathering opinions at conferences and conducting Facebook polls.’ Might some such formal feedback system have made action on the demands of ME/CFS patient less of a frustrating battle for both sides?
One organisation called Europa Donna, which involves breast cancer patients, is now being asked to serve on the committees that run clinical trials. Since these can involve highly technical decisions, Europa Donna is providing training, particularly in the research field. ‘The result,’ a spokesperson is quoted as saying ‘is they can do an effective job and not just rubber-stamp what is handed to them by the scientific investigators.’
Is it wildly optimistic to imagine patients sitting across a table from researchers discussing the release of the full data on statins- maybe some had been in the original trials – or making informed critiques of the trial of a new treatment for ME/CFS. Certainly something needs to change.