Of all the symptoms with which illness acquaints us, pain is famously the most difficult to quantify. Both a somatic and psychic experience, pain is a chameleon and stubbornly private. As Virginia Woolf observes in her essay On Being Ill, how can prose capture the nature of “this monster, this body, this miracle, its pain without slipping into mysticism”?

If pain is challenging for a writer, it is even more so for a doctor. Is a patient who is presenting with pain in as much discomfort and distress as they say they are? And, if so, how to treat them? And what if they are a malingerer or a hysteric and by alleviating their symptoms you reinforce their psychopathology?

This is the bind that also confronts patients suffering from migraines and poorly understood conditions such as endometriosis. But it is especially true of a condition known as a cluster headache (CH). Affecting one in 1,000 people, CHs typically occur in cycles lasting anywhere from a week to a year and are so excruciating that sufferers have sometimes been driven to take their own lives. Drugs such as sumatriptan can abort sufferers’ symptoms within 15 minutes but the relief is always temporary and repeated use makes the attacks more severe.

However, there is one substance that sufferers say makes a difference: magic mushrooms. Members of Clusterbusters, a patient group that promotes research into psychoactive fungi, say they provide instant relief and, in some cases, lengthy remissions from their symptoms. Moreover, you don’t need to undergo a full-blown trip to enjoy the benefits; repeated low-level doses of psilocybin – the psychoactive ingredient in magic mushrooms – is often sufficient.

To convince researchers to take their claims seriously, Clusterbusters, which has thousands of supporters worldwide, has meticulously documented the experiences of CH patients who have used psilocybin or LSD to treat their condition and in 2006 published 53 case studies in Neurology (22 of 26 psilocybin users reported that the compound aborted attacks). Since then, however, progress has been slow.

As Joanna Kempner, a sociologist at Rutgers University in New Jersey, explains in her compelling new book, there are many reasons for this. One is that there are too few CH sufferers to interest big pharma but too many for the condition to be designated an “orphan disease” – a disease that affects one in 1,500 people. Another is prejudice. All too often sufferers are labelled attention-seekers or stigmatised as drug-takers (psilocybin is classed alongside heroin and cocaine as a prohibited drug both in the UK and US).

The result, according to Kempner, is that “the most innovative psychedelic research on pain is happening almost entirely underground [through alliances] forged by outlaws and academics, working together, sometimes in partnership, sometimes in tension”.

Much of Kempner’s narrative is taken up with documenting these alliances and the negotiations between Clusterbusters and more established advocacy groups such as Maps (Multidisciplinary Association for Psychedelic Studies). In the process, we meet Rick Doblin, the charismatic founder of Maps, and other psychedelic evangelists, including tech bros hoping to cash in on research into substances such as LSD and MDMA to treat more commercially lucrative conditions like depression and post-traumatic stress disorder.

But while these alliances allowed Clusterbusters to get access to prestigious medical schools, Kempner shows how they were conditional on donor funding and quickly turned sour when the association with Clusterbusters and scientists with links to the criminal underworld threatened to jeopardise researchers’ professional reputations.

However, the most intriguing characters are the sufferers themselves – people such as Bob Wold, an engineer and sports coach from Illinois. Now in his 70s, Wold suffered his first CH attack in 1978. After trying dozens of therapies, and enduring four attacks so severe he was hospitalised, Wold turned to the internet for help. It was there that he met an IT specialist from Aberdeen, who had obtained relief from his headaches by taking LSD. Wold began investigating similar reports using magic mushrooms and soon was ordering the fungi via UPS.

Like most members of Clusterbusters, Wold had never experimented with psychedelics before and was amazed to find that low, sub-hallucinogenic doses alleviated his symptoms. “Expect very blue skies… and a smile on your face for four-five hours,” he says.

Precisely how psilocybin interrupts the cycle of headaches is an open question. Brain scans suggest it may help reset the neural pathways involved in the pain cycle. Another theory is that it has something to do with psilocybin’s indole ring, which is similar in shape to the neurotransmitter serotonin (LSD and lysergic acid amide, which can be derived from morning glory seeds, also have indole rings). The good news for sufferers is that after a decision in 2018 by the US Food and Drug Administration to award psilocybin “breakthrough therapy” status, researchers are beginning to take these claims seriously. Indeed, in April, the Yale School of Medicine published the results of the first randomised control study of psilocybin for pain relief. Although it enrolled just 10 CH sufferers, half of those who received psilocybin as opposed to a placebo reported a reduction in attacks.

Yale is now recruiting patients for a larger study. Interestingly, researchers believe the mechanism by which psilocybin alleviates pain is probably unrelated to the psychedelic experience. In theory, that augurs well for future research because it makes it more likely studies will be approved by conservative medical boards. However, the problem for psychedelic evangelists is such research tends to deflect attention from studies into psilocybin for the treatment of depression and mood disorders, which are far more common and potentially more lucrative.

All of which means, the trials and tribulations of CH sufferers are likely to continue.

Mark Honigsbaum is the author of The Pandemic Century: One Hundred Years of Panic, Hysteria and Hubris