‘Disbelieving and hostile’ – that’s how Herbert Benson describes the reaction of fellow cardiologists at Harvard Medical School in the early 1970s when they learned he was studying the physiological effects of transcendental meditation. They thought he’d sold out to the hippies. ‘I had to conduct two careers at that time,’ the 80-year-old told me over the phone from Boston. ‘One as a cardiologist and the other as my crazy thing.’ At one point there was a real possibility he could be thrown out of Harvard.

Thankfully, he held onto his job and during the long, distinguished career that followed – which has included a string of groundbreaking papers published in mainstream science journals – Benson has witnessed dramatic changes in physicians’ attitudes to meditation.

Emboldened by solid clinical research from scientists like Benson, doctors are now recommending tailored meditation programmes for preventing relapse in major depression, alleviating chronic pain and anxiety, and lowering blood pressure.

Far from becoming a laughing stock, these doctors are applauded by their colleagues and patients for providing an alternative (or supplement) to regular drug treatments that are free of side effects.

Benson’s ‘crazy thing’ has gone mainstream. In 2013, for example, the American Heart Association approved the use of a range of meditation techniques as a complement to more conventional treatments for lowering blood pressure. ‘First, there was hostility,’ said Benson, ‘but as the science has built up and as the side effects of the drugs have become more obvious, more and more people are looking for ways of lowering blood pressure that is non-pharmacologic.’

In his early work, Benson identified the body’s ‘relaxation response’ which is triggered whenever we break the train of our ordinary, mind-wandering thoughts by deliberately focusing our attention on something, whether it’s a mantra in transcendental meditation (TM), the breath in shamatha meditation, movement in yoga or repetitive prayer in Christian contemplative practice.

The relaxation response is the physiological flip side of the better-known ‘fight or flight’ response, which originally evolved to prepare our ancestors for mortal combat or to run for their lives. It has the polar opposite effect on the body, slowing heart rate, lowering blood pressure, reducing breathing and metabolic rate, and creating a sense of mental calm.

Research by Benson and others suggests that teaching patients how to evoke the relaxation response at will can help alleviate a whole range of conditions that are exacerbated by chronic stress, including hypertension, cardiac arrhythmia, irritable bowel syndrome, premenstrual syndrome and headaches.

When I interviewed Benson for Siddhartha’s Brain – my book about the science of meditation – he was keen to emphasise that over thousands of years people from all religions and cultures have found their own ways to evoke the relaxation response. In the past two decades, however, a related meditation practice with distinctively Buddhist roots – mindfulness – has made huge inroads into mainstream medicine.

In the largest, most definitive clinical trial to date, published in The Lancet last year, a form of the practice known as mindfulness-based cognitive therapy (MBCT) was revealed to be as effective as antidepressants for preventing relapse in people who have been through three or more episodes of major depression.

Remarkably, MBCT was found to be even more effective than the drugs among those patients who had experienced abusive childhoods. In recent years, similar mindfulness programmes have shown promise for helping people quit smoking and beat drug addiction.

What these treatment programmes have in common are exercises that evoke the relaxation response.  In addition, however, they also emphasise the importance of mindful acceptance of unpleasant emotions and sensations such as low mood, pain or cravings, without judgement. Against all our expectations, this mental attitude of openness and acceptance has the effect of reducing our suffering.

Some 2,500 years ago, this was the unique insight shared with the world by an Indian ascetic called Siddhartha Gautama – also known as the Buddha. But it reflects a universal truth about how the brain works which is now being tested by psychologists, and by neuroscientists using cutting-edge brain-scanning technologies such as magnetic resonance imaging.

What they’re finding is that focusing on the bodily sensation of pain or emotional discomfort, rather than indulging in thoughts such as ‘This is awful, it’s never going to end’, or ‘I’m a failure, it’s all my fault’, can actually make us feel a whole lot better.

We’re lucky to be living in an age where we can not only benefit from the wonders of modern medicine but also draw upon ancient wisdom about how to reduce suffering – wisdom that is being validated by science.

In the course of my research for the book, I was told again and again by clinicians and scientists that they initially risked career suicide by investigating the potential medical benefits of meditation. We owe a huge debt of gratitude to them for putting their careers on the line to discover the truth.

James Kingsland is a former science editor at the Guardian. He also commissioned and edited the science desk’s Notes & Theories blog.