On a recent visit to West Hampstead I saw a woman with a dog. She was mumbling something, to herself, the dog or no one; I couldn’t tell. She didn’t notice me. As I neared her, about to scuttle past, she snapped out of her reverie. “Sorry, I’m a little crazy,” she said, a small voice. “Who isn’t?” I called back without turning, not really expecting her to hear me, and yet she did and she said, just like that: “Thank you.”

I still think about this woman. Was she embarrassed? Apologetic? Matter-of-fact? Or maybe just vividly, unselfconsciously earnest the way kids can be, and also the crazy? And what did she thank me for?

It’s less, I think, a confusion of my own than a symptom of our time. Consider how we think and talk of mental health – and illness – both here in the UK and elsewhere: it’s shot through with ambivalence.

Over the past few years, for one, we’ve seen a big push from governments, charities and the Royals alike to put mental health on an equal footing with physical health. A slew of campaigns have sprung up to raise awareness of the scale of the problem and to fight the stigma attached to it. Major efforts have been thrown into depersonalising mental disorder, encouraging the afflicted to talk and their friends, families and employers to empathise.

One of the underlying messages of these campaigns is that mental illness is an illness like any other, affecting, we are told, 1 in 4 among us. In a way similar to cancer and diabetes, depression, for instance, comes down to some inherent glitch in the system, a matter of aberrant biology. Just like the cells of diabetics fail at converting glucose into energy, which is what cells normally do, and that leads to buildup of sugar in the blood stream, so the brains of depressives, the theory goes, come with a faulty wiring that inhibits mood regulation.

This runs counter to another, seemingly more insidious idea – that mental breakdown is not a real problem but a mere character flaw, a pretend suffering, an elaborate excuse for people who simply lack the discipline or mental fortitude to suck up and carry on. It is partly this belief that fuels the stigma behind mental health and that public campaigns strive to root out by re-framing mental illness as a biological fact beyond anyone’s personal control. Put bluntly: it’s in your brain and you can’t just snap out of it.

This model – mental distress as a brain disease – has garnered the support of eminent public figures from comedians Ruby Wax and Stephen Fry to Tony Blair’s former spokesman Alastair Campbell. ”Nobody ever gets blamed for getting physical illness”, writes Campbell, “so why on earth do we still talk about depression like it is the fault, and the lifestyle choice, of the depressive?” In a widely praised documentary he made for the BBC, Stephen Fry similarly talks about his bipolar disorder, which, he says, he had to accept as incurable and learn to live with.

Can mental health campaigns actually reinforce the stigma?

On the surface, we’ve made big strides at educating the public and stamping out the stigma. A 2014 report on a Time to Change – the most ambitious mental health campaign in the UK running since 2009 thanks to over £44 million pledged in funding – found increasing tolerance towards people struggling with psychological distress. Over 70% of respondents agreed that mental illness is an illness like any other and should be afforded the best possible care. A similarly high number said that the afflicted should not be held accountable for their condition.

To many such findings hail a dramatic positive shift in the general public’s acceptance of otherness. But some researchers suspect that the figures don’t tell the whole story. Recent evidence, for example, suggests that while mental health campaigns may have changed people’s minds, they have failed to move their hearts.

A number of credible studies have found that explaining mental distress in disease terms does promote tolerance, but only superficially. If you perceive mental illness as a brain disorder, these studies show, you are more likely to say nice things about people with a diagnosis and you tend to judge them less. At the same time, however, you are also more likely to want to avoid interacting with them as well as to rate them more prone to violence, less able to make sound judgements about their lives and less likely to recover. This discrepancy between what people say and what they actually believe has led the researchers to conclude that campaigns of the disease-like-any-other type may, in fact, reinforce the stigma, rather than remove it.

How can we explain this? Sheila Mehta, a psychology professor at Auburn University, conducted a fascinating Milgram-type experiment in which students were paired with undercover research assistants, who had to complete a task while the students gave them feedback through electric shocks. When the students were told that their partner had a brain disease affecting his biochemistry, they blamed him less when he made errors and rated the shocks they gave him as less painful than a control group that hadn’t been given the disease explanation. This outward kindness, however, was at odds with the students’ behavior: the disease model group actually gave harsher shocks, up to 170% more intense, than the control group. Here’s how Mehta explains her findings:

“The disease view may incline us toward an awareness that we ought to have generous feelings toward the afflicted but this view may also induce us to view those with mental disorders as set apart from humanity. (….) Biochemical aberrations make them almost a different species”.

Almost a different species. Put another way, disease like any other is still that – a disease. Despite their best intentions, mental health campaigns that cast psychological conditions as brain diseases may have merely subbed personal responsibility for biological pathology. As a result, we may not blame people when they break down; we will pity them instead. We’ll patronize them. We’ll be compassionate and understanding – but only at an appropriate distance. And so, instead of encouraging inclusion, the disease view of mental illness may be fueling inequality; instead of building a bridge, it may be driving an ever wider wedge between us the healthy and them the sick, us the normal and them the mental.

This is why, I’m beginning to think, the woman with the dog thanked me: because despite her crazy talk, I spoke to her as if she were just another ordinary person going about her ordinary business like the rest of us. As far as I was concerned, she had every right to stand tall and inhabit without apology her place in the order of things.

This, let me qualify, wasn’t tolerance on my part. When I blurted out that everyone is essentially crazy, I was simply putting in layman’s terms what one of the most extensive studies on mental health had already revealed: namely, that for any one of us mental illness may be lurking just around the corner.

A rare disease or a social epidemic: What the science really tells us about mental illness

If I were to ask you how many people you think would have a diagnosable disorder at some point in their life, odds are you’d say 1 in 4. It’s a statistic tacked these days on everything mental health-related: every news piece and every government report and every tube billboard and every charitable campaign. What’s interesting, though, is that this figure comes not from any conclusive evidence on the prevalence of mental illness but from surveys done by the UK Office of National Statistics that asked respondents if they’d had distressing symptoms in the past week.

The problem with such surveys is that how you feel on any given week may hold little clue about your mental state in general. Despite their rendering in popular imagination, mood disorders don’t knock you off into a bottomless pit where you sit and stare, eyes glazed over, into nothingness. They often vacillate through ups and downs: catatonic stillness then frenetic activity or even calm. If someone struggling took the survey on an unusually good month, that person would likely report no symptoms or downplay their severity and end up counted as non-symptomatic. Ditto for people in denial of their condition or those who’d hide it, reluctant to spill such intimate details to strangers.

But despite such obvious methodology flaws, the 1 in 4 figure stuck. To anyone concerned with mental health it must have seemed about right – high enough to render it of public concern yet still low, as pathologies are. Meanwhile, other studies existed, albeit few and far between, that tried to quantify the scale of mental illness. Some of these studies were better designed and hence less biased, yet they failed to make much of an impact.

One of them used civil registry data to track 5.6 million Danes between 2000 and 2012 and estimated that roughly 1 in 3 received psychiatric help during that period. And this in the world’s allegedly happiest country, in the presence of a universal free healthcare and without counting people in private therapy, on medication, or those who sought no treatment at all.

Other data came from the US National Comorbidity Survey, the country’s first large-scale study of the prevalence of mental illness. Between 1990 and 1992 over 8,000 Americans were interviewed, then reassessed 10 years later, followed by another batch of 9,282 new participants in 2001 and 2002. About 50% of all respondents turned out to meet the criteria for mental illness over the course of the study. Although high in comparison to previous estimates, this figure was likely still too low because of design limitations – from recall failure (we forget or misremember things over time) to selective participation (the people with the most severe symptoms tend to also be the ones most unwilling to disclose their medical histories).

In 2016 Terrie Moffitt and Avshalom Caspi, a research couple from Duke University, did their own estimation of the prevalence of mental disorders. Moffitt and Caspi are two of the most influential scientists today, widely recognised as pioneers in the field of gene-environment interaction, which took off in the late 90s with the spread of genome sequencing tools. Before their momentous meeting on a river cruise along the Mississippi, Caspi was looking at how childhood personality and family environment shape people’s life trajectories. Moffitt, meanwhile, was puzzling out the gene variations that made someone prone to extremes such as criminality and clinical depression. Eventually the two decided to join forces and wield the tools of their respective disciplines – hers: clinical psychology, neuroscience and genetics; his: sociology, epidemiology and behavior studies – to take on some of the biggest questions in human development. Drawing on staggering amounts of new tools and data, they are beginning to map out the complex dance of biology and psychology, of nature and nurture, as it plays out throughout human life.

When they decided to measure the true scale of mental illness, Moffitt and Caspi had an additional advantage over most other researchers. Moffitt is associate director of the Dunedin longitudinal study, one of the few research projects that track people from late childhood, when most mental disorders first manifest themselves, well into mid-life. The project began in the early 70s following 1,037 babies born in the same year in Dunedin, a lively town in New Zealand known as Edinburgh of the South for its Scottish heritage. Now in their 40s, the study’s subjects continue to show up for assessments every few years, flying, driving or walking over to the researchers’ offices for a day full of personal interviews, surveys, physical exams, blood samples, retinal imaging and other tests designed to poke into every nook and cranny of their lives – from their deepest despairs to their latest caries.

Moffitt came upon the Dunedin study by a curious accident. It was 1984 and she was a graduate student in clinical psychology at the University of South California. With her dissertation defense looming, she reckoned, as grad students do, that parachute jumping might help cure her performance anxiety. So she found herself an airplane and a parachute and jumped, winding up in a wheelchair with a broken leg. Her constriction meant that when a foreign researcher later came to visit campus, Moffitt was the one stuck with him. The researcher was Phil Silva and he headed a longitudinal study of child development in Dunedin, New Zealand. It had started as a humble experiment to see which toddlers would flourish by age three but then it took a life of its own. The data kept coming in, the study got extended and by the time Silva met Moffitt, he had amassed a wealth of information that she knew could help answer some of the most intractable questions in her field.

Most longitudinal studies have high dropout rates. 20% is normal but it can easily go up to 70% for longer follow-up periods as subjects get busy, move away or simply disengage. Not the Dunedin “kids”. More than four decades after the study began, an astonishing 95% of the surviving original cohort still show up for assessments. It’s this uncommon loyalty to the project that makes Moffitt and Caspi’s research so remarkable. Not only have they amassed a wealth of data free from much dropout-related bias, but by earning the goodwill of their subjects, they’ve managed to capture deeply personal details that other studies may have trouble eliciting.

Having a front seat to people’s lives unfolding before their eyes has so far yielded a trove of insights into human health and behavior – such as the finding that you are more likely to become obese the longer you sleep on weekends than you do on week days. Another revelation: for every hour children spent in front of the TV their risk of criminality later in life jumps by 30%.

To probe the prevalence of mental disorders, Moffitt and Caspi had experienced psychologists interview the subjects every few years between the ages of 11 and 38. When asked about their symptoms over the past year, 408 (41.3%) of the respondents reported episodes of depression, anxiety or substance use on more than three assessments, 409 (41.4%) on one or two assessments and only 171 – or 17.3% – had been symptom-free throughout. Of all participants, the overwhelming majority had met the criteria for mental disorder at least once between late childhood and mid-life. Mental illness, in other words, turned out to be not only more common than researchers had thought but it no longer looked like an exception, a deviation from the norm. It was the norm.

This can be hard to square with what we “know”. Accepting the Dunedin study’s results would put the prevalence of mental disorders at roughly 83% – meaning not 1 in 4, not even 1 in 2, but 1 in 1.2, if that made any sense at all. Surely something about the study must be off. Were the subjects overly sensitive in the first place? Was the definition of mental illness too liberal? Could it be that petty miseries got lumped together with legitimate ills?

These are fair concerns. Moffit and Caspi have thought about them, too, but concluded that they fail to explain the study’s surprising results. The researchers showed, for example, that in terms of general health and past-year incidence of mental distress, the Dunedin sample was similar to the wider population of New Zealand as well as the US. And as for measuring mental illness, they based their diagnoses on the Diagnostic and Statistical Manual of Mental Disorders (DSM) – the standard (if increasingly controversial) tool used by clinicians, researchers and drug regulation agencies around the world.

Granted, the 83% lifetime prevalence of mental illness that Moffitt and Caspi found reflects a range of symptoms and severities. The subjects diagnosed at more than three assessments displayed a wider range of disorders – such as PTSD, ADD, schizophrenia and mania – that were rare or non-existent in subjects diagnosed only once or twice. This more chronic group also had a higher chance of suffering from more than one disorder simultaneously and their actual impairment was more serious.

The milder cases, in comparison, had relatively brief episodes of anxiety, depression and substance use. But while it’s tempting to dismiss them as small matter, they are anything but. Moffitt and Caspi don’t share specific figures, but according to a NHS clinician I spoke to, a mild episode would typically last 23 days. This number has been echoed by recent economic studies, which estimated the productivity losses from ill mental health in the UK at 21 to 30 days per person per year. Furthermore, when Moffitt and Caspi compared the mildly affected with those never diagnosed, they found that the latter enjoyed significantly higher life satisfaction, social and economic status, as well as more education and better relationships.

What does this all mean? It means that what looks like a rare disease may in fact be a social epidemic – or, rather, human experience. It also means that what we think of as the madness of the few may turn out to be the secret we all share.

This is not to say that genetic risks and brain chemistry don’t matter, or that a four-week bout of depression exacts the same toll as one that puts you in the psych ward. It is to say that when it comes to the human mind, things defy the simple and often deterministic explanations of biology. The boundaries where normal ends and crazy begins are blurry. Mental health, it turns out, is not a given, nor is mental illness a genetic verdict. Perhaps the most fascinating – and foreboding –possibility suggested by the Dunedin study is that ours is a mad world. That today you don’t need “bad” genes or a rotten childhood to go crazy; being human may be enough. Living in what journalist Sebastian Junger called “an overfed, malnourished, sedentary, sunlight-deficient, sleep-deprived, competitive, inequitable, and socially-isolating environment” – may be enough.