Joe Dunthorne tells us he originally envisaged this book as a story of his grandmother’s childhood escape from the Nazis; the reality turned out to be more complex. Narrated with the twists and turns of a detective story, Children of Radium is a family memoir that records the mazy path by which the prize-winning Welsh novelist discovered just how little he knew of his German Jewish heritage. His journey begins with “a foot-high block of A4”: a 2,000-page unpublished memoir by his great-grandfather, Siegfried, a Jewish scientist who worked at a secret chemical weapons laboratory near Berlin before he and his family left for Turkey – not the panicky flit Dunthorne imagined, but a relocation bankrolled by employers with plans for what he could yet do.

Hunches, tip-offs, false trails and dead ends abound in Dunthorne’s quest to determine how much Siegfried knew – and when – about his work’s murderous potential after he was reassigned in 1928 from toothpaste manufacture by his firm, a specialist in radioactive products. Siegfried’s memoir is circumspect, and the hunt for answers isn’t straightforward: not only was the site of Siegfried’s lab heavily bombed, but Dunthorne’s mum also chucked his papers into the recycling while clearing out her late mother’s flat.

An eye for that kind of comedy, honed in Dunthorne’s novels – the best known is Submarine (2008), filmed by Richard Ayoade – brightens a quixotic voyage into the heart of 20th-century darkness. The trail leads through libraries, museums and medical records, but also less obviously writer-friendly locales: in Germany, he wriggles belly-first into a fenced-off radioactive site in a clandestine hunt for soil to test for gas traces; and in Turkey, Dunthorne blags his way through military checkpoints in the company of a formerly jailed member of the Kurdistan Workers’ party, having learned that one of the letters he has from Siegfried might hold evidence of culpability for a massacre in an eastern mountain town prior to the second world war.

Dunthorne’s voice – affable, warm, wry – casts a spell right from the book’s dedication (“This book is for – and, arguably, by – my mother”), making light work of tricky ground as he weaves fact and guesswork, reading and testimony. Despite everything, humour is never far away. When, in Germany, he suddenly feels the need to apologise to an elderly interviewee for Siegfried’s work with chemical weapons, the man demurs and instead apologises on behalf of all Germans to Dunthorne, a descendant of expatriated Jews; at which point the author apologises for putting him in the position where he felt he needed to apologise. It takes a special writer to generate embarrassment comedy from this material, but you come to feel that Dunthorne is probably the kind of author who is witty in his sleep: the Nazis didn’t deploy poison gas on the battlefield, he says, because Hitler personally vetoed its use, “creat[ing] the uneasy situation in which my great-grandfather’s work might have been far more lethal without an intervention from Hitler”, a line that manages to be heartfelt as well as undeniably comic.

As discoveries and ambiguities mount, the book plays out as a tangled investigation of complicity, courage and cowardice, ceaselessly yo-yoing between potential indictment and mitigation. Dunthorne’s instinctively jokey tone doesn’t minimise the ever-present horror, yet he recognises, too, that the darkest aspects of his story are tricky to separate from the frisson of proximity, the fundamental thrill of the chase (one chapter ends: “The real revelation came, several weeks later, via email…”). We catch his perverse sense of disappointment when his hard-won soil sample contains traces of everything but gas from Siegfried’s lab. There’s steady intrigue, also, in the unmistakable resonance of Dunthorne’s decision to embark on an all-consuming pan-continental research quest while slap-bang in the middle of early fatherhood. It’s also a kind of stealth post-Brexit narrative, as Dunthorne obtains German citizenship – an ambivalent reintegration by which the convenience of an EU passport is weighed alongside “formal reconciliation with the country which had tried to systematically eliminate [his mother’s] forebears”.

Dunthorne recently told the Guardian how much he admired Laurent Binet’s tricksy 2010 novel HHhH, a book that conspicuously shows its own working as it unreliably imagines its way into the Nazi era, and you can feel its imprint here. Children of Radium put me in mind, too, of Richard Flanagan’s Question 7, another genealogical retracing that turns into a meditation on guilt, atrocity and unforeseen consequence. Dunthorne’s tome is a humbler enterprise, keenly aware that the writerly ego can be led astray by an impulse to join the dots: witness the belated recognition that his focus might after all be entirely in the wrong place, thanks to a splendidly deflating comment from his mother, who wonders if Dunthorne should be writing instead about her great aunt – Siegfried’s sister – who bravely oversaw a Munich children’s home, caring for Jewish refugees amid rising persecution.

By necessity, Children of Radium is piecemeal, inconclusive, full of pregnant silences, maybes and what ifs. Near the end, Dunthorne and his mother soak up memories in north London, where Siegfried spent his last days in a care home, regularly greeting his granddaughter and her boyfriend – the author’s dad – with a meal of ox tongue. “It speaks to a paucity of other research materials that I thought it worthwhile to cook an ox tongue,” Dunthorne tells us. He didn’t know what to expect, and was alarmed when the length of flesh seemed to revive in the pan, “writhing and flexing”, refusing to stay put when prodded with a spoon, “lifting the lid off the pot”. A metaphor, you can’t help think, but it befits the procedures and conclusions of this slippery marvel that we can’t quite say for what.