His second novel, At Night All Blood is Black, was inspired by his Senegalese great-grandfather's silence about his experiences in World War One.
The £50,000 prize will be split between Diop and the book's translator - US author and poet Anna Moschovakis.
It fended off competition from five other shortlisted titles, including Éric Vuillard's The War of the Poor.
The International Booker Prize is awarded every year for a single book that is translated into English and published in the UK or Ireland.
This year's winning entry, by the French-Senegalese writer and literature professor, portrays a young man's descent into madness, and tells the story of the Senegalese who fought for France during the conflict.
"More than a century after World War One, a great new African writer is asking these questions in a spare yet extraordinary novel about this bloody stain on human history," said The New York Times in its review.
The Star Tribune said: "Though short, it is an immersive, propulsive read, one that searingly evokes the terrors of trench warfare, the relentless loss of life, and the irreparable damage inflicted on the human soul."
Speaking to the BBC recently, Diop said of his great-grandfather: "He never said anything to his wife, or to my mother, about his experience. That is why I was always very interested by all the tales and accounts which gave one access to a form of intimacy with that particular war."
The book was announced as the winner by the chair of the judges Lucy Hughes-Hallett, during a virtual celebration at Coventry Cathedral on Wednesday afternoon.
"This story of warfare and love and madness has a terrifying power," said Hughes-Hallet. "The protagonist is accused of sorcery, and there is something uncanny about the way the narrative works on the reader.
"We judges agreed that its incantatory prose and dark, brilliant vision had jangled our emotions and blown our minds. That it had cast a spell on us."
Prior to the win, Diop's novel had already bagged a raft of other literary awards, including France's Prix Goncourt des Lycéensm and the Swiss Prix Ahmadou Korouma, as well as the Strega European Prize in Italy.