His novel tells the story of a young black man who tries to reinstate slavery and racial segregation in a suburb of Los Angeles.
Amanda Foreman, chair of the judges, said the book managed "to eviscerate every social taboo".
Beatty's win was announced at a ceremony at London's Guildhall on Tuesday.
Picking up the £50,000 prize from the Duchess of Cornwall, Beatty, 54, was clearly overwhelmed with emotion and struggled for words as he began his acceptance speech.
"I hate writing," he admitted later in the speech.
"This is a hard book," he went on. "It was a hard for me to write, I know it's hard to read. Everyone's coming at it from different angles."
This is the third year that the £50,000 prize has been open to writers of any nationality. The shortlist included two British, two US, one Canadian and one British-Canadian writer.
The Sellout beat five other novels, including Madeleine Thien's Do Not Say We Have Nothing, the bookies' favourite, and Graeme Macrae Burnet's Scottish crime thriller His Bloody Project.
'Top of their game'
Foreman said the judges took about four hours to reach their unanimous decision.
She said The Sellout was "a novel for our times" that contained "an absolutely savage wit" reminiscent of Jonathan Swift or Mark Twain.
"This is a book that nails the reader to the cross with cheerful abandon," she elaborated. "But while you are being nailed you are being tickled."
Beatty was "someone writing at the top of their game", she added.
"This is a first-class piece of serious literature wrapped up in a shawl of humour."
The novel is narrated by Bonbon - a resident of the fictional "agrarian ghetto" of Dickens on the southern outskirts of Los Angeles, which has been removed from the map to save California from embarrassment.
Bonbon is on trial in the Supreme Court for attempting to reinstitute slavery and segregation in the local high school as a means of bringing about civic order.
Frances Gertler, web editor at Foyles bookshops, described The Sellout as "a smart satire with a memorable narrator".
"Brave and funny - it takes a bit of getting into but once there, you don't want to leave," she said.
Waterstones fiction buyer, Chris White, said: "The Booker judges have awarded the most significant of literary prizes to what feels like the most significant novel to have emerged in these strange and difficult times."
Born in Los Angeles in 1962, Beatty's three previous novels are Slumberland, Tuff and The White Boy Shuffle. He now lives in New York.
The shortlisted authors each receive £2,500 and a specially bound edition of their book. The winner receives a further £50,000.
This is the second consecutive Man Booker Prize success for independent publisher Oneworld, following Marlon James's win with A Brief History of Seven Killings in 2015.