The fight over statues of historic racists and colonisers is far more nuanced than politicians can seemingly grasp.

The federal Coalition party room didn't have time this week to talk about the Black Lives Matter protests. It did, however, manage to share some vapours over the defacing of Winston Churchill's statue in London. Somebody had written the words "was a racist" on the plinth, as an unironic statement of what is an objective fact.

But irony, these days, is the exclusive preserve of the ruling class, so Josh Frydenberg dropped a pearler on the party room which was duly shared with the media: if Churchill was a racist, he asked, then what would you call the guy he stopped? If Hitler is the last word on human evil (not an unfair proposition), and he was definitely a racist, then, according to Josh, Churchill was not. To say otherwise would be what, illogical?

Looked at another way, Churchill is just lucky he isn't in the Thames right now. The statues of ancient racists are toppling like ninepins, from Bristol to Brussels. King Leopold of Belgium, responsible for the deaths of half the population of the Congo, on the ground. Edward Colston, merchant and philanthropist, whose fortune came from the slave trade, in Bristol harbour.

In the USA, home to more statues of racist overlords than anywhere else in the world, it looks grim for General Robert E Lee and a host of other Confederate heroes, but Congress is not ready to take down the bust of General Nathan Bedford Forrest from its halls. Forrest, legitimately a war hero on the losing side, helped found the Ku Klux Klan. That was a very racist thing to do.

Down this end of the world, Captain Cook attracts much of the attention but what about Robert Towns? His statue was dedicated only in 2005 in Townsville, the city named for him. He was also, it seems, a blackbirder.

In the mid-1800s, South Pacific Islanders were routinely kidnapped in large numbers to work on Queensland plantations in conditions indistinguishable from slavery, and Towns was in the thick of it. Statues celebrate greatness, which is basically the old form of what we now call celebrity. Until very recently, being racist or sexist or homophobic or transphobic or just generally filthily bigoted was not in any way an obstacle to greatness, which was measured on other standards.

Some of those standards — like conquest, colonisation and forced conversion — used to be good, and now are bad. However, for the antagonists on either side of the history wars, the distinctions between partly bad history and entirely bad history seem to be irrelevant. The argument is simplistic: the statue must stay up, or it must come down.

Tony Abbott, of course, is opposed to the movement that wants Oriel College at Oxford to remove its statue of Cecil Rhodes. He says that "pulling down statues of past heroes is cultural vandalism of the worst sort". Yes, Rhodes had some standards that we might find challenging (he was seriously, seriously racist) but it would be arrogant to assume that our current opinions are superior to his.

The statuary debate is just a physical representation of the endless and tedious argument over racism and all the other -isms. It is, on the side of those who defend history as an immutable progression of civilisation over barbarism, a very simple question: why are you offended?

These men (check: yes, they're always men) did great things. They should not be judged by today's more enlightened standards, because to do so is to airbrush history into something it wasn't.

But Rhodes was not a hero to the people whose lives he ended, whose freedom he took, whose cultures he crushed. He is not a hero to their descendants.

There is not an easy answer to this. History should never be erased. The deepest irony is that its whitewashing is not the result of statues being downed, but more often is a consequence of their retention. What does it say that the founder of the KKK is displayed as a figure of greatness, no matter how qualified, in America?

There was once a picture of Mao in every room in China; of Stalin in Russia, likewise. And of Hitler, in Germany, too. They all came down. The idea that any statue, once up, must remain, is as specious as suggesting that those gods should still be revered.

No, it's far more nuanced a problem than Tony Abbott could ever grasp or Josh Frydenberg's line suggests. Like it or not, white people, the descendants of the victims of your heroes are offended by their outsized bronze presence.

Statue by statue, building name by building name, these conversations are going to have to be had. History — not its truth but its representation — is no longer the preserve of the victors.

We do not disclaim anything about this article. We're quite proud of it really.