The Nazi symbols adorn an anchor that was on display in Pointe-des-Cascades, Quebec.
Corey Fleischer, who goes around Montreal removing hateful graffiti, tried to paint over the swastikas.
But the mayor stopped him and had police remove him from the park, arguing the anchor is a part of local history.
The anchor has a plaque that identifies it as a "souvenir of Nazism", and says it was used in Europe at the end of War, probably on a merchant boat, and found in 1980.
But in a statement on the town's website, the mayor says the anchor belonged to a merchant vessel that predates World War Two and was found by local divers 25 years ago.
"The village of Pointe-des-Cascades does not endorse Nazism," said Mayor Gilles Santerre in a statement online.
"Our village has a beautiful community and family spirit, and creates events that bring people together."
The village has about 1,500 people and is located about 50 miles (80km) from Montreal.
The statement cites an article by Radio Canada, which says that before 1920 the swastika was a symbol of peace.
It is used in many religions around the world, including Hinduism and Buddhism.
Around the world debates have raged over historically significant monuments containing swastikas.
In Japan, temples are often marked by swastikas, and a push to stop using the symbol on tourist maps sparked a backlash.
To avoid confusion, the mayor of Pointe-des-Cascades says the town will put up a plaque that better explains the context of the anchor.
However, Mr Fleischer says older meaning is irrelevant and the symbol on the anchor is clearly one of Nazi Germany.
It is painted black and laid over a white circle, a stylistic choice that he says was only employed by the Third Reich.
As founder of Erase the Hate, Mr Fleischer has travelled the world to remove hateful and anti-Semitic graffiti.
"Maybe the city did not know," he told the BBC. "But I know exactly what this is. There is no ifs ands or buts about it."
Mr Fleischer said that as a piece of history, it belongs in a museum where its meaning is clear, instead of in a public park.
"It is a place where people come to feel safe and this is being displayed for everybody to see," he said.