The sale is due in January in New York by Guernsey's auction house. The seller is Christo Brand, Mandela's former prison guard in the notorious jail.
But Nathi Mthethwa, South Africa's culture minister, said there had been no consultation with the government.
"This key belongs to the people of South Africa," he said.
"It is not anyone's personal belonging," the minister added.
Guernsey's says its planned auction on 28 January is to raise funds for a memorial garden and museum around Mandela's burial site.
Mandela, who is a national hero in South Africa, spent 18 out to 27 years of his time in prison on Robben Island. Mr Brand later became a close friend of Mandela.
The notorious island, within sight of the city of Cape Town and Table Mountain, acquired its name from the seals that once populated it in multitudes - robben being the Dutch word for seal.
Mandela was released in 1990 as South Africa began to move away from strict racial segregation (apartheid) - a process completed by the first multi-racial elections in 1994 when he was elected the country's first black president.
Mandela served a single term, stepping down in 1999.
He died in 2013 aged 95.