Small, who won the World Cup with South Africa in 1995 beating the All Blacks in a gripping final that went to extra time, was 50.
The South African Rugby Union said Small suffered a suspected heart attack late on Tuesday in Johannesburg.
"We are saddened to learn that Springboks great and 1995 Rugby World Cup winner James Small has passed away. A fierce competitor on the field and great friend off it, James will be missed by the entire global rugby community," the All Blacks tweeted.
Small played 47 tests for South Africa between 1992 and 1997, scoring 27 tries, and was a key part of the champion 1995 team.
The 15-12 victory over New Zealand in the final by South Africa's integrated team was a symbolic moment for the country, coming shortly after the end of apartheid.
In the final, Small lined up opposite New Zealand's star player Jonah Lomu and successfully prevented him from scoring.
The All Blacks had been struck down by sickness in the days before the final, with rumours of food poisoning.
Small is not the first player from that Springbok team to have died - loose forward Ruben Kruger died from brain cancer in 2010 and scrumhalf Joost van der Westhuizen from motor neurone disease in 2017.
In addition, coach Kitch Christie died in 1998.
The African National Congress joined the tributes to "patriot" Small.
The party released a statement after news of his death was confirmed. It read:
The statement read:
"The African National Congress (ANC) is shocked and saddened by the passing of former Springbok player, James Small.
"He was a member of the legendary South African Rugby squad that won the World Cup in 1995, a year after the ushering in of a democratic government in South Africa.
"The class of 1995 contributed enormously towards uniting the nation and displaying to the world South Africa's sporting prowess.
"James Small was a patriot who fought like a soldier in the field of play and was prepared to die with his boots on.
"He played with extraordinary passion and commitment, for he knew that every time he walked on to the field, he carried the hopes of the nation on his shoulders.
"The ANC conveys its heartfelt condolences to his family, friends and the entire sporting world."