The children were students at the Kamloops Indian Residential School in British Columbia, which closed in 1978.
The discovery was announced on Thursday by the chief of the Tk'emlups te Secwepemc First Nation.
Prime Minister Justin Trudeau said it was a "painful reminder" of a "shameful chapter of our country's history".
The First Nation is working with museum specialists and the coroner's office to establish the causes and timings of the deaths, which are not currently known.
Rosanne Casimir, the chief of the community in British Columbia's city of Kamloops, said the preliminary finding represented an unthinkable loss that was never documented by the school's administrators.
Canada's residential schools were compulsory boarding schools run by the government and religious authorities during the 19th and 20th centuries, with the aim of forcibly assimilating indigenous youth.
Kamloops Indian Residential School was the largest in the residential system. Opened under Roman Catholic administration in 1890, the school had as many as 500 students when enrolment peaked in the 1950s.
The central government took over administration of the school in 1969, operating it as a residence for local students until 1978, when it was closed.
What do we know about the remains?
The Tk'emlups te Secwepemc First Nation said the remains were found with the help of a ground-penetrating radar during a survey of the school.
"To our knowledge, these missing children are undocumented deaths," Casimir said. "Some were as young as three years old."
"We sought out a way to confirm that knowing out of deepest respect and love for those lost children and their families, understanding that Tk'emlups te Secwepemc is the final resting place of these children."
The main administrative building at the Kamloops Indian Residential School is seen in Kamloops, British Columbia
The tribe said it had reached out to the home communities whose children attended the school. They expected to have preliminary findings by mid-June.
British Columbia's chief coroner Lisa Lapointe told Canadian broadcaster CBC "we are early in the process of gathering information".
What reaction has there been?
The reaction has been one of shock, grief and contrition.
"The news that remains were found at the former Kamloops residential school breaks my heart," Trudeau wrote in a tweet.
Canada's minister of indigenous relations, Carolyn Bennett, said residential schools were part of a "shameful" colonial policy. The government was committed to "memorialising those lost innocent souls", she said.
Terry Teegee, the regional chief of British Columbia's Assembly of First Nations, called finding such grave sites "urgent work" that "refreshes the grief and loss" of communities in the region.
Those views were echoed by other indigenous groups, including the First Nations Health Authority (FNHA).
"That this situation exists is sadly not a surprise and illustrates the damaging and lasting impacts that the residential school system continues to have on First Nations people, their families and communities,'' its CEO Richard Jock wrote in a statement.
What were residential schools?
From about 1863 to 1998, more than 150,000 indigenous children were taken from their families and placed in these schools.
The children were often not allowed to speak their language or to practise their culture, and many were mistreated and abused.
A commission launched in 2008 to document the impacts of this system found that large numbers of indigenous children never returned to their home communities.
The landmark Truth and Reconciliation report, released in 2015, said the policy amounted to "cultural genocide".
In 2008, the Canadian government formally apologised for the system.
The Missing Children Project documents the deaths and the burial places of children who died while attending the schools. To date, more than 4,100 children who died while attending a residential school have been identified, it says.