The move marked a sharp turnaround from as recently as last week, when some officials were still insisting they could organise the Games safely with some fans.
It all but strips the global sporting showpiece, which is due to start on 23 July and run until 8 August, of its last vestiges of pomp and public spectacle.
Prime Minister Yoshihide Suga said it was essential to prevent Tokyo, where the highly contagious Delta variant of Covid-19 variant was spreading, from becoming a flashpoint of new infections.
People will also be asked not to gather for events on public roads, such as the triathlon, though officials said some venues outside the greater Tokyo metropolitan area would allow small numbers of spectators
"It is regrettable that we are delivering the Games in a very limited format, facing the spread of coronavirus infections," Tokyo 2020 President Seiko Hashimoto said, following talks between government officials, Tokyo organisers and Olympic and Paralympic representatives.
"I am sorry for those who purchased tickets."