Many struggled to hold back tears at the ceremony held at Ground Zero, the site of the Twin Towers destroyed in the attacks by al-Qaeda militants.
"Twenty years feels like an eternity, but it still feels like yesterday," cried Lisa Reina who lost her husband.
A minute's silence was held at the exact time each hijacked plane crashed.
George W Bush, who was the US president at the time, gave a speech in Pennsylvania, where one of the planes crashed into a field after passengers overpowered the hijackers.
"The world was loud with carnage and sirens, and then quiet with missing voices that would never be heard again," he said.
"It's hard to describe the mix of feelings we experienced."
The official memorial in New York started with a minute's silence at 08:46 (12:46 GMT) - the exact moment the first plane hit the North Tower of the World Trade Center in 2001.
All morning, roses continued to be placed beside the names of the 2,977 victims etched into the Ground Zero memorial.
There were five more moments of silence over the next few hours - marking the time when the second plane crashed into the South Tower, when a third jet struck the Pentagon just outside Washington DC, when the fourth plane crashed in Pennsylvania, and finally when each tower collapsed.
The tributes will continue into the night, when two beams of light will shine four miles (6.4 km) into the sky where the towers stood.