The eight-carriage train reportedly hit a construction vehicle that had slipped onto the tracks at the tunnel's mouth.
Rescuers combed badly damaged carriages inside the tunnel to find survivors, some of whom smashed windows to flee.
The train, from the capital Taipei to Taitung, was carrying people travelling for a long-weekend annual holiday.
Many people may have been standing because the train was so full.
The 408 train is one of the fastest deployed on a network that is generally considered safe. It can reach speeds of 130km/h (80mph).
Friday's crash is Taiwan's worst rail disaster in decades. President Tsai Ing-wen has sent her condolences to the families of the victims and ordered an investigation.
The latest reports from the National Fire Agency say 494 people were on the train, with 50 dead and 66 injured and taken to hospital.
The crash took place at about 09:00 local time (01:00 GMT).
Some people at the back of the train were able to walk away unscathed, while 100 were rescued from the first four carriages. Many of the dead, injured and trapped were in four crumpled carriages inside the tunnel.
"It felt like there was a sudden violent jolt and I found myself falling to the floor," one female survivor told Taiwan's UDN. "We broke the window to climb to the roof of the train to get out."
Another rescued woman said: "My whole body fell to the floor. I hit my head and it started bleeding."
A 50-year-old survivor told Apple Daily she saw many people trapped under their seats and when she walked out of her carriage she saw bodies everywhere.
Local media reports say the train driver is among the dead.
Images show a large, yellow flatbed truck lying at the side of the tracks. A construction project has been under way near the north end of the tunnel.
It is not known how the vehicle slipped down the embankment.