Eleven bodies have so far been pulled from the remains of the barracks in the central province of Quang Tri.
One official says he heard landslides "exploding like bombs" in the night.
Heavy rainfall has hit much of Vietnam in the past week. Floods and landslides have killed at least 70 people.
There are fears floodwaters could rise further in the coming days.
The government said a unit of Vietnam's 4th Military Region was hit by the landslide in the early hours of Sunday morning.
"From 2am, there have been four to five landslides, exploding like bombs and it feels like the whole mountain is about to collapse," local official Ha Ngoc Duong was quoted by the VnExpress news site as saying.
As rescuers worked to retrieve bodies, senior officials warned of the danger of further landslides and a safer way to access the site may be needed.
The deadly landslide comes days after 13 members of a rescue team - many of them soldiers - were found dead trying to save workers from a hydropower plant that had been hit by a landslide in the neighbouring province of Thua Thien Hue.