Eleven bodies have so far been pulled from the remains of the barracks in the central province of Quang Tri.
One official says he heard landslides "exploding like bombs" in the night.
Heavy rainfall has hit much of Vietnam in the past week. Floods and landslides have killed at least 70 people.
There are fears floodwaters could rise further in the coming days.
The government said a unit of Vietnam's 4th Military Region was hit by the landslide in the early hours of Sunday morning.
