Video on social media showed a torrent of black mud plummeting down from a mountain top and on through the city towards the sea. Several houses were destroyed or buried.
Kyodo news agency reports that the bodies of what appear to be two victims have been found in the port district.
A witness told national broadcaster NHK: "I heard a horrible sound and saw a mudslide flowing downwards as rescue workers were urging people to evacuate. So I ran to higher ground."
Prime Minister Yoshihide Suga has put together a task force to respond to the disaster and the wider emergency caused by heavy rainfall on the Pacific coastline.
Atami, a hot-spring resort in the prefecture of Shizuoka, has had more rainfall in the first three days of July than it usually sees in the whole month. The weather has been similar in neighbouring Kanagawa prefecture.
The mudslide struck at 10.30am (local time) according to a local resident.
Police, firefighters and members of Japan's military are involved in a search operation, according to NHK.
Residents in parts of three prefectures - Shizuoka, Kanagawa and Chiba - have been ordered to evacuate following warnings of further flooding in low-lying areas.
Japan is prone to mudslides and flooding during its annual rainy season. Dozens were killed in flooding in July last year, and more than 200 died in 2018 when parts of western Japan were inundated.