Video on social media showed a torrent of black mud plummeting down from a mountain top and on through the city towards the sea. Several houses were destroyed or buried.
Kyodo news agency reports that the bodies of what appear to be two victims have been found in the port district.
A witness told national broadcaster NHK: "I heard a horrible sound and saw a mudslide flowing downwards as rescue workers were urging people to evacuate. So I ran to higher ground."