Two residents were badly injured jumping from windows of the building in Vanersborg, about 80km (50 miles) north of Gothenburg.
The cause of the fire is unknown though police have opened an arson inquiry.
Sweden is still debating controversial comments last week by US President Donald Trump on crimes related to its immigration policies.
The fire broke out on the third floor of one of the buildings at the Restad Farm shelter shortly after 04:00 local time (03:00 GMT).
