The stark findings are the result of an extensive survey by Australia's Anti-Slavery Taskforce.
The Taskforce's Executive Officer, Alison Rahill, said the Australian government needs greater scrutiny of the Pacific Labour Mobility scheme and that better support is needed for its seasonal workers.
The Taskforce is a social agency established by the Catholic Archdiocese of Sydney, providing support for potential victims of slavery-like practices.
Ms Rahill said isolation and other factors increases the risk of abuse.
"The issue of sexual harassment and sexual assault for Pacific women is when you're alone in a country hundreds of miles from anywhere, where you've got no support system, the isolation, the language, no wi-fi, and there might not be any community nearby .... so that increases the risk of abuse.
"What access do these workers have to support, where they can ask for help without fear of retaliation?"
The Samoan government is currently undertaking a review of the seasonal work scheme in both Australia and New Zealand after repatriating 200 of its workers from Australia due to breaching their work conditions.
At least 18 of those workers had given birth while overseas.
Rahill said duty of care for female seasonal workers who find themselves in these situations lies with the Australian government. However, the Taskforce has stepped up to provide pastoral care.
"When I hear the Australian government talking about if you do something wrong you'll bring shame on your country and that kind of thing but it's like, well, the programme actually needs greater scrutiny," she said.
"Where we can, we've arranged for pre-natal care, access to hospital care to give birth, and other needs for newborn babies, because these are women who are in particularly vulnerable situations.
"For workers who are sent home early, my question is 'well, they're here to work - what actually happened?' and more often than not, the workers could have raised a number of issues - and instead of dealing with those issues, the workers are actually punished."