In the TV3 Reid-Research poll - released last night - Kiwis were asked whether they wanted change now that they had seen the final four alternative flag options.
Only 25 percent said yes, while the majority said no.
But PM Key - who has previously voiced his support for the silver fern on the flag - said the question asked was too basic, as there were people who had yet to make up their minds.
He told Morning Report fewer than 50 percent of people want to keep the current flag, and that number was falling.
The entire project is expected to cost more than $25 million.
Four flag designs were unveiled earlier this month, as the country decides the future of New Zealand's flag.
A government-appointed panel whittled down more than 10,000 design submissions to a longlist of 40, which was made public in August, and the final four choices were revealed on 1 September.
A referendum will be held between 20 November and 11 December to decide which of the four designs will run against the current flag in a second referendum in March 2016. The second referendum will decide which flag will represent New Zealand in the future.
Last week a political standoff was sparked between Labour and the Government over adding a fifth design, Red Peak, to the flag referendum, with Key challenging Labour to back the inclusion of the Red Peak flag in the line-up of options, and the referendum as it stands, while Labour wanted Key to ask voters, in the first referendum, whether they even want to change the flag.
The Red Peak design has received strong support on social media, and an electronic petition asking the Prime Minister to include the Red Peak design referendum has been signed by 50,000 people and presented to Parliament.