New Zealand led 1-nil for much of the match after Olivia Chance scored late in the first half with Belgium scoring the equaliser with only minutes remaining.
The Football Ferns were reduced to ten players midway through the second half after defender Claudia Bunge was sent off after receiving a second yellow card.
New Zealand won the shoot-out 7-6 and will now play either Portugal or Italy in the semi-finals.
Sarah Gregorius made her 100th international appearance when she came on as a second half replacement and she promptly announced her retirement from top level play after the match.
Belgium are ranked six places higher than New Zealand on the world rankings with Belgium at 17 and the Football Ferns 23rd.
"It certainly was nerve-wracking but it was a terrific performance by the players," coach Tom Sermanni said.
"In large parts of the first half particularly, I thought we played some of the best football we've played in a long time. The players then showed a huge amount of character and determination,"
"The fact that we kept our discipline, our organisation and our concentration really said a lot about the determination of the team to get a result. It was a great result and a really good performance."
Sermani was pleased Gregorius got to end her career on a high.
"To end her last game by scoring a goal with her last kick - it doesn't get any better than that," Sermanni said. "It was a great send off for her and a perfect way to finish her career."
The Football Ferns will now take on either Portugal or Italy in the semifinals on Sunday morning.