A special caucus meeting will be called on 12 December, when a new leader will be decided upon.
He said it had been a privilege to serve the people of Helensville, and he will stay in Parliament long enough to avoid a by-election.
Mr Key made the announcement at Parliament today.
He told his Cabinet and caucus colleagues this morning that he did not intend to stay on for a fourth term as leader, he said.
"To me, this feels like the right time to go."
The timing would give caucus and the new leader time to settle in prior to next year's general election, he said.
It was "the hardest decision I've ever made," he said.
He did not know what he would do next.
Mr Key said he had received "amazing support" from the New Zealand public.
Mr Key said he just did not feel comfortable looking down the barrel of the camera and saying he would stay for a full fourth term.
He said he had no option, but to do it right now.
"I've come too far to mislead the public. I'm just not going to do that."
Whoever was chosen as the next leader would have his "unwavering support", but if Deputy Prime Minister Bill English put his name forward, then Mr Key would support his bid, he said.
"I didn't want … ambiguity on my part.
"If I didn't think he was right to be Prime Minister, then I shouldn't have thought he was right to be deputy," said Mr Key.
Mr English's office released a statement shortly after the announcement, but made no mention of whether or not the he would put his name forward for the leadership, although he has called a press conference for 3pm.
Instead, Mr English said Mr Key "will be judged by history as one of New Zealand's greatest leaders".
He thanked Mr Key for his "years of dedicated and outstanding service".
"While the gap he leaves is huge we understand and respect his decision to step down from a job from which there is no respite. We wish John and his family every success with their life out of the public eye."
In his maiden speech to Parliament, Mr Key paid tribute to his mother, saying she had instilled in him the idea that there is no substitute for hard work and determination.
By August 2004 he was the National Party's finance spokesperson and early the following year, he was promoted to the front bench and ranked at number seven.
After months of speculation about the leadership, Mr Key became the National Party leader in November 2006, following the resignation of Don Brash.
On taking leadership, he signalled a retreat from hard-Right positions on welfare, Màori issues, social issues and climate change.
In 2008 he led the National Party to victory in the general election and took the option of forming a minority government with the support of two minor parties.
In his election night speech, he stressed the importance of individual achievement to New Zealand's prosperity.