The 32-year-old was hurt in a tour match against the Chiefs on June 20 and has been struggling with headache-related problems ever since.
Consequently, Payne – who earned 20 Ireland caps – will end his playing career and take up a role as defence coach with Ulster, the Pro14 side have confirmed.
"It's been a good ride but unfortunately every good thing has to come to an end," he told Ulster's official website.
"Playing rugby has taken me to places I never thought I'd see and allowed me to meet people I never thought I'd meet.
