Of brother Prince William and his wife Catherine, he said: "I always hoped that the four of us would get on.
"But very quickly it became Meghan versus Kate.
"And that, when it plays out so publicly, you can't hide from that," he told Tom Bradby, in an ITV interview.
BBC reports in the 95-minute interview to promote his memoir, titled Spare, he recalled the days when he was the "third wheel" at official engagements and other outings - but his relationship with Prince William and Catherine was particularly warm at that time.
He said he imagined that would continue when he found his own partner and they became a foursome, but it was not to be.
He said negative attitudes about Meghan - particularly around her being an American actress - were heightened by a hostile press.
Those stereotypes about the new woman in his life had been a barrier to his brother and sister-in-law "welcoming her in" to the family, he said.
Prince Harry said while Prince William never tried to dissuade him from marrying Meghan, he did "air some concerns" and warned Harry: "This is gonna be really hard for you".
"I still to this day don't truly understand which part… he was talking about. But maybe, you know, maybe he predicted what the British press's reaction was gonna be."
In the interview, Harry also spoke of the emotional impact of the death of his mother and in his book describes returning to the scene of the car accident in Paris - and asking to be driven through it at the same speed as she was in 1997.
"I'd always imagined the tunnel was some treacherous passageway, inherently dangerous but it was just a short, simple, no-frills tunnel. No reason anyone should ever die inside it," he said.
He saw no point in reopening inquiries into the car accident - but questioned the official conclusions about the night of the crash.
A 2008 inquest found Diana was unlawfully killed partly due to the "gross negligence" of her driver who had been drunk and driving at excessive speed.
In the interview, Prince Harry said that the tunnel posed "no danger of anybody losing control even after a drink or a couple of drinks… [It was] almost physically impossible to lose control of a vehicle unless you are completely blinded at the wheel," implying that distraction from the paparazzi may have caused the driver to crash.
"The people that were predominantly responsible for it, all got away with it," he said.
Prince Harry also said that, after Diana's death, he and his brother William were sat down in a room and told that the events leading to the car crash were "like a bicycle chain".
"If you remove just one of those links from the chain, the end result doesn't happen," Harry recalled being told.
When William asked what might have happened had the paparazzi not been chasing Diana's car, he was told that "the result wouldn't have been the same."
During the interview, Harry also expressed his hope that he might one day reconcile with his father and brother.
"Forgiveness is 100 percent a possibility, because I would like to get my father back. I would like to have my brother back," he told interviewer Tom Bradby.
"At the moment, I don't recognise them, as much as they probably don't recognise me."
He blamed the tabloid press as the "antagonist", who he said "want to create as much conflict as possible".
The ITV interview is the first of four broadcast appearances to be aired over the coming days to promote Spare.
Prince Harry also spoke to two US TV networks. He gave an interview to Anderson Cooper for 60 Minutes on CBS News, which will air on Sunday night, and also spoke to Michael Strahan of Good Morning America, which will be broadcast on Monday. He will also appear as a guest on Stephen Colbert's Late Show on CBS on Tuesday.
Although Spare is not due to be published until Tuesday, extracts were leaked after some copies went on sale early in Spain. BBC News has obtained a copy and has been translating it.
Kensington Palace and Buckingham Palace have both said they will not comment on the contents of the book.