He pointedly failed to thank Mr Trump or praise his policies.
The former ExxonMobil chief had a series of public rifts with the White House after being appointed last year.
Mr Trump officially fired Mr Tillerson via Twitter, naming CIA Director Mike Pompeo as his replacement.
Mr Trump also named Gina Haspel to become the CIA's first female director.
Speaking to reporters at the Department of State, Mr Tillerson said good work had been done to have better relations with China, and rein in North Korea's nuclear weapons programme.
