Evan Fournier poured in 28 points for the French as the Americans, who have won gold in the past three Games, were beaten for the first time since 2004.
A strong third quarter, where France outscored Gregg Popovich's star-studded team 25-11, proved decisive.
Rudy Gobert was also influential with 14 points and nine rebounds.
It was a disastrous start for Team USA who struggled in their preparation for the tournament, losing warm-up matches to Nigeria and Australia.
They looked to have taken control of the game with an eight-point half-time lead.
But, with Kevin Durant in foul trouble early on, they could not maintain that momentum as a flat third quarter saw them score just 11 points.
Jrue Holiday, with 18 points in just under 28 minutes of action coming off the bench, was a bright spot in their struggles and helped them to a six-point lead in the fourth quarter.
But the Americans missed four open three-point attempts in the final 30 seconds to hand France the advantage and Boston Celtics guard Fournier led them to an impressive victory.
Popovich will now hope his side can respond against Iran on Wednesday, with France taking on the Czech Republic.