French culture minister Roselyne Bachelot said restoring it to its rightful owners was an acknowledgement of the crimes they suffered.
She said the painting bore witness to the "broken lives" of the Nazi era.
The French state bought the work, its only Klimt, in 1980 without realising its history.
The pre-war owner of Rosiers sous les Arbres (Rose Bushes Under the Trees) was Nora Stiasny, from a well-known Austrian Jewish family. She had inherited it from her uncle, the Austrian industrialist and art collector Viktor Zuckerkandl, Ms Bachelot told a news conference at the Musée d'Orsay in Paris.
- Painting found in wall confirmed as stolen Klimt
- Austrian panel opposes return of Klimt frieze looted by Nazis
Ms Stiasny was forced to sell it in August 1938 at a knock-down price to survive financially, months after the Nazis annexed Austria.
In 1942 Ms Stiasny was deported to a concentration camp in Nazi-occupied Poland, and died the same year.
The art dealer held on to the masterpiece until his death in the 1960s. France bought it at auction in 1980 for the Musée d'Orsay.
"Today we know that it is indeed a work that was looted in Austria in August 1938," Ms Bachelot told reporters.
"The decision we've taken is of course a difficult one. It results in taking a masterpiece out of the national collections which is the only painting by Gustav Klimt which France owned.
"But this decision is necessary, essential. Eighty-three years after the forced sale of this painting by Nora Stiasny, this is the accomplishment of an act of justice."
The beneficiaries will be the descendants of Nora Stiasny's sister. The French government will have to pass a bill to allow it to be released from the national collection and returned to the family, Ms Bachelot added.
In 2017 a floral Gustav Klimt painting was sold for nearly £48m at Sotheby's in London.
France is to return the only painting by Gustav Klimt in its national collection to the heirs of the Jewish family that was forced to sell it by the Nazis more than 80 years ago.
The colourful 1905 oil masterpiece by the Austrian artist called Rosiers sous les Arbes (Rose Bushes Under The Trees) has been hanging in the Musee d'Orsay in Paris for decades.
- U.K. Property Prices Expected to Tumble in Coming MonthsMansion Global
It will be returned to the family of its pre-war owner - Nora Stiasny, a Jewish woman and Holocaust victim - who sold it under duress to an art dealer at a knock-down price to survive financially after the Nazis annexed Austria in 1938.
She was deported to Poland in 1942 and died the same year.
The art dealer, a Nazi sympathiser, held on to it until his death in the 1960s and France, unaware of its history, bought it at auction in 1980.
French culture minister Roselyne Bachelot said Ms Stiasny had inherited the painting from her uncle, Austrian industrialist and art collector Viktor Zuckerkandl, who bought the work in 1911.
"It is in recent years that the true origin of the painting has been established," the minister said, adding it was "the only Gustav Klimt painting owned by France".