Before their game, Germany's players covered their mouths during the team photo to protest against FIFA following its decision to stop plans to wear armbands that were seen as protesting discrimination in host nation Qatar.
Nancy Faeser, Germany's sports minister, attended the match at the Khalifa International Stadium and was sitting beside FIFA president Gianni Infantino while wearing the same "One Love" armband that FIFA had outlawed with its threats of consequences.
It was only the third time Germany had lost its tournament-opening game after defeats against Algeria in 1982 and Mexico in 2018. In the other World Cup openers for Germany, the team had won 13 matches and drawn four.
Despite giving away the penalty for a clumsy challenge on left back David Raum, Japan goalkeeper Shuichi Gonda made a string of saves and was player of the game.
"We fought as a team," Gonda said. "We have to make sure we never stop."
Japan will next play Costa Rica on Sunday, while Germany will face Spain. The Spanish routed Costa Rica 7-0 in the other Group E match.
Germany's buildup was fraught by protests and political statements because of Qatar's human rights record and its treatment of migrant workers and members of the LGBTQ community.
Germany midfielder Joshua Kimmich complained Tuesday that he had the feeling he wasn't able to fully enjoy playing at the tournament because of all the negative coverage.
Germany was playing at the World Cup for the first time since its surprising group-stage exit as defending champion in 2018, while Japan is appearing in its seventh straight World Cup and is looking to reach the quarterfinals for the first time.