Nadal could not contain a barrage of relentless, brutal groundstrokes from the Czech, 26, on his Wimbledon debut.
Rosol moved into a two-one lead but Nadal fought back to win the fourth.
Play was suspended while the roof was closed and it was Rosol who settled first to win the final set and the match 6-7 (9-11) 6-4 6-4 2-6 6-4.
The result now opens up the draw for Britain's world number four Andy Murray, who was seeded to meet the Spaniard in the semi-final and beat Ivo Karlovic in four sets earlier in the day.
Rosol, who is making his main draw debut at Wimbledon after losing in the first round of qualifying for the last five years, will play Philipp Kohlschreiber next after the German beat Malek Jaziri of Tunisia.
It was the first time Nadal had lost before the third round of a Grand Slam since he was beaten by Gilles Mueller in the second round at Wimbledon in 2005.
Nadal, who won the Wimbledon title in 2008 and 2010, arrived at the All England Club in good form having sealed his seventh French Open title.
Rosol, meanwhile, has never got past the third round of a Grand Slam and has remained a relative unknown since turning pro in 2004, although he was ranked as high as 65 last year.
"I don't know what to say," Rosol told BBC Sport. "I'm not just surprised but it's like a miracle. I never expected something like this. There are so many emotions - I don't know what to say.
"[Nadal] is a superstar and I'm very sorry for him. I played unbelievably today. I hope I can play another match like this.
"I'm very happy for my support. Before the match I was thinking to play three good sets so I don't lose 3-0.