Keys saved two match points on her serve in the 10th game of the third set, stripping a forehand on the first and escaping the second as Czech Muchova misfired on a service return.
The American emerged with a 4-6, 6-3, 7-5 victory - her third of the season that included saving a match point.
"I'm really happy to get that win," Keys told the crowd in Montreal.
"After losing the first set and even being match point down, being able to figure it out -- it's always a great day."
Osaka did not need heroics as she blew past Anastasija Sevastova 6-1, 6-0 in just 49 minutes.
The four-time Grand Slam champion from Japan notched the second-quickest tour-level win of her career, after a 42-minute 6-1, 6-1 rout of Ana Sofia Sanchez in the first round at Florianopolis back in 2016.
The dominant victory also marked the first time Osaka has conceded just one game or fewer in a completed match since a 6-1, 6-0 victory over Danielle Collins at Beijing in 2018.
Perhaps more importantly, it's the first time Osaka has reached the last eight of a 1000-level event or Grand Slam since Doha in 2024.
Osaka did not play in 2023 after the birth of her daughter and has struggled for consistency since returning to the game a year ago. She fell in the first round of the French Open and the third round at Wimbledon this year and arrived in Montreal ranked 49th in the world.
But she has shaken things up this week, announcing she was parting with coach Patrick Mouratoglou after less than a year and immediately getting to work with Poland's Tomasz Wiktorowski - former coach of Iga Swiatek.
"I had a solid plan coming in here and it just happened to work out pretty well," Osaka said of her dominant performance against Sevastova, a former world number 11 also on the comeback trail after maternity leave who had beaten two-time defending champion Jessica Pegula in the third round.