The Dane, who left Brentford to join Tottenham last summer, left the pitch to chants of "sacked in the morning" from Tottenham fans who endured yet another home defeat.
Tottenham have won twice in 11 home Premier League games this season, losing six, and with his team in 14th place he is now fighting to keep his job.
"We seem to be in the perfect storm at the moment. A last-minute defeat and when everyone feels they have given everything, including the fans, and you get a sucker punch, it's unbelievably tough to take for them, the players and me," he told reporters. "I promise it's extremely tough for me as well."
Tottenham trailed to a 15th-minute goal by Crysencio Summerville but after Cristian Romero headed an equaliser it looked as though they would go on to take three points.
But Callum Wilson scrambled in a late winner for 18th-placed West Ham to boost their survival chances.
Once again Tottenham left the field to boos from their fans and they will have to recover quickly as they face Borussia Dortmund at home in a crucial Champions League game on Tuesday.
"I've probably had better times, it's probably not the best time of course. But I understand, I'm the man in charge. So the blame will go to me. That's fair, no problem in that sense," Frank said.
"As long as they are backing the players, doing everything they can, to support them and drive them forward, that's what we do, and we will keep going forward."
It was Tottenham's third successive loss in all competitions and they have now won twice in their last 13 league games.
Frank's job was made harder on Saturday by his starting lineup reportedly being leaked on social media on Friday.
"It's not optimal. I don't think it's the first time in world history or football history, it happened also in other clubs," he said. "Of course, I prefer it not to happen."
While it was doom and gloom for Tottenham, West Ham's first win in 11 Premier League games boosted their hopes of avoiding relegation.
"It's special to win on the last moment of the game," manager Nuno Espirito Santo, once sacked by Tottenham, said. "I'm very proud of the way the players have been dealing with the situation."