Liverpool made their early dominance count and went ahead midway through the first half as Robertson's header from Fabinho's lofted pass sailed over Nick Pope in goal.
The Reds had plenty of chances to double their advantage before the break as Mohamed Salah and Sadio Mane went close on more than one occasion.
But Burnley hung on and showed even more grit in the second half as they levelled and earned a point through Rodriguez's sharp finish from a set-piece.
It was the first time the Reds have dropped points at Anfield in over 18 months and, speaking after the game, Klopp admitted it was frustrating to watch.
"It was a really good game but we left the door open for Burnley and we should have closed it. We should have scored two, three or four goals at least," Klopp said.
"The referee let lots of challenges go so it was clear that if the ball comes into the box it was dangerous. They did what they are good at and I respect that."
The Reds boss singled out Burnley goalkeeper Pope for his inspired performance after the Englishman made some crucial saves to keep the visitors in the game.
"Of course it is frustrating, we should have scored much earlier. We warmed Nick Pope up and at the end he was properly really warm, he made some sensational saves," Klopp said.
"It's our fault because we should have closed the game but we didn't do. Burnley is a team who stay in the game and they have two top strikers who constantly go for each ball.
"It's an intense period and at the end we were pushing up and they are always dangerous. That's why you have to take your chances.
"But we stand here with only one point and that is not exactly what we wanted."
He continued: "For moments it was Liverpool against Pope, he did really well, but we should have scored. We did everything right and he made saves but we should have scored more, that's on us.
"We didn't close the game and they took their moment. It feels like we lost a game, we have to look after the game better.
"We were angry with the referee but we have to criticise ourselves first for not finishing the game.
"The team do it right 99% of the time but I will never stop criticising them."