Pompey suffered a 2-0 defeat at the hands of Aston Villa on Saturday.
“(The match was) very frustrating,” the Pompey boss told Sky Sports.
“I thought up until the first goal there was not a lot in it, we were holding our own and looking like we were getting a grip on it and then two defensive mistakes that are unexplainable. I can’t get my head round that one.”
“We got a grip on it second half and played really well and really, apart from our mistakes it’s either a 0-0 or we might nick it,” he said.
“You just can’t account for that, you can’t legislate for that and it’s inexcusable.”
Hart added: “We played well throughout the game, we created quite a lot in the second-half, we created a few things in the first-half.
“We’re entitled to come here – a team that’s expected to be in the top four – and get a grip on the game, which we were getting a grip on and we’ve let ourselves down again.”