Manager Craig Levein was left furious with his side, who passed up some excellent chances in front of goal in the first half and then had two players dismissed in the second.
Levein now faces a tricky run-in without regular Gary Kenneth who was sent off in the closing minutes of the match for a second bookable offence. While Danny Swanson also faces a ban after picking up a red card.
The visitors made the better start to the match and should have taken the lead early on.
Christophe Berra turned away a Christian Kalvenes effort at the back post after Jose Goncalves chose not to close down the Terrors man.
And Noel Hunt passed up a certain scoring opportunity when he completely missed the ball six yards out from goal. Craig Conway had whipped an excellent cross into the area, but Hunt fluffed his chance in front of the net.
Levein's men were punished for missed chances when Kingston found the back of the net on 26 minutes.
The diminutive Ghanaian had looked out of sorts up to that point in the match, but that all changed as he netted his first goal since the end of November.
Eggert Jonsson threaded the ball through to Kingston, who swept a right-footed shot across goal and just inside the right-hand post.
Saulius Mikoliunas could have doubled the Jambos' lead five minutes before the break, but keeper Lukasz Zaluska did well block the Lithuanian's shot.
United again started the second half well and Mark De Vries tested former team-mate Berra early on, as the Hearts defender managed to get a foot to his shot to give away a corner.
With first-choice goalkeeper Steve Banks suffering with a virus, Anthony Basso was deputising in goal from Hearts and he had a nervous moment at the corner as he spilled the ball and was luck to gather again with Tangerine shirts gathering around him.
Mikoliunas passed up a second great scoring chance at the other end, this time skying the ball from six yards out.
While Hearts didn't make the most of their chances, United rarely looked like getting back on terms. Things went from bad to worse for the Tangerines as Kenneth was given his marching orders for a second bookable offence ten minutes from the end.
And he was quickly followed off the pitch by substitute Swanson, who was sent off for a bad challenge on Christian Nade.