The winger set up United's first three goals before producing a wonder strike to seal the victory.
Scott Severin scored his first goal for United after 25 minutes with a downward header before David Goodwillie nodded home seven minutes before the break.
Goodwillie doubled his tally when he headed home two minutes after the restart before Conway netted with a wonderful strike after 49 minutes.
Substitute David Silva pulled a goal back for Killie with a long-range strike before Manuel Pascali nodded home a second in the dying stages, but United held on for a deserved win.
The result moves United eight points clear of fifth-placed Kilmarnock, who have now gone four matches without a win under caretaker boss Kenny Shiels.
Source: PA
Source: PA