A frantic first half saw Rangers take the lead when Steven Davis struck a low left foot shoot in off the post past Forster from 20 yards.
Celtic replied 6 minutes later as Georgios Samaras nodded in Izaguirre’s cross.
Both sides went close in the second half, but it was Jelavic who found the winner in the 98th minute, latching onto a quick Weiss free kick and slotting home.
Celtic saw red minutes before the final whistle for Izaguirre’s professional foul on Vladimir Weiss.
The victory marks Walter Smith’s 20 trophy with Rangers, with Smith resigning at the end of the current season, as both Old Firm clubs chase top spot in the league.
