South Africa were 148 for five at tea on Monday, an overall lead of 97.
Kallis made a dogged 62 and put on 67 for the fifth wicket with AB de Villiers before he played a rash pull against medium-pacer Scott Styris, the seventh bowler used by New Zealand, and was caught by Daniel Vettori at fine leg.
The dismissal came when South Africa were clawing their way back into the match after being in deep trouble when they lost their first three wickets before wiping out a first innings deficit of 51.
South Africa's problems started when out-of-form captain Graeme Smith was trapped in front of his stumps by Chris Martin for seven in the third over of the day when just four runs had been added to South Africa's overnight total of four for no wicket.
Smith's opening partner, fellow struggler Herschelle Gibbs, failed for the second time in the match when he was caught at second slip off James Franklin for two.
First innings top scorer Boeta Dippenaar started promisingly but when he was on 16 he got a ball from Jacob Oram which kicked and he edged a low catch to New Zealand captain Stephen Fleming at first slip.
As on the second day, the new ball deviated alarmingly at times during the morning period, with uneven bounce and sideways movement.
Conditions improved for batting in the afternoon but South Africa lost Ashwell Prince soon after lunch when he flashed at a lifting ball from James Franklin and was caught behind for 11.
Kallis and De Villiers stabilised the innings and both were looking comfortable when Fleming called on Styris to bowl for the first time in the match.
Stryis' third ball was short and outside off stump and Kallis went for the pull shot but top-edged a high catch to Vettori.