The last wicket fell to the last ball of the day as India claimed a first-innings lead of 41.
Graeme Smith, Hashim Amla, Jacques Kallis and Mark Boucher made half-centuries for the home team but Kumble's haul of four for 117 kept India on top.
Opener Smith was dismissed early when his limp drive to a ball from Kumble was caught by Virender Sehwag at short mid-on.
Smith had scored an aggressive 94 off 142 balls with 14 fours and a six, sharing a stand of 159 with Amla for the second wicket.
Fast bowler Shanta Sreesanth struck another blow for India three overs later when Amla chased a wide delivery and was caught behind by wicketkeeper Dinesh Karthik for 63.
Smith had taken the game to the visitors, driving left-arm paceman Zaheer Khan through mid-on for two consecutive fours.
But only 33 runs were scored in 20 overs before lunch after Smith fell.
Kallis and Ashwell Prince put on 83 for the fourth wicket before the former swept a leg-break from Sachin Tendulkar to Munaf Patel at mid-wicket on 54.
Eleven balls later Prince, on 26, dragged a ball from Kumble on to his leg stump as South Africa slipped to 260 for five.
Herschelle Gibbs was also out for seven before tea, nudging a delivery from off spinner Sehwag to Wasim Jaffer at short leg.
Kumble, Tendulkar and Sehwag tied the South Africans down efficiently enough for captain Rahul Dravid to delay taking the new ball until the 116th over.
Shaun Pollock was next to go for 31 with 10 overs remaining, steering a delivery from Zaheer into the hands of Sourav Ganguly at first slip to end a seventh-wicket stand of 69 with Boucher.
Seamer Patel had Boucher caught behind by Karthik for 50 before Kumble bowled Dale Steyn for one and trapped Makhaya Ntini lbw for a duck.
South Africa lost their last three wickets for one run, leaving debutant Paul Harris 11 not out.