South Africa were 165 for four at tea on Tuesday.
South Africa, weakened by the absence of star batsman Jacques Kallis because of a back injury, lost their first three wickets for 28 runs as their top order failed for the second successive match.
Gibbs made up for his pair in the first Test humbling by the Indians as he made a stroke-filled 63 before he was the fourth man out but Prince - who made a valiant 97 in the second innings of the first Test - was unbeaten on 47 at tea.
Opening bowlers Zaheer Khan and Shanthakumaran Sreesanth took two wickets each. The left-armed Khan took his wickets in an opening spell of two for five in six overs.
South African captain Graeme Smith continued a dismal run of form when he was out for five to start a collapse similar to that suffered by the host nation in both innings when they were beaten by 123 runs in Johannesburg in the first match of a three-Test series.
Smith top-edged a pull against Khan, the ball went high in the air and Sachin Tendulkar ran back about 15 metres from first slip to take the catch.
Hashim Amla played around his pad and was leg before to be Khan's second victim before AB de Villiers, moved up the batting order to partner Smith, was caught at first slip by Tendulkar when he edged an attempted drive against an out-swinger from Sreesanth.
Sreesanth, man of the match in the first Test, bowled a first poor spell, but struck with the third delivery of his second spell when he switched ends to replace Khan at the northern Umgeni end of the ground immediately after the drinks break.
Gibbs hit 13 fours in his 88-ball innings and played some fluent strokes interspersed with some loose ones. Just when it seemed he was getting on top he top-edged a pull against Sreesanth and wicketkeeper Mahendra Singh Dhoni dived to his right to hold a good catch.
While Gibbs provided the entertainment in a stand which lasted 109 minutes the left-handed Prince dug in as he had in the second innings of the first Test. He had an escape on 41 when he edged VRV Singh low to Tendulkar at first slip.
Gibbs, who had been expected to bat at number six, replaced Kallis at number four in the order.
A team spokesman said Kallis woke with a sharp pain on Sunday and had difficulty moving. Kallis made centuries in each of the previous four Tests at Kingsmead. His absence further weakened a vulnerable South African top order.
All-rounder Andrew Hall replaced Kallis.
South Africa were already without fast bowler Dale Steyn because of a thigh strain. Fellow fast bowler Morne Morkel, 22, won his first cap.
Morkel is the younger brother of Albie Morkel, who has played in one-day and Twenty20 internationals for South Africa.
India named an unchanged team, deciding that fast bowler Munaf Patel was not fully ready to come back into the side after an ankle injury despite bowling well in a two-day practice match.