Gibbs was appearing before International Cricket Council match referee Chris Broad after South Africa completed a seven-wicket win in the first Test against Pakistan Monday, following a complaint by the Pakistan Cricket Board.
Television viewers heard Gibbs making insulting remarks about Pakistan supporters when a stump microphone picked up what he said to team-mates in the centre of the field.
He is also due to appear before a Cricket South Africa disciplinary commissioner Tuesday in connection with the same incident.
Smith said he could not condone Gibbs' action but he made strong comments about the circumstances which led to the incident.
"There's a lot of anger," he said. "There was quite a lot of racial abuse from the Pakistan fans towards our players and I think that provoked a lot of what happened.
"There was a particular incident where I went down and got security to move guys away from (boundary fielder) Paul Harris. Herschelle was down at third man and he was copping a lot of abuse and I think even racial abuse.
"For me going forward the worrying thing is that Pakistan always have a large support base around the world. Security needs to be looked at.
"There was an incident where Makhaya (Ntini) was hit on the head by a Pakistan flag going up the stairs. The guys were provoked and that is why they are angry but we understand that what Herschelle did was wrong."
Smith said the television company responsible for the stump mike were to blame for allowing Gibbs' comments to be broadcast. "I think the television companies are actually in breach of their contracts with the stump mikes being on during overs.
"I think they're only meant to be on when the ball is about to be bowled. The television companies need to answer for that. That is disappointing but I guess, you know, they were on."
Cricket South Africa chief executive Gerald Majola said he understood that in terms of the complaint from the Pakistan team management, Gibbs would be charged with "using language that is obscene, offensive or of a seriously insulting nature to another player, referee, team official or spectator."
He said CSA had charged Gibbs with contravening a clause in the body's code of conduct relating to verbal abuse or conduct "on the basis of race, religion, colour, descent or national or ethnic origin." The hearing will be conducted by former judge Mervyn King.
Majola said in a statement: "CSA has investigated the matter and we have heard the remarks made by Herschelle Gibbs in response to verbal abuse directed by a number of Pakistan supporters at Paul Harris while he was fielding on the boundary.
"Herschelle says these remarks were for the ears only of his team-mates in his proximity, and were directed in general terms at that section of the crowd that had verbally abused Paul Harris. He has apologised if he has caused offence to anyone."
Majola said the action was taken in terms of CSA regulations which stemmed from the International Cricket Council's anti-racism policy.
"CSA will continue to enforce these regulations rigidly, and we make a special plea to spectators to support their teams in the right spirit of the game," said Majola.
It was reported from Pakistan that Saleem Altaf, the PCB director of cricket, had confirmed that a complaint had been made. Altaf said both the Pakistan and South African boards had copies of tapes of the remarks.