Ponting's comments follow South African wicketkeeper Mark Boucher's urging of South African crowds to give the Australians a hard time in response to what he and his team-mates received on their recent tour of Australia.
South Africa had a terrible tour of Australia losing the Test series 2-0 and failing to make the triangular series final amidst a poisoned atmosphere in which several South African players were taunted in Afrikaans as "kaffirs" or "kaffir brothers", racially derogatory terms for a black person.
Ponting said just before the team departure Monday that he was puzzled by Boucher's view that he had lost respect for some Australian players during the recent series.
"To tell you the truth I'm not really sure where Mark's coming from there. It's disappointing to hear those sorts of things," Ponting told reporters at Sydney Airport.
"It was a tough hard series. No doubt about that. But I don't think there was anything spiteful on the ground," he said. "I think at the end of the day the Test series was played in the right spirit."
Boucher was in no mood for compromise as he told The Wisden Cricketer magazine: "I hope our public give them a bit of stick because we've taken a serious amount.
"In the past our crowds haven't been too great with them but trust me, we're not going to sit back and say,'shame, poor things'."
Boucher said that talk of the racist abuse in Australia coming from South African emigres was utter nonsense.
"The Australian press are trying to say ex-pat South Africans are the culprits," said Boucher.
"Well, years ago Brian McMillan and I ran after a guy who was abusing Makhaya (Ntini) at the SCG. He was definitely Australian. They know what gets to us and as long as it is within the boundaries of the expected that's fine.
"Once they step over those boundaries it is not acceptable and you have to take a stand as a team. Obviously it's not the majority of the crowd. It's the guys who have had too much liquor and they're trying to be smart arses," he said.
"So we understand it's difficult to control. But we've made our point."
Ponting said after a tense series in Australia his team was expecting a hot reception from South African crowds, and he suggested friendly banter to defuse any hostilities.
"I've always tried to have a bit of fun with the crowds around the world - try to get involved with them and have a bit of fun," he said.
"I understand it can be a bit hard at times when some of the stuff coming back at you is not that much fun. We have to be really sensible with the way we handle it," he said.
"The guys who are fielding on the boundary, it's important that they are being sensible with it as well. It can be something that could distract our players at different times.
"We'll talk about it in the lead-up to our first game over there and make sure that we're doing the best that we possibly can to deal with it."
Ponting said discarded Test fast bowlers Jason Gillespie and Michael Kasprowicz were the front-line candidates to take Glenn McGrath's place if, as expected, he missed next month's Test series in South Africa.
While chairman of selectors Trevor Hohns has been non-commital, Ponting made it clear he wanted one of his old warhorses back in the attack for the three-Test series starting on March 16.
"We're going to need some experienced guys back in the Test side if Glenn doesn't come back, so I'm sure their names will be at the top of the list," Ponting said.
McGrath, whose wife Jane has suffered a recurrence of breast cancer, has been told he needs to play a domestic match before he can be considered for the Test segment of the tour.