A boycott by up to 11 Zimbabwe players is seen as the first direct consequence of the simmering crisis over the ousting of national captain Heath Streak, which has split cricket in the African nation along racial lines.
Australia are scheduled to play two Tests and three One-dayers in Zimbabwe, with the opening warm-up game starting on May 17.
Zimbabwe's most capped player, Grant Flower, has warned the Australian tour could prove farcical and told the Sydney Morning Herald on Tuesday that he believed there was a "very good possibility" Zimbabwe could lose their Test and One-day International cricket status.
But CA spokesman Graeme Vimpani said in Melbourne on Tuesday that the uncertainty within the Zimbabwe team would not affect the decision on whether to tour.
"We are committed to the tour contingent on safety and security - as long as those two boxes are ticked we will go," Vimpani said.
"Our position is not to make any comment on what happens in Zimbabwe. We are concerned with our own backyard. It is up to Zimbabwe to work through those issues, it's not up to us."
Australian Test batsman Matthew Hayden said the situation in Zimbabwe cricket was a concern, but he believed the tour should go ahead.
"I think it's taken them and will take them a long time to become a world leader in cricket but that doesn't neglect the fact that we need to take the opportunities to develop that," Hayden said on Tuesday.
"It took New Zealand 27 years to win a Test match -- we need to go over there and show our support for the global game."
Streak triggered the biggest crisis to hit Zimbabwe cricket since it became a full member of the International Cricket Council (ICC) more than a decade ago when he called earlier this month for the removal from the selection panel of anybody who had not played at least first-class cricket, and of a TV commentator.
Streak was immediately sacked and there was no debate about his concerns.