A letter from 12 white Test players to the Zimbabwe Cricket Union (ZCU) has called for a shake-up of the way the game is managed following the dismissal of captain Heath Streak, who was sacked when he called for the removal of two selectors without first-class experience.
"If a power like Australia does pull out it could be the end of cricket for Zimbabwe - that's the last thing we want," Gilchrist said in Sydney on Wednesday.
"It would be a long way back, if at all. There are several countries around where cricket is under pressure from various factors and we don't need to be dropping off and losing teams."
Gilchrist's stance supports that of Cricket Australia (CA) which has declared only a heightened security risk will stop the world champions' tour proceeding.
World cricket's governing body International Cricket Council (ICC) had said on Tuesday that it would not interfere in the race row threatening to derail Zimbabwe cricket.
ICC president Ehsan Mani said it was powerless to intervene in the internal affairs of one of its members.
Gilchrist said it would be disappointing if Australia was to face a severely-depleted Zimbabwean team.
"Ideally, every team would have its best team on the field - but it's hard for me to say, I haven't grown up in the countries where these issues are such high priorities," Gilchrist said of the issues confronting Zimbabwe cricket.
"From a cricketing perspective it would be great to have guys like Heath Streak and Grant Flower and so on still playing because they have done so much for Zimbabwean cricket."