England team management said Giles flew out of Perth late Friday evening.
The left-arm spinner has been replaced in the squad by off-spinner Jamie Dalrymple.
England coach Duncan Fletcher said Giles's departure had had no effect on the team before play Saturday.
"When some of the players heard it last night and it was discussed they were pretty upset, but this morning we just said we've got to rise to the challenge and it's for Ashley that we have to go out there and play some good cricket," he said.
Fletcher said he felt for Giles, who had only just come back into the England squad after 12 months off with injury.
"It's very difficult for the man, having had that operation and especially knowing the man. He's a real professional," Fletcher said.
"I've been involved with England for seven years and I rate him probably the most professional cricketer I've dealt with in England - or in the top three.
"He really wanted to play for England and he got upset when he wasn't, and to come back and take the flak he did and then suddenly this news came along. It must have been very hard for the man.
"But he handled it very, very well and when he left last night we just wished him well and hope everything turns out all right for the family."