Vaughan admits he cannot stand Smith, who captains the World XI team - which includes Freddie Flintoff - in today's Super Series Test with Australia.
And the England captain, married with a daughter and another child on the way, revealed: "Smith was rarely short of a word."
"When we toured last winter, he astounded me by continually calling me 'queer'".
"I found it very odd and childish, the kind of thing you'd say in a playground."
"He also kept calling Flintoff a big baby - which I felt was courageous but remarkably silly.Smith was also involved when I was fined my 6,000 match fee in South Africa for talking about the umpires' interpretation of the bad-light regulations."
"Match referee Clive Lloyd and the umpires wanted to see me and when I went to see them I was told Smith had been called as a witness against me."
"They said it was to disprove my comments about the South African team asking for bad light."
"Smith claimed he hadn't appealed for it, insisting all he had said was that his outfielders were struggling to pick up the ball."
"If that wasn't an appeal against the light then I'm not sure what is, short of trooping off the pitch."
"The feeling in our camp was that South Africa wanted me to get a ban and that's why Smith had gone in there.Whatever the situation, I know that I would never put myself in a position where I would be acting as a witness against an opposing captain."
Vaughan, in his book 'Calling the Shots: The Captain's Story' which is published this week, recalled: "I hadn't criticised Smith or his team."
"In their position, I'd have wanted to come off as well." I just said that the umpires had been inconsistent on the second day of play in Johannesburg.
"I lost my whole match fee and, from that moment on, my relationship with Smith became very frosty. " When I see him doing things like that a loss of respect is inevitable.
"Whether or not he was sent in by their coach, I'm not too sure. "But I just call him 'The Witness' now and I don't think he likes it."