Hair and the other field umpire Billy Doctrove penalised Pakistan on charges of ball-tampering during the fourth Test against England at The Oval last month which led to the first forfeit of a Test match in the history of the game.
Later on, ICC revealed that Hair offered to walk away from umpiring in return for compensation of 500,000 dollar.
''It really upsets me when people describe me as racist, because they have no idea how I spent my childhood and how that shaped my beliefs in adult life,'' Hair, who is on the ICC Elite Panel of Umpires, told Daily Mail.
''Money has never been a driving force in my career as an umpire.
No umpire I know does it for money. I can honestly say I am earning less than half I could have commanded in salary had I stayed in the private sector as sales manager of a clothing manufacturer in Melbourne.
''How can people judge me to have prejudices when I went to school in Australia alongside Chinese children, Hungarian refugees and all manner of other nationalities? ''I grew up in Orange, in central New South Wales, living next door to a large family of Aborigines. I had some of my happiest times playing rugby and cricket with the children.
They were like brothers and sisters to me,'' Hair said.
The 53-year-old Australian who now resides in England expressed his dismay over the vicious nature of some of the comments that has upset him and his wife.
''Amanda has been affected by this. Normally she is very strong, an implacable woman. I think it has affected her because she knows a lot about cricket. No matter how much I tell her not to worry, she does worry because she does not like to see what she perceives as injustice.''
Hair said he suffered sleepless nights with worries over how his 85-year-old widowed and sick mother in Australia is coping with the situation, but said his spirits have been buoyed by hundreds of letters of support.
Insisting that he wants to carry on his career as an umpire, Hair defended his action and said what he did was required to uphold the spirit of the game and on account of the seriousness of the matter.
''Cricket places the onus on captains for them to ensure that the game is played fairly within the laws and spirit of cricket. The umpires intervene only when they decide the game is not being played within the spirit of the game. People will tell you in matches I have umpired in the past 20 years that I have shown considerable restraint.
''I prefer the principle of non-intervention. But the decision to intervene all finally depends on the seriousness of the matter,'' he said.
Hair also claimed that he was well received in the subcontinent and he had no problem officiating there.
''Pakistan is particularly receptive to a visiting umpire like myself. They are very hospitable people. The subcontinent has so many happy memories for me and officiating in that region has been instrumental in improving many aspects of my umpiring,'' he said.