The former South Africa coach said he had wanted time off from the game the first time and was not satisfied at the way the offer came about the second time round.
"The first time I was approached was soon after the 1999 World Cup when my contract with South Africa was over, but I wanted some time out from cricket and didn't quite consider the offer," Woolmer said.
The 57-year-old, who is here with the Pakistan team that is touring for two Tests and three One-dayers, said he was interested in the assignment in 2003 but the officials did not have proper discussions with him.
"On the second instance the circumstances weren't right and therefore I declined," the former England player said. "They were making offers of money without trying to sit down with me for a chat."
Although Woolmer's name was rumoured in cricket circles to replace Dav Whatmore three years ago, it was believed at the time that violence in the country had made him decline the offer.
But Woolmer said security was never a concern.
"That's far from the truth. I love this place and my wife loves this place. It didn't worry me one bit. If I am going to die while coaching cricket, then so be it," he said.
Sri Lanka were desperately searching for someone to replace Whatmore after the last World Cup and had an interim coach in former skipper Duleep Mendis before Australian John Dyson was given the job.
Dyson coached Sri Lanka for 18 months before being replaced by present coach Tom Moody, also an Australian.
Woolmer also indicated that he did not have many years left in coaching. "I am getting old for the job. Throwing hard at practice sessions and giving high catches are becoming difficult," he said.
"I am contracted with the Pakistan Cricket Board till the 2007 World Cup and will assess what to do after that."
Woolmer made a name for himself as a coach when he was in charge of English county Warwickshire and later moved to South Africa, where he worked for five years.
He was later appointed as the International Cricket Council's High Performance Manager and took up his present assignment in 2004.