Ponting became the first Australian captain to lose the Ashes since 1986-87 when his lineup went down 2-1 in the five-match series that finished earlier this week in London.
While England's win sparked euphoric celebrations in England, the performance of No. 1-ranked Australia in general has been heavily criticized.
Lillee, an Australian fast bowling great, suggested 36-year-old legspinner Shane Warne should be made captain to give the squad more leadership and give Ponting time to concentrate on his batting.
"Everybody is entitled to their opinion, but the thing that gets me down most is that I don't think Dennis Lillee would have one idea of what goes on around our team,'' Ponting told Saturday's The Australian newspaper.
Lillee "hasn't been any help to any of the guys when it's been needed over the last few years - it's easy to make judgments from a long way away.''
Ponting returned to Australia on Wednesday, when Lillee's opinions were published in Western Australia. Lillee wasn't alone among leading former internationals in criticizing the team, although his comments were the most pointed. But Ponting remains defiant.
"As long as I'm doing everything right by my teammates and by everyone concerned with Australian cricket, I will always feel I'm doing my job right,'' he said. "For Dennis to sit back and say I should be sacked and be replaced by Shane Warne, it hurts to a certain degree.''
He also defended John Buchanan's role as coach, but expressed the need for more expert assistants on staff, following England's lead.
"One of the big things that stood out to us was the resources (England) had available right through the series,'' Ponting said. "There's no doubt they've studied what we've done over the last three or four years with the structure around our team and coaching staff."They've had a really good look at our model and made it even better, turning theirs into the leading model in world cricket.''
Ponting said the Australian bowlers and batsmen would have benefited from having specialist coaches on the tour.
"If you look at the way some of our individuals struggled through the tour, it's a pity we didn't have any expert coaches there or available to help those guys out.''
Buchanan, who is out of contract next month, had one assistant coach, Jamie Siddons.England coach Duncan Fletcher had an assistant plus separate batting and bowling coaches as well as other resources.
Although he agreed the coaching staff could be bolstered, Ponting defended Buchanan's management style.
"He's probably approached things a different way to some but he's certainly taken the Australian cricket team to a place it has never been before - I think we can all get back there again with (Buchanan) as coach.''
Leading into the Ashes, Australia had won six consecutive series after Ponting, 30, replaced Steve Waugh as captain - starting with a 3-0 win in Sri Lanka last year.
"We've had an unbelievably good run over a long period but it's all changed fairly dramatically over the last few weeks,'' Ponting said. "It's a strange feeling for all of us."We all understand how much the Ashes means to a lot of Australians because it certainly means a lot to us.
"None of us have ever played in a series where we have lost the Ashes, obviously, and some of the guys have never even played in a losing series full stop - the good thing is the next (Ashes) series is less than 18 months away, which is exciting.''
Despite an ongoing review by Cricket Australia, Ponting didn't anticipate wholesale changes when Australia regroups for the three one-day internationals and six-day super test against the Rest of the World XI next month.