In 2005 the Australian skipper became the world's best-ranked batsman and played his 100th Test.
In one of the closest counts of the award's seven years, Ponting pipped Mike Hussey by three votes to become the first dual medallist, having won also in 2004.
Ponting, 31, polled 83 votes from players, media and umpires to beat Hussey (80) by three votes, while rejuvenated fast bowler Brett Lee finished third on 77 votes.
Upon accepting his award, Ponting also had a message that he wanted the Ashes back.
Ponting got an extra dose of motivation to win the Ashes back from England from November this year, courtesy from a tongue-in-cheek television segment from former England bowler Phil Tufnell, which riled the Australians.
In the segment Tufnell perched at a bar had a shot at Ponting's side "dropping the Ashes" like South Africa's Herschelle Gibbs did the World Cup in 1999.
"Warney ... just because you're a mate of Kevin Pietersen's, it didn't mean at that last day at The Oval you had to drop him twice," Tufnell teased on Monday.
"Do you wake up in the middle of the night thinking you might have dropped the Ashes?
"I have got Herschelle Gibbs' phone number here if you want some counselling."
Ponting was not impressed.
"Quite a few guys at my table were pretty fired up at that," Ponting said as soon as he got up to the podium.
"I'm not sure whose idea that was, but it certainly made us a bit hungrier to take on the English who are coming here in 290 days time, so we're looking forward to that."
Legspinner Shane Warne won the Test player of the Year award on the strength of a superb Ashes tour last year and an outstanding past 12 months, which reaped 93 wickets.
Hussey won the limited overs award on a count back after he, Andrew Symonds, Adam Gilchrist and Lee finished tied on 22 votes.
The Allan Border Medal had its first ineligible winner, when Symonds - who would have claimed the one-day award with six three-vote matches - was ruled ineligible because he was suspended for a drinking binge during last year's Ashes tour.
Former Australian captains Bob Simpson and the late Monty Noble were inducted into Australian cricket's Hall of Fame.
Although 2005 was dominated by the Ashes loss, Ponting's individual form could not be questioned.
He hit 1,596 Test runs with seven centuries and also scored 1,137 runs in One-day internationals, with another two hundreds.