In a Super Eights match reduced by a wet outfield to 22 overs per side, McGrath took three wickets for 16 runs in five overs to move two clear of Pakistan great Wasim Akram's previous record of 55 World Cup wickets.
Chasing 105 for victory, after restricting Bangladesh to 104 for six, Australia finished on 106 without loss, victory achieved with more than eight overs to spare as they made it five wins out five in this World Cup.
Adam Gilchrist was 59 not out and Matthew Hayden, who finished the match with a six off Saqibal Hasan, 47 not out.
McGrath, 37, put himself in the record books when Aftab Ahmed failed to clear the tall Nathan Bracken at mid-off and was caught for 11.
The New South Wales seamer, who has appeared in the last three World Cup finals and will retire from cricket after this tournament, broke Akram's record in five fewer matches than the left-arm legend (33 compared to 38).
His achievement Saturday was all the more creditable because the delay meant he only had a maximum of five overs. Yet in four he'd taken three for nine.
Defeat for Bangladesh in their Super Eight opener was made worse by the sight of pace bowler Tapash Baisya leaving the field after pulling up in his delivery stride.
Gilchrist completed his 50th one-day international fifty in 41 balls with seven fours when he off-drove left-arm spinner Mohammad Rafique for six.
Earlier, Bangladesh were indebted to an innings of 25 not out from Mashrafe Mortaza and 25 from Hasan.
After the start was delayed by five hours it was no surprise when Australia captain Ricky Ponting won the toss and fielded in a bid to exploit any moisture still in the pitch.
Teenage opener Tamim Iqbal fell when he miscued left-arm quick Bracken to Brad Hogg at mid-on for three.
Four for one became eight for two when McGrath, the most successful fast bowler in Test match history, drew level with Akram when expertly yorking Shahriar Nafees for one.
Ahmed hoisted the wayward Shaun Tait for four over mid-wicket while left-hander Hasan cut the fast bowler for six over third man after the bowler dropped short and wide.
These were two rare controlled shots from a Bangladesh top order that appeared panicked by the prospect of having so few overs to play with.
McGrath then showed Tait how it should be done when Ahmed holed out off a full length ball and Bangladesh were 25 for three in the sixth over.
Mohammad Ashraful made 100 during Bangladesh's shock five-wicket win against Australia in Cardiff two years ago. But this time around he fell for six when a huge skied drive off McGrath was well caught by the back-pedalling Ponting at mid-on.
One concern for Australia was the sight of all-rounder Shane Watson leaving the field four balls into his second over with a left calf strain.
Watson missed the whole of Australia's 5-0 Ashes thrashing of England with a hamstring problem.
Australia, bidding to become the first team to win three successive World Cup titles, won their Super Eight opener against West Indies by 103 runs here Wednesday.
Bangladesh are back in action here Tuesday against New Zealand while Australia have a week's break until they return for their next Super Eights match against England on Sunday, April 8.