Left-hander Siddique scored a brilliant 85 for his maiden half-century to anchor his team's successful chase of a 202-run target in 45.3 overs at Shere Bangla National Stadium after paceman Mortaza took 4-44 to jolt the tourists.
Mohammad Ashraful notched a return-to-form 60 not out to guide Bangladesh to their first win over the Black Caps in 12 one-dayers, as the home team skipper and Siddique added 109 for the third wicket to upset their formidable opponents.
New Zealand's dreams of regaining a second spot in the one-day rankings with a 3-0 sweep of the series were shattered by Bangladesh who faced an uphill task after 14 of their frontline players defected to a rebel Indian league last month.
Mortaza and left-arm spinner Abdur Razzak (3-32) set up the sensational win by restricting New Zealand to 201-9 in their 50 overs.
Jacob Oram top-scored with a solid 57 and added 70 runs for the seventh wicket with skipper Daniel Vettori (30) but that was not enough for New Zealand to avoid their lowest one-day total against Bangladesh.
Their previous lowest total against Bangladesh was 224 at Chittagong in 2004.
Bangladesh lost opener Tamim Iqbal (12) early but Siddique and Mushfiqur Rahim (30) added 67 for the second wicket before Ashraful sealed the fate of the match.
Siddique hit eight boundaries during his 139-ball knock while Ashraful hit five boundaries and a six in his rapid 56-ball innings.
Earlier it was Mortaza who jolted the New Zealand top order after Bangladesh won the toss and elected to field.
Mortaza was brilliantly supported by Razzak as both grabbed five of the first six wickets to fall after New Zealand raced to a solid start of 47 through Jesse Ryder (34) and Brendon McCullum (14).
Bangladesh derailed the tourists once Mortaza dismissed McCullum in the ninth over as five more wickets fell in the space of just 32 runs.
Mortaza dismissed Ryder and Jamie How (seven) in his first spell and returned for his second to dismiss Kyle Mills (16).
Razzak kept the pressure up by dismissing Scott Styris (four) and Daniel Flynn (six) before the Oram-Vettori repair work.
Oram hit four boundaries and a six during his fighting 85-ball knock before he holed out off Razzak in the 47th over.
The two left-handers erased the previous best seventh wicket stand of 43 against Bangladesh, involving Vettori and Chris Cairns at Dhaka four years ago.
The second match will be played at the same venue on Saturday while the final match is at Chittagong on Tuesday.