Mashrafe Mortaza trapped Aaron Redmond (two) leg-before in the third over before debutant Mahbubul Alam bowled Jamie How (eight) in the next which left New Zealand at 10-2 after the home team won the toss and elected to field.
Jesse Ryder (18) and Brendon McCullum (one) were unbeaten at the break on a day when play finally started following complete wash out on the first three days due to rain and bad weather.
Bangladesh's third seamer Shahadat Hossain then removed a dangerous looking Ross Taylor in his first over when the batsman left an incoming delivery and was bowled. Taylor made 19 with three boundaries.
Bangladesh, who have no realistic hopes of forcing a win to square the two-Test series after losing the first Test by three wickets at Chittagong, gave Alam his Test debut to bolster their pace attack.
Alam replaced off-spinning all-rounder Naeem Islam from the first Test line-up.
The tourists were forced to make one change from their winning squad, bringing in Grant Elliott in place of all-rounder Jacob Oram who returned home due to a back problem.