Mashud, 32, was dropped from the team for the World Cup 2007 and his last Test was against Sri Lanka in June last year.
"The board gratefully acknowledges Mashud's immense contribution to the national team for over a decade," Bangladesh Cricket Board said in a statement.
Mashud, also a handy batsman, scored 1818 runs in 126 one-day internationals, with 91 catches and 35 stumpings.
The announcement came as Bangladesh cricket struggles with the sudden retirement of at least 13 players, including seven from the national team to play in the unofficial Indian Cricket League (ICL).
Mashud, whose career highlight was an unbeaten century that helped his side secure a draw in their first Test in the Caribbean in 2004, said he was saddened to quit the game.
"I have made this decision after a lot of soul searching and this is a very emotional moment for me," he said in a statement.
"For the last 13 years I have tried to serve the country to the best of my ability and I thank the board for putting faith in my capabilities as a player and captain."
It was unclear whether he might play in the ICL.