A crowd in the village square pleaded in vain with the militants to spare the man, Reuters news agency reports.
Last week, a couple who had sex outside marriage were stoned to death in the town of Aguelhok, controlled by the Islamist group Ansar Dine.
The group has also destroyed ancient shrines in Timbuktu, denouncing them as promoting idolatry.
Mali's weak interim government said the amputation in Ansongo - carried out by the Movement for Oneness and Jihad in West Africa (Mujao) group - was a "vile act".
Military intervention in the north was now inevitable, it said in a statement.
' Sharia demands it'
Mujao official Mohamed Ould Abdine said the amputation was in accordance with the "law of God", the AFP news agency reports.
" Sharia demands it," he said.
A local government official said dozens of people saw the amputation, AFP reports.