Bodies reportedly lay strewn on the streets of the town following an assault by the Islamists on Wednesday, with hundreds feared killed.
The town is near the border with Chad and had housed a military base staffed partly by multinational forces.
Soldiers from Niger had been there but were not present when it was attacked.
The BBC's Nasidi Yahaya in Abuja says the decision of Niger is clearly a big blow to Nigeria, which had said that the presence of a multinational task force in the area would help defeat Boko Haram.
The withdrawal of the troops from Niger means Nigeria's neighbour now has no forces directly involved in tackling Boko Haram.
Chad has also withdrawn its forces.Nigerian officials told the BBC that almost all of Baga had been set on fire and militants had raided the surrounding area.
Niger Foreign Minister Mohamed Bazoum told the BBC Hausa service: "We have 50 soldiers there and decided to withdraw them after Boko Haram captured Malamfatori town in October and continued to operate in the area with impunity.
"As you know, Baga is under [the control of] Boko Haram terrorists and unless the town is recaptured from them, we will not send back our troops.
"But we are still determined to work with our neighbours Cameroon, Chad and Nigeria to contain the situation - it is a problem for us all."
'Burnt down'
A separate French-led initiative called for all four West African countries to contribute 700 troops each to a multinational force against Boko Haram, but no country has implemented the plan.
Niger has blamed Nigeria, saying it has not kept to commitments regarding its own troop levels.
Civilians who fled from Baga this week reported that they had been unable to bury the dead. Corpses littered the town's streets, said Musa Alhaji Bukar, a senior government official in the area.
He said the town, which had a population of about 10,000, was now "virtually non-existent".
"It has been burnt down," he told the BBC Hausa service.
Boko Haram launched a military campaign in 2009 to create an Islamic state.
It has taken control of many towns and villages in north-east Nigeria in the past year.
The conflict has displaced at least 1.5 million people, while more than 2,000 were killed last year.
Nigerian lawmaker Maina Maaji Lawan said Boko Haram now controlled 70% of Borno state, which has been worst-affected by the insurgency.
Drowned
As well as Baga, Boko Haram has seized 16 neighbouring towns after the military retreated, Mr Bukar said.
While he raised fears that about 2,000 people may have been killed in the latest raids, other reports put the number of dead in the hundreds.
Boko Haram's offensive continued on Thursday, with fighters setting up checkpoints and killing people hiding in the bush, Mr Lawan said.
About 10,000 people are reported to have fled to Chad since Saturday to escape the violence, raising fears of a humanitarian crisis.
Some are said to have drowned as they tried to cross Lake Chad on the border.
Others fled to Maiduguri, the capital of Borno, in buses provided by the government, the senator said.
Nigerian President Goodluck Jonathan declared a state of emergency in Borno and two neighbouring states in 2013, vowing to defeat the militants.
However, Boko Haram has stepped up attacks since then and there are fears that many people in the north-east will not be able to vote in the general election because of the conflict.
