Paul Scully has made a bizarre U-turn after suggesting there are “no-go areas” in parts of London and Birmingham.

The Tory MP and former London mayoral candidate said on Monday that people were “concerned about, more and more, their neighbourhoods changing”.

Scully, who was minister for London until the latest reshuffle, was making the point while criticising Lee Anderson’s comments about Sadiq Khan.

But after a backlash over the “no-go areas” statement, Scully told The News Agents podcast on Tuesday that he is “slightly furious with himself” about his own remarks.

Anderson had the Tory whip removed over the weekend after he chose not to apologise for saying “Islamists” had “control” over London mayor Khan, who is Muslim.

The incident has escalated into a major row over how the government has handled the issue.

Sunak has claimed Anderson is not racist or Islamophobic, but security minister Tom Tugendhat told MPs he was ousted from the party because of “anti-Muslim hatred”.

Anderson has remained unrepentant – and appears to have shifted his position. After initially accepting the suspension, he told Channel 5 News on Tuesday the PM had “made a mistake”.

Scully said: “I think I do regret the use of the word ‘no-go areas’, because I think that actually has other connotations that I hadn’t really taken on board.

“And I’m slightly furious with myself, the fact that actually that’s allowed my message to be misconstrued.

“And what I was trying to say, was that the kind of comments that we’ve heard from Lee Anderson and others in recent days and weeks, the populist approach, sometimes is fuelled by conversations that people have, and perceptions that people have in cities around the UK.

“I mentioned Tower Hamlets and Birmingham, where very small groups of people really misuse and abuse their doctrine in terms of Islamic gangs, but it can be black gangs, white gangs …

“And then people write-off a whole community because of the actions of mindless and slightly threatening small groups of people.

“And so, it’s areas that people feel uncomfortable in on occasion, but that’s the different from a ‘no-go area’ in terms of the wider context, and for that I do apologise.”

Speaking to BBC Radio London on Monday, Scully said: “The point I’m trying to make is if you look at parts of Tower Hamlets, for example, where there are ‘no-go’ areas, parts of Birmingham, Sparkhill, where there are ‘no-go’ areas, mainly because of doctrine, people using, abusing in many ways, their religion because it’s not the doctrine of this land, to espouse what some of these people are saying.”

But Andy Street, the Tory mayor of Birmingham, said Scully’s remarks were “nonsense slurs” about his city.

“I for one am proud to lead the most diverse place in Britain,” he said.

On Sky News, legal migration minister Tom Pursglove also took Scully to task.

He said: “I don’t recognise the depiction that was set out in Paul’s comments, and as government we take it really seriously that we need to help people integrate into society.”