The equalities minister, Kemi Badenoch, has been criticised after leaked messages revealed she claimed not to “care about colonialism”, amid warnings that Conservatives could haemorrhage support from the black community.

Badenoch, whose brief was recently expanded to include a junior ministerial position in the Department for Levelling Up, Housing and Communities, reportedly wrote: “I don’t care about colonialism because [I] know what we were doing before colonialism got there. They came in and just made a different bunch of winners and losers.

“There was never any concept of ‘rights’, so [the] people who lost out were old elites not everyday people.”

The leaked WhatsApp messages were revealed by VICE World News this week, and were posted on a group chat called Conservative Friends of Nigeria.

Funmi Adebayo, a former member of the WhatsApp group, said she leaked the messages following Badenoch’s promotion, which includes an additional portfolio in the Foreign Office.

Adebayo, the founder and chief executive of Olorun, which produces the Black Monologues podcasts, warned that “dangerous” comments such as Badenoch’s would drive black supporters away from the party and government.

“What’s going to happen is what happened with me, and I left,” she said. “When I looked across the political spectrum at the time, there were very few young Nigerian outspoken politicians, and I emailed all of them and Kemi was the only one who was willing to actually meet me.”

She said she was drawn to the party because she wanted to bring about change and it was the party that was in government. She added the party was also making a concentrated effort to include people from different backgrounds in positions of power.

But she left after disagreements with the equalities minister. “I told her that I couldn’t be associated with her any more, which is why at the time I left … If this is what it takes to actually have a seat at the table then I don’t need to be there, because I’d rather be able to sleep at night knowing that I’m not selling my soul for the sake of anything.”

She added: “I’m no stranger to sitting opposite people who have completely different opinions to me … But we can connect as Nigerians, and be honest about the fact that colonialism had an impact on Nigeria and that it was awful. It wasn’t as simple as winners and losers; I think it’s such a crass way to respond. Those losers are people who died, and were murdered and raped.”

Adebayo said that divisive comments like these are forcing people who “genuinely wanted to create change, which is why they actually engaged at all, to completely walk away from the years of work that they’ve done”.

She pointed to Samuel Kasumu, No 10’s former race adviser, who resigned and accused members of the government of waging a culture war that is endangering the country. “It’s not easy to become the adviser to No 10, to walk away from that role. It’s untenable.”

She added: “Surely we can all come around the fact that we’re all black and we all want to see life better for black people. We can have different opinions about how we get, but let’s at least agree there’s a problem and we need to address it.”

Simukai Chigudu, a professor of African politics at the University of Oxford, described the minister’s comments as ahistorical. “At its most fundamental level, it doesn’t make any sense. How are we understanding and interpreting rights? Does she have sufficient knowledge of the culture and the languages, and the diverse social and political formations in different regions and groups throughout the continent?

“The other thing is that it’s just not true. An awful lot of work by historians and anthropologists has shown different iterations, forms and concepts of what rights looked like going deep into the African past.”

Hakim Adi, professor of the history of Africa and the African diaspora at the University of Chichester, said: “The orientation of all governments has been to support the interests of the rich and powerful, now and in the past. To them the empire was a ‘remarkable achievement’ to be celebrated just as they celebrate human traffickers.

“The minister has a Eurocentric view of Africa. She imagines Africans had no conception of rights and the removal of the right to determine their own affairs was of no importance. On the contrary, Africans formulated the first modern conception of human rights. They gave their lives to rid Africa of colonial rule and today still struggle to remove all the vestiges of colonialism and foreign intervention, which remain a blight on the continent.”

A government spokesperson said: “The government does not comment on leaked private correspondence.”