I think racial prejudice gets caught up on self-serving people of color vs white people constructs. Racial prejudice tends to legitimize acts of colonialism, which historically includes the taking of slaves and/or creation of an economic underclass. That’s not unique to Black folks in the US and it’s not unique to POC/white interactions.