Nah. It's more like all the disparaging memes about gingers or albinos. There's much bigger differences between the races of humans than between different breeds of horses, let alone different colors of the same breed.
You have chestnut hair, right?Please don't come any closer.