it seems rather racist

That was the core reason it was struck down. It fell afoul of the Equal Protections Clause. You can’t treat people differently based on their skin color.