To deny access to any one group on the basis of an immutable characteristic of their physical being is a dangerous precedent to set for a government. It just gives a license to discriminate against any out group. I believe you have a right to do whatever you want, so long as doing so does not violate the rights of others.

To take it to a logical extreme, would you defend the right to drink and drive, given that stupid people should be allowed to do stupid things, even if it is incredibly dangerous to the drinking party and everyone else around them? No? Then don’t tolerate the intolerance of others. That’s how the social contract frays.