Firstly, note that I'm assuming you're opposed to racism and homophobia. If not, trying to reason with you would take too long, and probably fail anyway.
Now then: In olden times, 'you're a white man' was an oft-used compliment to mean you're a good person. The etymology of this phrase traces back to the supposed fact that blacks are not good people.
This phrase is generally considered racist, if for no reason other than the root of its meaning. You may very well have said it to a black guy. Clearly you didn't mean they were literally white, just that they were good, kind, generous, &c.
Same with calling something 'gay' to mean bad. You might say someone is gay even if you know he's not a homosexual. You just mean they suck (figuratively).
Thus, using 'gay' as an insult is analogous to using 'white man' as a compliment. If you consider one okay, you should consider the other okay as well.
I, and most people, consider 'white man' a bad compliment, and I don't recommend using to compliment anyone, especially someone who's black.