Personally, I have nothing against anyone of any sexuality (paedo's aside of course)- its genetic apart from anything and not a matter of choice.
I think the reason a lot of blokes worry about being hit on by gay men is because, as a man, we know just how 'up for it' blokes tend to be. So in a roundabout sense, we judge others by our own standards.
That's my theory anyway.
My philosophy: if it harms no-one, do as you will.