To refuse would be... or whatever. The full saying is 'to refuse a woman's advances would be a man's shame.'