The power to say NO

Isn't it the toughest job- mustering up enough courage to say that all important two letter word.
I had a chance to say NO and I wanted to say that. But as always I didnt feel that i had a good enough reason to say that. Is it lack of courage or just a propensity to come out with my hands clean out of a dirty job??
Somehow i feel guilty about it .. everytime I want to say NO but couldn't do it.