You have to find somewhere else to eat and that takes time and effort.
It annoys you that they cater for vegetarians and for different religions, but not for you.
You can't eat out with your friends if they want to go somewhere that doesn't cater for you.
Often you end up in somewhere expensive because they are the only place you can eat at.
You often can't use discounts because they only apply to food that you can't eat.
You feel that you have a disadvantage because everyone else can eat wherever they like.