Must Everyone Comment on the Size of My Pregnant Belly?

Believe it or not, a few days ago, my sister actually told me, "you don't look pregnant, just like you ate a little too much for lunch." Now, i'm barely five months pregnant, and i know what she was getting at, i can still wear my normal clothes, they're just a bit tight in the waist, but she basically told me i don't look pregnant, just fat. Being my sister, i could tell her that was the LAST thing a pregnant woman wants to hear, and make sure she never said anything like that to anyone other than a sister. Unfortunately, most of the people who comment on my size are not my sister, and i don't understand why they feel the need to say anything at all.

I'm not overly skinny, nor overly heavy. I think I'm about average, probably since my clothes are a size M which is right in-between small and large. But the fact is my first baby at full term weighed only six pounds, so, no my belly did not get extremely huge. I remember people telling me a week before my baby was born, there was no way i was going to have this baby on time, it was too small. While he was small, he was also completely healthy, and is currently above average in height and average in weight. But i often had to remind myself that just because these people who didn't know me, and probably weren't experts in women's health care either, told me i was too small, didn't mean i should feel bad about it. And i think that's unfortunate. When someone asks, "how far along are you?" regardless of the answer, if they're going to follow up, can't they just keep it to something simple such as, "congratulations," or even an, "oh, how exciting!" would be okay. But what constantly comes after this question for me is, "No way! You can't be that far along, i can barely tell you're pregnant!"