My family doesn't like the guy I'm seeing.. for no reason?

Okay, so I'm 19 and I live at home with my mum and sister. I've been seeing this guy for about 3 or 4 weeks and he's come round to my house about 5 times, I haven't been to his yet. My mum has said that she doesn't like him - she hasn't even had a conversation with him other than small talk?! She's annoyed that I haven't been to his and he always comes here, he said his mum is 'overprotective' and 'nosy' and he hasn't exactly said he doesn't want me to go to his, but he seems uncomfortable about it. Also, I'm more comfortable staying where I am, because the meeting parents thing really makes me nervous. My mum seems to think that he comes here because he can do what he wants, which I don't understand because he doesn't do anything that he couldn't do at home and he's always very polite. My sister, who hasn't even been in the same room as him let alone said a word to him, says she doesn't like him because he 'smells' (obviously not true) and he's loud and he has small feet?! Not really good reasons to take such a strong dislike to someone. I really like him, we get on so well and it's the first time I've been this happy in a long time, which is why I'm really upset that my family can't just be happy for me and not try and pick at him for no reason. When he comes, we literally spend all our time in my room and so it's not like we're treading on anyones toes. I guess my questions are, do they have a right to be so nasty about someone they don't even know? If you live with people, is it their business as much as yours who you see? If I confront them, what do I say?