How do I stop caring what other people think?

If they are true friends you will not lose them over the guy you choose to go out with. A true friend should stand by you and be happy if you have found someone that you enjoy spending time with. You are the only one who should decide whether he is the right guy or not since you know him the best. Feel good about yourself and your decision making and go with your gut and your heart. If these are real friends that are worth keeping, they will be on your side.