What would you do in this situation?

Keep your cool and stand your ground. Don't show any frustration because that will prove to others that they are bothering you and therefore keeping the social barrier visible. Keep your confidence and keep a positive attitude and ignore the negativity. Don't "doubly" try or appear to be trying too hard because here again, it's keeping the problem at the forefront. Just be yourself, don't show that anyone is getting you down, and either people will cool off and accept you for you, or they won't and they'll be lesser people for it.