Generally, I would just say let them stare (or glare). Nobody is going to physically attack you so it doesn't put your safety at risk.
I read an blog post from a dad who was worried about the same thing. He had his answer all planned out, it was something like "I'd be happy to trade you places in line if you would trade me and have YOUR child be autistic"

We went to Disney for the day with our three-your old and did not know about the card, we found out about it afterwards and it really would have made our day much easier and much more enjoyable for everyone. There were very few rides that had short enough lines for our son to tolerate - he is also fairly high functioning. He also found the noise a bit upsetting at one point about one hour and a half into the visit but luckily got over it. Otherwise we would have had to go home early which is a shame as Disney is so expensive. I would say if I was to go again we would definitely get the card.

As an aside I have also told airline staff that is he is autistic when they have challenged whether he (at over two) needs extra time to board early.

We face so many challenges as parents of autistic kids I think anything you can do to preserve your sanity and make life more enjoyable for your kid is worth the occasional odd looks from people.

Don't let them get to you, let them stare. It's not you trying to take advantage of anything, it's you trying to let your child enjoy the rids just like everyone else. If you were to stand in the long lines like these people want, then your child would start flippin' and having a melt down and you'd get the nasty stares. You can't win for nothing with some people. I've thought about getting it before too, you do what you need to do and don't let no one get ya down.

We use the pass every time we go and I don't really feel guilty about it. I did at first but it would be hard to do the parks without it. If the line is short we just go on it the typical way but if the line is long we use it. I think if I'm not mistaken they made a new rule that you can only use it if the line is 20 minutes or over so they try to keep it fair to everyone. If it is truly a problem for your son to wait on lines, have no guilt. We never know with Daniel if he will have trouble waiting for the rides so we use it. He doesn't meltdown but he has crying episodes at times and the first few days he has sensory issues with some things there. Disney has been great about the pass. They want us to have a Magical time, don't worry about it at all :-) Have a great trip!

Disney parks have fast pass on many rides, so people are used to seeing others using the exit to bypass lines. We just came back from Disney and only used fast pass. I would have used the special access but wasnt aware of it.