A Combination of steps you should take to combat this problem
* Be aware of the warning signs no matter how small or weird
* Also be aware of what can happen if the matter, is ignored.
* Ask questions an on the kind of person which more likely to cause a scene if the person with invisible difficulties expense
* If you notice they are uneasy, upset or emotionally never tell them off or make and lectures on their behaviour at any level you are making them feel that using their difficulties as blame on something is not their fault & you are humiliating them. In addition of the bully notice your interaction with the affected person with a signal thinking they have won and seen this as an encouragement to escalate the abuse further.
* Non-verbal insults aim to hurt aimed cause indirect insult. If you don't notice this and the person affected has stood up the offenders. Never tell a person with difficulty off for exercising their dignity or to tell them they are causing trouble. Also never create an argument by accusing them of embarrassment. Not only you let the offender win, the person with difficulty will be more likely not to be comfortable with spending time with you and they will not forget the hurt you have escalated.
* Never use your impatience to push, shove or to grab a person with invisible difficulties body. Also, you should never disrupt what person with invisible difficulty when they are undertaking a practical task or by grabbing items of them during any practical actions (especially when they are not causing physical harm to you to them or to the surrounding environment). Instead, ask them to stop and communicate any corrections needed or to make suggestions calmly shortly after the situation happened and don't create an attitude to use them upset further in any way.
Remember the person with difficulties is not intended to irritate you or to test your patience.