Of course you have to. That’s how ableism works. People don’t assume that you can’t do what you’re asking, they assume you’re refusing. That sucks, but it’s the world. In the meantime, of course you have to explain yourself to people. Either that, or live in a bubble. That’s how human relationships work.