I would recommend answering in a full sentence, not just with a "yes" or "no":
so:
You don't mind being here?
Answer: No, I don't mind being here
or : Yes, that is correct: I don't mind being here
or: no, I actually do mind a bit. can we go somewhere else please?