No, because this harm is not greater than that which people encounter in everyday life
People on average got annoyed on even small things such as a missing pen or dirty laundry.
When conducting a study, the refusal to participate from the subject would not be considered as a violation as long as you don't force them any further.