You're saying that it is the same thing to add or subtract B. This means that B must be the zero vector.
This works with numbers as well, if you feel more familiar to think it this way: in general, if you add something to a number you make it bigger, whereas if you subtract it, you make it smaller.
The only way to cause adding and subtracting to be the same thing is to add or subtract zero: in both cases, you're doing nothing.