The water level tries to be horizontal, even beneath the tilted mug. That means that in the "lower half" of the mug, the water penetrates inside the mug. The pressure and density of the air stays nearly constant, so air must penetrate outside the mug on the other side. Now, it is geometrically possible for the air to escape in the direction of the arrow and create bubbles.
It's actually encouraged to do so, I think, because the pressure of the air is slightly higher than the pressure of water at the beginning of the arrow. Why? Because the pressure is nearly uniform in the whole confined air – the hydrostatic altitude-dependence in the air is negligible due to the air's low density – and it is equal to the pressure of water at the horizontal level beneath the mug. At the beginning of the arrow, the hydrostatic pressure is lower, so the air has the inclination to move to the right in the direction of the water, and it eventually gets outside the "vertically beneath the mug" space and it may escape.
I hope that I don't have to explain why bubbles of the air move "up" in water whenever they can. ;-)
The critical angle at which the bubbles start to from is determined by the surface tension: the bubbles don't want to be too small because of this additional force and a small initial excess of the pressure may fail to be enough to form such bubbles. Approximately when it becomes possible for the minimal sustainable bubbles to form and escape, they will.