well, let's bear in mind that the hypotenuse is never negative, since it's just a radius unit.
now, π < θ < 3π/2, is another way of saying, the angle θ is in the III Quadrant, where as you already know, the sine and cosine as well as opposite and adjacent sides are both negative.

![\bf \pm√(5^2-(-1)^2)=a\implies \pm√(24)=a\implies \stackrel{III~Quadrant}{-√(24)=a}\implies -2√(6)=a \\\\[-0.35em] ~\dotfill\\\\ ~\hfill cos(\theta )=\cfrac{\stackrel{adjacent}{-2√(6)}}{\stackrel{hypotenuse}{5}}~\hfill](https://img.qammunity.org/2020/formulas/mathematics/middle-school/elfxg4dr03sdyk8b6ob1p9bji71udgxxfu.png)