Recall that
cos²(x) + sin²(x) = 1
for any x. Also, remember that cotangent is defined as
cot(x) = cos(x) / sin(x)
We're given that cos(t ) = 5/13 and cot(t ) < 0, which means sin(t ) must be negative. So when we solve for sin(t ) in the first relation, we have to take the negative square root:
sin²(t ) = 1 - cos²(t )
sin(t ) = -√(1 - cos²(t ))
sin(t ) = -√(1 - (5/13)²) = -12/13
Then
cot(t ) = (5/13) / (-12/13) = -5/12