The function, as presented here, is ambiguous in terms of what's being deivded by what. For the sake of example, I will assume that you meant
3x+5a
f(x)= ------------
x^2-a^2
You are saying that the derivative of this function is 0 when x=12. Let's differentiate f(x) with respect to x and then let x = 12:
(x^2-a^2)(3) -(3x+5a)(2x)
f '(x) = ------------------------------------- = 0 when x = 12
[x^2-a^2]^2
(144-a^2)(3) - (36+5a)(24)
------------------------------------ = 0
[ ]^2
Simplifying,
(144-a^2) - 8(36+5a) = 0
144 - a^2 - 288 - 40a = 0
This can be rewritten as a quadratic in standard form:
-a^2 - 40a - 144 = 0, or a^2 + 40a + 144 = 0.
Solve for a by completing the square:
a^2 + 40a + 20^2 - 20^2 + 144 = 0
(a+20)^2 = 400 - 144 = 156
Then a+20 = sqrt[6(26)] = sqrt[6(2)(13)] = 4(3)(13)= 2sqrt(39)
Finally, a = -20 plus or minus 2sqrt(39)
You must check both answers by subst. into the original equation. Only if the result(s) is(are) true is your solution (value of a) correct.