So now you just need to use LaGrange multipliers to minimize the function of a unit sphere (a compact set) and work out for which values of p the minimum is non-negative.
2a²b+9b²c+12c²a≥pabc
for a,b,c≥0.Letting a=3k,b=2m,c=n yields
36(k²m+m²n+n²k)≥(6p)kmn
k²m+m²n+n²k≥(p6/5)kmn
We now use AM-GM on k2m,m2n,n2k to get that
k²m+m²n+n²k/3 ≥ (k³m³n³)∧13
k2m+m2n+n2k≥3kmn
So we know that this applies to all p18. Furthermore, taking k=m=n turns the inequality into equality, therefore any p>18 fails. As a result, p=18 is the maximum.
Call that function f, and keep in mind that f(x,y,z)=(x,y,z) for all >0 and x,y,zR. So all you have to do is ensure that f is non-negative on the unit sphere.
To learn more about