There's something very off about this question.
In spherical coordinates,
x² + y² + z² = ρ²
so that
f(x, y, z) = 6 - (x² + y² + z²)
transforms to
g(ρ, θ, φ) = 6 - ρ²
When transforming to spherical coordinates, we also introduce the Jacobian determinant, so that
dV = dx dy dz = ρ² sin(φ) dρ dθ dφ
Since we integrate over a sphere with radius 4, the domain of integration is the set
E = {(ρ, θ, φ) : 0 ≤ ρ ≤ 4 and 0 ≤ θ ≤ 2π and 0 ≤ φ ≤ π}
so that the integral is
Computing the integral is simple enough.
but the mass can't be negative...
Chances are good that this question was recycled without carefully changing all the parameters. Going through the same steps as above, the mass of a spherical body with radius R and mass density given by
for some positive number k is
so in order for the mass to be positive, we must have
5k - 3r² ≥ 0 ⇒ k ≥ 3r²/5
In this case, k = 6 and r = 4, but 3•4²/5 = 9.6.