f(x) = 1/(1 - x)
g(x) = (x - 1)/x
The inverse functions f ⁻¹(x) and g ⁻¹(x), if they exist, are such that
f( f ⁻¹(x) ) = x
g( g ⁻¹(x) ) = x
So we have
f( f ⁻¹(x) ) = 1/(1 - f ⁻¹(x) ) = x
Solve for f ⁻¹(x) :
1 = (1 - f ⁻¹(x) ) x
1 - f ⁻¹(x) = 1/x
f ⁻¹(x) = 1 - 1/x
f ⁻¹(x) = (x - 1)/x
and so f ⁻¹(x) = g(x).
Similarly, you can solve for g ⁻¹(x) :
g( g ⁻¹(x) ) = (g ⁻¹(x) - 1) / g ⁻¹(x) = x
1 - 1/g ⁻¹(x) = x
1/g ⁻¹(x) = 1 - x
g ⁻¹(x) = 1/(1 - x)
So we know f(x) and g(x) are inverses of one another,
f ⁻¹(x) = g(x)
g ⁻¹(x) = f(x)
Then
(a) (f o g)(x) = x
(b) (g o f ⁻¹)(x) = g(g(x)) = (g(x) - 1)/g(x) = 1 - 1/g(x) = 1 - x/(x - 1) = 1/(1 - x)
(c) (f o g ⁻¹)(x) = f(f(x)) = 1/(1 - f(x) ) = 1/(1 - 1/(1 - x)) = 1/(x/(x - 1)) = (x - 1)/x