Recall the following identities:
sin²(x) = (1 - cos(2x))/2
cos²(x) = (1 + cos(2x))/2
Then
sin²(π/8) = (1 - cos(π/4))/2 = (1 - 1/√2)/2
cos⁴(3π/8) = (cos²(3π/8))² = ((1 + cos(3π/4))/2)² = ((1 - 1/√2)/2)²
and so
sin²(π/8) - cos⁴(3π/8) = (1 - 1/√2)/2 - ((1 - 1/√2)/2)²
… = (1 - 1/√2)/2 • (1 - (1 - 1/√2)/2)
… = (2 - √2)/4 • (1 - (2 - √2)/4)
… = (2 - √2)/16 • (4 - 2 + √2)
… = (2 - √2)(2 + √2)/16
… = (4 - 2)/16
… = 1/8