If you would like to simplify (3xy^2) * (4xy) * (2xy)^3, you can do this using the following steps:
(3xy^2) * (4xy) * (2xy)^3 = 3xy^2 * 4xy * 2^3x^3y^3 = 3 * 4 * 8 * x * x * x^3 * y^2 * y * y^3 = 96 * x^5 * y^6
The correct result would be 96 * x^5 * y^6.