That expression is really a mess.
5x³ + 4x² + x³
First of all, notice that the first and third terms are amounts of the
same thing ... you have 5 of x-cubed, and then another 1 of x-cubed.
There's no reason they have to stay separate like that.
5 of anything plus another 1 of the same thing is always 6 of them.
This is called "combining like terms".
6x³ + 4x²
Now, you could factor 2 out of both terms: 2 (3x³ + 2x²)
Finally, you could factor x² out of both terms: 2x²(3x + 2) .
I'm not sure this is any simpler than the original expression was.
But I think this is what the homework exercise is looking for.