I shall rewrite the initial formula to make it more clear what the proper notation is.
6x³y + 6xy - 12x² - 12
To begin this problem, we can first find any common denominator in each term, and it turns out that each term can be divided by 6. So we will factor out 6.
6(x³y + xy - 2x² - 2)
We will group the first two terms and the second two terms. From here we can factor out the greatest common denominator from each group.
6 ( (x³y + xy) + (-2x² - 2) )
6 ( xy (x² + 1) - 2 (x² + 1) )
The greatest common denominator is x² + 1. Factoring out this leaves us with the following:
6 (x² + 1) (xy - 2)
Therefore, 6, (x² + 1), and (xy - 2) are all factors of the initial formula.