The area of a square is A=s² (s being the side). Since you know this, you can write an equation.
A=(6a - 2b)²
Expanding this would give you,
A=(6a - 2b)(6a - 2b)
Then to solve this you would want to FOIL (Multiply first terms, multiply outer terms, multiply inner terms, multiply last terms)
36a² - 12ab - 12ab + 4b²
Then combine like terms,
36a² -24ab + 4b²
So the answer would be D