1. The given equation is probably supposed to read
y'' - 2y' - 3y = 64x exp(-x)
First consider the homogeneous equation,
y'' - 2y' - 3y = 0
which has characteristic equation
r² - 2r - 3 = (r - 3) (r + 1) = 0
with roots r = 3 and r = -1. Then the characteristic solution is

and we let y₁ = exp(3x) and y₂ = exp(-x), our fundamental solutions.
Now we use variation of parameters, which gives a particular solution of the form

where


and W(y₁, y₂) is the Wronskian determinant of the two fundamental solutions. This is

Then we find


so it follows that the particular solution is

and so the general solution is

2. I'll again assume there's typo in the equation, and that it should read
y''' - 6y'' + 11y' - 6y = 2x exp(-x)
Again, we consider the homogeneous equation,
y''' - 6y'' + 11y' - 6y = 0
and observe that the characteristic polynomial,
r³ - 6r² + 11r - 6
has coefficients that sum to 1 - 6 + 11 - 6 = 0, which immediately tells us that r = 1 is a root. Polynomial division and subsequent factoring yields
r³ - 6r² + 11r - 6 = (r - 1) (r² - 5r + 6) = (r - 1) (r - 2) (r - 3)
and from this we see the characteristic solution is

For the particular solution, I'll use undetermined coefficients. We look for a solution of the form

whose first three derivatives are



Substituting these into the equation gives



It follows that -24a = 2 and 26a - 24b = 0, so that a = -1/12 = -12/144 and b = -13/144, so the particular solution is

and the general solution is
