I'll use the integrating factor method for the first DE, and undetermined coefficients for the second one.
(a) Multiply both sides by exp(7t ):
exp(7t ) dx/dt + 7 exp(7t ) x = 5 exp(7t ) cos(2t )
The left side is now the derivative of a product:
d/dt [exp(7t ) x] = 5 exp(7t ) cos(2t )
Integrate both sides:
exp(7t ) x = 10/53 exp(7t ) sin(2t ) + 35/53 exp(7t ) cos(2t ) + C
Solve for x :
x = 10/53 sin(2t ) + 35/53 cos(2t ) + C exp(-7t )
(b) Solve the corresonding homogeneous DE:
d²x/dt ² + 6 dx/dt + 8x = 0
has characteristic equation
r ² + 6r + 8 = (r + 4) (r + 2) = 0
with roots at r = -4 and r = -2. So the characteristic solution is
x (char.) = C₁ exp(-4t ) + C₂ exp(-2t )
For the particular solution, assume an ansatz of the form
x (part.) = a cos(3t ) + b sin(3t )
with derivatives
dx/dt = -3a sin(3t ) + 3b cos(3t )
d²x/dt ² = -9a cos(3t ) - 9b sin(3t )
Substitute these into the non-homogeneous DE and solve for the coefficients:
(-9a cos(3t ) - 9b sin(3t ))
… + 6 (-3a sin(3t ) + 3b cos(3t ))
… + 8 (a cos(3t ) + b sin(3t ))
= (-a + 18b) cos(3t ) + (-18a - b) sin(3t ) = 5 sin(3t )
So we have
-a + 18b = 0
-18a - b = 5
==> a = -18/65 and b = -1/65
so that the particular solution is
x (part.) = -18/65 cos(3t ) - 1/65 sin(3t )
and thus the general solution is
x (gen.) = x (char.) + x (part.)
x = C₁ exp(-4t ) + C₂ exp(-2t ) - 18/65 cos(3t ) - 1/65 sin(3t )