X+Y=10
XY=-300
Solve for one of the variables
X=10-Y
Plug into the other equation
Y(10-Y)=-300
Now, you have a quadratic...
-y²-10y+300=0
I would multiply everything by -1, so there's no coefficient on y²
y²+10y-300=0
And solve...
-10±√(100-4(-300))
----------------------------
2
Y=-5±5√(13)
Plug back onto original equation...
X+(-5±5√(13))=10
X±5√(13)=15
X=15±5√(13)
Note: There are 2 sets of answers, keep the negatives with the negatives, and the positives with the positives when dealing with the ±.