8 1/2 can be written as 17/2, an improper fraction.
5 2/3 can be written as 17/3, an improper fraction.
Make a common denominator, 3 and 2 are the current ones, so what is a number they both can make by multiplying (the smallest one...), its six, they both equal six when multiplied by 2 and 3 respectively
So 3 * 2 = 6
And 2 * 3 = 6
17/2, multiply the top and bottom by 3 so that the bottom is 6
17/3, multiply the top and bottom by 2 so that the bottom is 6
And you get.....
51/6 which is equal to 17/2
34/6 which is equal to 17/3
Since they both have the same bottom number, or a common denominator, you can subtract easily, ignore the denominator first and just subtract 34 from 51 and you get 17. Now put back the denominator and you get 17/6.
So you have 17/6 cups of flour left.
If you want to write this as a mixed fraction, divide 17 by 6, you get 2 with remainder 5, so the mixed fraction equivalent is 2 5/6.