Answer:
It matters but not like you think.
GPA, extracurricular activities, and SAT/ACT are the biggies. Truthfully it won't change a thing if you took accelerated pre-algebra versus normal. In fact, spoiler alert, the SAT/ACT (I'm assuming your american) will matter more than your courses ever did. I took two years of college physics in highschool and i made more off my SAT. Not only that, but an A in an easy class looks better than a C in a hard one on transcripts.
My advise is slow down, relax, and learn it well. Better to learn the concept fully in normal so that you can apply it, than half learn it in advance and struggle in future applications. After all, math ALWAYS builds. Take it slow, take it easy.
I would recommend talking to a school guidance counselor about your academic stress. Im sure their professional insight will help put you at ease. It may also help to get you going in the right direction so that you achieve your goals.
Good luck. Enjoy life