Organic chemistry was definitely harder for me, though this is a matter of opinion and learning style. If you are good with logic, then organic chemistry might not be so hard, or at least that's what my professor said. A lot of physics was just knowing how to use formulas and then plugging and chugging, and knowing basic outcomes of physical reactions. It's a whole other story if you take calculus based physics.
Good luck.