Switching from undecided to CS - good idea?

Currently undecided but I've been really enjoying my intro programming course. I'm worried about whether I can handle the math requirements (I'm okay at math but not amazing). Is it too late to switch as a sophomore? What should I expect from upper-division CS courses?