Oh man, that's the whole game right there! Skipping the sanding step is how you end up with those ugly ridges and bumps. I use 120 grit between coats, it takes off the high spots without digging in too deep. A quick once-over is all it needs, you don't have to sand it all the way back down. That final light sanding before paint makes it disappear completely. It seems like an extra hassle but it saves so much time fixing a bad finish later.
Sanding between coats makes such a huge difference, doesn't it? What grit sandpaper did you end up using for that step, like a 120 or something finer? I always second guess which one works best for a smooth finish.