[Darryl_P]

It's most certainly possible and you'll likely get there faster because you won't have to attend seminars and lectures that are irrelevant to what you want to learn. The hard part is motivation, though. The temptation is too great to just put it off without strict deadlines and fixed goals like exams. A good idea would be to bet an ambitious undergrad physics student that you can get a higher score than him on the physics GRE, say. That should give a good motivational push.