I play horribly against these people.

Learning to play a pretty standard game with calculated deviations from standard play is key to becoming a good poker player. Frequently overadjusting to situations like the one you described is a pretty common problem.