It would be a big mistake. For one thing, 4th street is a half bet, so you can be a little looser (unless someone paird their door, in which case you can feel free to mindlessly fold). In the specific example you gave, I'd bet 4th and maybe even 5th.

However, I think there's lots and lots of hands where you call 1 bet on 4th and fold 5th unimproved.