Table talk / collusion -- let it slide?

I was playing 3/6 at the Trump Indiana last Friday (wonderful loose game, by the way) when the following occurred (for full disclosure, this was my first foray into casino poker):

An older man was sitting directly to my left; his wife was just to his left (on the other side of the dealer). Both bought in for about $50, which seemed quite short.

We were playing a kill pot, and I had raised AJs from middle position and gotten 4 callers (including the wife but not the husband). The flop was Ace-high and rainbow, and I raised an early-position bet from a LAG and still got 3 callers. After my turn bet, I saw the wife lean behind the dealer and mouth to her husband: "Does he have it?" He nodded, and she folded.

This didn't bother me too much until I thought about it later (although I was a lot more careful about how I peeked at my cards just in case). I didn't bother saying anything at the time because both players seemed quite weak; however it later became apparent that the wife almost always called down with any piece of the board or any pocket pair, especially in big pots, so this may or may not have cost me 2 BB (calls for my turn bet and river bet).

Should I have done anything here? Should I have been more willing to let it go to keep them happy and at the table because they seemed beatable? At the very least they were discussing the hand; at the worst he'd seen my holecards. Because I'm a B&M newbie I don't know how to handle this.

Any thoughts?