Fold the set???

[The Ram]

2-4 NL @ Stars:

BB w/ 55. 2 limpers and SB calls up, I check.

Flop: 6-4-3 rainbow. SB bets $4 (minimum). I called. Wrong?? One limper calls.

Turn: 5. Now what do I do? I have a set on a 3-4-5-6 board. SB bets $4 again, I chose to raise to $16. Limper makes it $40, SB makes it $64.

Can I keep calling, knowing that limper might go in for another $115 or so? If the board pairs I am pretty much assured a win, but I'm only 10 in 46 for that, and even that assumes neither of them touched the board aside from the straight draw.

I chose to call, and sure enough, the limper moved in and the SB called. Can I call again? I assume that at this point I am obligated, but I don't know. (FYI, the SB had me covered for another $100 or so if the board paired on the river).

Did I totally misplay this hand?