I never thought non-monogamous marriage could work. I'm not worried about morals, It's just that human nature is what it is.
I don't know (I'm a one woman man and expect the same), but being in a monogamous relationship is a little bit "boring." They have peaks and valleys, and at some point - probably during one of those valleys - somebody is going to fall for someone else.
My personal life goal is to rude off into the sunset with the good woman I have. Maybe I'm just too boring and passionless. Sex is the best thing ever, but without a solid pair bond, I'm not sure how it would work.
On the other hand, everyone is different and your mileage may vary. But it does seem to come back to fighting human nature.