How do you know?

If the good times start to out weigh the bad, then its time to start a dialect about what's happening in the relationship. Relationships are hard work, but not so hard that you are starting to feel miserable and unhappy. The time when to give up is when you, him or the both of you come to the realization that this is to unhealthy to continue and its time to move on.