He took me and my daughter in, now he's changed

I am 28, and for five months, my six-year-old daughter and I have been living with a man who is 31. He is not her biological father and we come from different communities. We recently had a fight and in the process, he said that I am too old for him and that my child is too old. I realised that he may be having issues with the fact that I have a child from another man and he was just waiting for a chance to vent it out. I was so hurt by this and suggested that we separate, but he is now really begging me not to leave. He has since apologised, but I am confused, don’t know what to do. Please advise me.