Help me figure out if I’m wrong in this argument please?

My husband is out of town building our home while I am home with our 3 toddlers (2 yr and 14 mo. twins). He is only 1 1/2 hrs away but has been staying at his sisters nearby. We are at the very last of our $ and sold our son’s snomobile given to him for Xmas for 1/2 what we paid for $. He also did not take his visitation 2nite because he said he is so busy. I just spoke with him and he asked me to deposit snowmobile $ into account so he can ATM and have money to RACE his ’86 truck at a drag strip 45 minutes away. He said “I need to find something fun to do.” I understand he’s working his butt off, but I havent’ left the house in 8 days to save on gas. And 1) we don’t have money for this 2) what if it breaks and needs repair 3) coming home to spend time with his family isn’t fun? and 4) why didn’t he ask if I wanted to come? I expressed all this and he told me he was sick of me not understanding he needed alone time, told me it was over and hung up on me. I’m at a loss here.
Sorry, he has 2 kids from a previous marriage that he has visitation every other weekend with. He asked me to call the ex wife and tell her he was too busy to see his kids this weekend. That was another thing that I was upset about.
Hey everyone, I so much appreciate your input (especially from some you guys out there ), but I don’t read questions that are very drawn out so I tried to keep it short. Let me explain, my husband took 2 months off of work (with pay) to finish the build of the house. I am a stay at home mom. We are married.
And by over, he meant he wanted the marriage over. He says that EVERY time we get into an argument which, I admit, has made me afraid to speak up sometimes. But after reading all these answers, I feel stronger about sticking up for me and the kids.