Breaking up with our photographer

My FH and I decided to terminate the contract with our photographer as we were extremely unhappy with the engagement photos. The session was scheduled for an hour but took 45 mins, 15 mins were me changing into my second outfit. We got 17 photos back and there's a few with my FH's eyes closed. I was extremely disappointed. In addition to this he had canceled our original eshoot date the morning of and we had to reschedule for a few weeks later. This isn't a huge deal but I had paid $70 for my nails and had scheduled time for hair and make up. The day of the rescheduled shoot, he didn't want to commute to the original agreed location and said "I'm not driving all the way there if it's going to rain" and didn't consider looking at my Pinterest board of photo ideas that he requested. I wasn't able to cancel the second time around since I had paid again for my nails and then another $150 for my hair and makeup so we went along with the shoot.


After receiving the pictures and being disappointed we decided to find a new photographer and part ways. We were also having a south Asian wedding and no longer felt comfortable with his ability to deliver for our wedding.
I prepared a thoughtful and kind message to let him know of our decision and confirmed that I know the $500 deposit is non-refundable. He was very unhappy and today invoiced me for $338 for the cost of the eshoot (it was complimentary in the package we purchased). I informed him I would not be paying this as I have already paid the $500 and will not pay anymore. He then tried to aruge that the deposit was to hold the date and that under this contract I am not allowed to cancel. This is incorrect as the contract had no termination clause. Additionally the terms of the contract state this specifically "the client agree to pay the photographer a deposit of $500 out of the total package cost at the time of signing".
He is now threatening us by saying we either pay the $338 or the remaining $2000 of the contract or he will "escalate" the matter.
My intention was to part ways in good faith and not leaving a negative review of the company. I'm at a loss and would love some support or thoughts.