Husband taking advantage of me?

I am in desperate need of advice. My husband sometimes has sex with me after I take Ambien BUT I am still awake (according to him but I do believe him) and coming onto him very strong. I told him multiple times before not to do this but he would say he couldn't tell that I had taken it. I've taken Ambien for about 20 years for a terrible insomnia problem. I wake up having no memory of these incidents and only know I've had sex because my clothes are off. When asked my husband will always admit to having sex so I know 100% that it's happened. The last time it happened was just 2 days ago and he said that he asked me if I had taken the Ambien and I mumbled something (he was vague on what I said). When I ask why he didn't stop then he said I was just so turned on and begging him for sex that he went ahead with it anyway. He claims that when I wake him up for sex he is in such a sleepy state that he's not totally aware and can't think straight. I know the Ambien makes me horny and have told him to absolutely not have sex with me if I initiate in the middle of the night. He knows I take Ambien every night. I don't think he is being malicious but I do think he is taking advantage of the situation. Is it possible that this is an impulse control problem that he can get help for? I am at a loss for what to do. Stopping the Ambien would solve the problem but I can't stop it cold turkey and nothing else has worked for me. Plus even if I was able to stop, should I leave him anyway over this? I have 2 kids at home. Our relationship has been rocky over the past few years due to other issues (not cheating or anything). I decided a few weeks ago to pursue divorce and he convinced me not to. But now he does this again?! I am so confused- I appreciate any help or advice..

EDIT to original post:

Wow, I really appreciate all of the comments and advice. This situation has been tormenting me so I am considering everything right now. A couple of details that I neglected to add in my original post:

My husband also has a history of badgering me to perform certain ways sexually. I've said no countless times but he will always come up with another reason why I should do these things (it's his birthday, he's getting a vasectomy, etc). When I asked once why he would want me to do something that I've been coerced into doing because of guilt, etc he says that it turns him on more to see me uncomfortable. These acts are on the demeaning/humiliating side IMO and he says it make shim feel "powerful and masculine" to do them. Once or twice I had flashbacks of doing these things the next day and when confronted he admitted they happened. Who knows how many times this has happened while under the influence of Ambien because I usually have no memory at all. So this makes me think he sees me under the influence of the medication and coming on to him and has trouble pushing me away because he knows it will be a better experience for him. There was someone who said her husband would just put her back in bed or leave the room when she did something similar. I wish my husband would do that because that would mean he was looking out for me. He has apologized for these things but usually only when pressed. Even when I told him I wanted a divorce during our counseling session he wouldn't only give these vague half-apologies despite me being clearly upset and in anguish. He has promised a lot of things to me to try to get me to stay and has made some significant positive changes but falls short about apologizing for this. It's embarrassing and sad to be in this situation.

I take Ambien every night because it's the only things that has worked for my severe insomnia. I appreciate everyone's concern and sharing their stories. I want to stop taking it but have to have a plan and the time to do so where i won't get any sleep for a while. For those of you who suggested marijuana, I'm assuming that doesn't have similar effects?