How hard would it be to relinquish my son's father's parental rights?

I have been a single mother for upwards of 13 years, in that time my ex has paid little to no child support and at most he has seen my son about 5 times a year. He is an alcoholic don't keep a job and is very verbally abusive towards my son. It has gotten so much worse since I recently got engaged. Last month it really started to hit the fan, I let my son go with his Grandma to their family reunion with the specific instructions that he was not to go anywhere especially with his father, that if his father wanted to take him anywhere he would have to talk to me first and that she was not to let my son go until she heard it was ok from me, well she didn't listen to my direct instructions and let him go to a place where his father goes to party all the time, during which time the ex gave my son a drink of liquor (really not good) then he tells my 14 year old that I said it was ok to go home with him for a week (never talked to me) so (son) called me to make sure at which time I told him no that he needed to be dropped off to me. Ex tells son he is old enough to make his own decisions that he don't have to listen to me and g-ma agreed with father...I let son go for 3 days, the next weekend comes and father wants to take son to a picnic at an amusement park..I say he has to be home by 6pm because we had to go up north..father's girlfriend says no prob..I get a call at 6:02pm that the are still at park not going to make it..that they will bring son up north when done..I call back at 7 no answer on cell..8:30 son calls still at park not ready to leave yet..i say ok I want you up there in about 2 hours Its 1陆 hour drive..get another call at 10:10 just leaving park and not coming up north..long story short, big argument ending with son being delivered..around 11:30pm..son gets to me and tells me that father was calling me bad names and that father wants to be physical with my fiance (not good) this last week school started and I called ahead to let them know that son would not be in school, that we would be out of town..no prob..father finds out and when he sees son with friends he (chews his a**) as father puts it for not being in school, instead of calling and talking to me, son calls me in shambles, my mother confronts father because I was gone and father tells my mom that son don't need meds he takes for ADHD and that he will take son to his doctor, also that son should have been in school that father would have driven him if necessary! father don't have drivers license, due to drinking and driving. Father gets into fight with girlfriend and tells son to grab clothes and takes him up north after having a couple. father also told girlfriend when mad that he would "knock her teeth down her throat" in front of son...very not happy!!! What do I do, Do I have a legal right to deny visitation? Should I call P.S. on father? not sure where to go with this! I don't trash father in front of son its not his fault, father is always putting son in bad place between he and I...

Well this sounds fun ok, there are several things you can do first off, I would suggest getting a restraining order on the father, and when you go down to file for one it will ask you why you need one I would copy and paste what you just told me.
Now considering the father is not acting like an adult AT ALL the judge will see this and most likely grant it. Since you can file one for your son cause he is a minor its okay cause he still under your roof.
Now second I would go to the court and if he supposed to be giving you payments tell them they have there own little police that will either pick him up or tell the local cops ie. give a warrant and have him arrested.
Now as for him giving up his parental right I know that its just the matter of signing a piece of paper that basically in a HUGE ammouamountlegal jargon I GIVE UP RIGHT TO SEE KID. But instead of just saying that its like 5 pages of that in all terms and languages you've never heard.
I know that you can talk to the court and maybe social services might help you or whatever you have where you live that protects kids, if you can get evidence that clearly shows he is a unfit parent then what happens is his parental rights are voided and he's yours either to grow up and hate you or love you.

Get hard proof. It is extremely difficult to terminate parental rights. Even when the parent does not have a job and drinks a lot. He can even fight with his girl friends all day long and only feed your son Macaroni & Cheese all day long. It is extremely difficult to fully terminate parental rights.