Rear door panel removal
 

I need to replace the door lock mechanism. Am having some trouble removing the door panel. Its OK along the bottom pulling the plugs out but I'm not sure about the Arm rest and door handle. I started to pull up on the arm rest but it won't lossen at the front or where it butts up to the main panel. Don't want to force it as maybe there is a special way it needs to be done. Also does the window control and door handle need to come off as part of the panel removeal. WOuld appreciate any help or pointers on the chore. Love the car, have had it since new in 2000 and don't want to start messing it up through ignorance now. Its a 2000 Millenia S. Forgot to mention it is the rear drivers side door.
Tim