I will be honest unless you can prove to the court that the father is abusive or neglectfull you will have a hard time getting custody back, once the court grants custody to a parent then inorder to file to modifly that order there has to be a reason to show the court why you feel it is in the childs best interest to be with you, you can file that is your legal right, but the odds are against you if you cannot prove the father is abusive, or neglectfull. You would also need a lawyer because if your ex contest the custody it becomes a court battle. As for visitations the father has to allow you the stipulated court ordered visits, if he does not then you can file a petition to enforce that order, this can be done without a lawyer by going to the county court house where the order was granted. The father if he told you that you can see the child more than the stipulated order states then that is considered a verbal agreement and the courts do not aknowledge verbal agreements, so legally speaking he does not have to give you any additional visits. You can file a petition to modifly the orignal order and ask for more visitation.

Well sounds like you are just going back and forth , and stuck in the same situation,, The way i see it , you have to relax a little bit , and dont show or use any hostility towards the Father of your child and do be as calm as a cucumber when handling a civil matter. The visits have to be kept in a regular pattern , take notes of all your visits , and any requests make sure its in writing and sent by certified mail with a return receipt ., Reiterate that the visits are ordered by the court because it is in the childs best interest to spend some time with you., if he does not honour your visits or ignores the schedule , Then you hire a lawyer to enforce the order , Follow whatever is required , and get involved in board meetings and other community challenges ,, this will get you references , be prompt at work and on time ,, and take a lot of happy smiling pictures with your daughter , and with the grand-parents involved in different activities ,, Then eventually with his cooperation you can both agree on a schedule and you can now seek Joint physical custody , where the child is with you for a week or two and then with him a week or two ,,, you have to gradually move to sole custody , but i am sure you want whats in the childs best interest .