I have three children at home, and as any parent can attest, there are no two kids alike. They all behave differently, eat differently, sleep differently, grow differently. They're just different.
As different as they are, some things are the same. For instance, if they want to grow up to be healthy and strong men, they need to eat good, nutritious meals. And, they can't just pick at them either. They have to eat it all in order to get all the vitamins and minerals they need. Without them, their growth is stunted.
Our spiritual lives are much the same.
A lesson I have learned is that when I don't read my Bible or pray daily, my spiritual growth sputters, and I actually shrink in my relationship with God. What's interesting is that just like my children's growth is dependent on them actually feeding themselves, my growth with Jesus depends solely on me. It has to be my choice to "self-feed." I have to be the one to open my Bible. I have to be the one to pray. I have to make the conscious decision to grow closer to God.
The best illustration I can come up with is that of the prodigal son. At the end of the story, the father (representing God) was always waiting for his son to come to him. The son (representing all of us) had to make the personal decision to return to the father.
So, what's keeping you from growing back to God?