"Have any of you been in a similar place?"
Many times.
"I was wondering if any of you have any advice on what I should do."
My advise is pretty simple and may not amount to much, from what I have picked up from your post is this though.
"to be passionate about serving him, just like so many people I know."
Comparing ourselves to others is a dangerous game, I have done it many times and have found it leads to these feelings you are having, it leads to questions of who you are and what you are doing compared to others, questions like "Bill is such an awesome Christian, why aren't I as passionate for Christ as Bill?" When we ask these kind of questions and compare ourselves to others we tend to forget that people that are passionate for Christ may very well be further down the road than we are or have a different personality than we have.
I use to listen to Brother Leonard Ravenhill on this site and other Godly brothers and be amazed at the knowledge, love, and passion they had/have for Christ Jesus. Then look at myself and see I was nothing like them nor did I know as much, nor have a passion at they did.
I beat myself up many many times over this until I realized it took those brothers many years to develope that passion, knowledge, and Love for Christ. Remember the Mustard Seed of Faith is very small at first but if properly cultivated becomes one of the Largest trees of the garden. Don't be ashamed of your small love but rejoice that there is Love there and look to the future and your progress as Christ increases your Love for Him.
I have given this example before and it has helped me greatly being a dad, my children have taught me many things concerning God, and I hope it helps you as well. If you have a 1 year old and your job is to see it to maturity, you wouldn't expect it to mature over night. If you wanted it to read you wouldn't get frustrated and mad at the child because of its illiteracy but because you know its ability you simply love the Child where it is. I don't get angry with my children because they don't know how to speak fluent english BUT as a father I delight in hearing their broken english saying I lub you dada, Nothing makes my heart happier and Christ Jesus is the same way, He delights in our immature Love even if we get frustrated at our progress sometimes. God knows what He is doing.
In times of depression due to this feeling that I loved God less or little I started to thank God for the Love that He gave me to Love Him with and slowly but surely He has increased that Love.
I would make one more note its good sometimes to examine our hearts and see if there are things we have wasted our time on but its vital that this is a spirit lead process and that Christ Jesus is our strength. Remember just because you feel you have a Lack of Love doesn't mean Christ Jesus has a Lack of Love for you, He loves you where you are and where you are going. Sorry this post was kind of long winded but I hope and pray it helps you out.
God Bless,
Matthew