I have been pondering this myself, and this is the answer that I feel I have received from the Lord.
Being worldly is the equivalent of having an adulterous heart with respect to our love for God.
We are commanded to love the Lord with all our heart, soul, mind and strength. If we love anything else that is not for the purpose of loving God or glorifying him, this is adultery.
It can be compared to a husband and wife relationship (as the Lord does throughout the scriptures). If either spouse were to even be attracted to another person, this would obviously adultery, as Christ clearly stated. Even more so, since Christ died to call us out of this world, it is adultery when we are attracted to the things of this world. This is why 1 John states that he who loves the world does not have the love of the Father. The two are incompatible.
What this looks like for the individual believer is different, because we all have different lusts that tie us to this world. As you mentioned, a friend felt that tending her garden was worldly for her. That would not be the case for me, because I could care less about my garden. However, I have different lusts in my heart that pull me into this world. For me, following those lusts would be worldly living.
I was explaining this to my kids the other day as we drove by the local fair and they were commenting on the rides. I commented to my children that when they feel that their soul is pulling them to something that is of the world (entertainment, food, clothes, the home, etc.), that is adultery toward God and they should pull away from that. It is not to say that we cannot enjoy anything. It means that anything that is enjoyed NOT for the purpose of loving and glorifying God is worldly.
It comes down to being led by the Spirit (and being merciful towards others without judgment because you do not know where the Spirit is leading them). It is not following a bunch of dos and don'ts. It is spending time with God and in His Word so that He can speak directly to you and guide and cleanse you with His Spirit.