In a small group, you have the opportunity for true fellowship, prayer is a paramount objective. Stir people up to good works. If you are the moderator, learn to direct the conversation, as long as you are spiritually minded and discerning, yet do not attempt to dictate the conversation like an authoritarian. Scripture is always to be the light and the guide in all fellowship, you are merely to encourage others to see what the Bible says for itself. Let it speak, and just attempt, in faith, to speak what it says. Be prepared to correct gently and with love and grace, but also be incredibly patient. If all you do is correct, well...you may not have a home fellowship after long (contrary to popular belief, heresy is something that is to be tolerated with patience, mercy, compassion and grace, wolves with agendas and men who make followers after themselves are not to be tolerated.) Learn to pick your battles, you will earn peoples respect through patience and long-suffering just like the Lord does. Jesus was extremely patient with the disciples...
Going through a book conversationally and expositionally can be a huge benefit, but not just to understand it in a head-case fashion. Rather the point of studying it together is to encourage one another to actually perform it. If you happen to have a laptop with a concordance, and study software on it for quick and easy language help, that can be a benefit as well to help analyze nuances of the word when there is confusion or disagreement. Disagreement is not immoral, it is a-moral.
If you confess your faults and sins to others, then others will do the same. In all things lead by example and not by fiat. Be prepared to be the one who gets the most convicted by the word and the most willing to be humble and bear others burdens.
As far as structure goes...well structure is and ought always to be an overflow of a people with cleansed hearts and pure minds. No amount of external structure corrects those issues. But on a practical level, if one person is always dominating the conversation, then a private encouragement to that individual to consider others above themselves may be in order. If someone does not speak often, do not pressure them to do so beyond what God is doing through them. Although I have rarely found this to be a problem.
If these people start praying more, and obeying more, then you know Jesus is working through you and in your study group. If you are the host, or know a host family, make sure that hospitality is their gifting, or they seek to be as hospitable as possible. Food is good. The early church always shared meals like a family because they were family. I think this is a great thing to continue in. Feed Jesus sheep both the word, and food. It is a great combination.
In all things, be humble, and teachable in your own spirit. Recognize that Jesus is the head of your meetings, not you. Love trusts in all things, trust that God is working in those who attend and believe that God is good, and wants them to flourish. Encourage that flourishing.
Something I keep praying for and would like to see in the home fellowship that I am a part of would be a vision around which that little group of people could become united in the purpose of the advancement of the kingdom. It would be awesome to see home fellowships be like relay stations where brothers and sisters come to get informed on the callings of what it means to be the body of Christ, and then they get "sent" to perform, together as a cell, something for which Jesus has prepared for them from the foundations of the earth to do. My pastor always says, and I think it is very wise, "You cannot push a rope, you can only pull it." Your home fellowship is going to be a reflection, not of how you push others into service, but how you pull them by your own Godly example. This is where the skids are greased in ministry. I still have a lot to learn and have high hopes for what God purposes through his kids in regular home fellowship. The things I have shared are just some things I have learned...
Also, in James it says that we are to "Confess our faults to one another." And in 1 John it says that we need to confess our sins. James also says that, if we are sick, or in sin that we are to come to the elders and ask for prayer and the anointing of oil. Home fellowship is a balance between teaching, and actually DOING what the word says. If you come across something in scripture in your study that is an admonishment to perform and the setting allows you to obey that scripture then do it. Don't just teach about doing it. 1 John also says that if, "We walk in the light as he is in the light we have fellowship with one another, and truly our fellowship is with him." You need to make sure you are walking in the light above all things, and encouraging others to do the same with the end goal in mind of fellowship with God. What an amazing truth.
Home fellowships should be a place where sheep are comfortable, knowing their privacy and best interests are protected at all costs. Knowing that they can trust one another to ask for prayer and find a small body of people who are not just "brothers" in a technical sense, but brothers in deed.
I hope some of this helps. May God richly bless your meetings.