Glory,
I would be hestitant to wade into this situation from the outside. Relational puzzles are like a rubics cube where all the sides of the story are mixed together. Which color do we pick to fix first, yellow or blue? Likewise, which problem do we pick to 'fix' first, neglect or rebellion? Often our own dissapointments and expectations shape the "solution" we want for others. Often our own hurts, experiences, values and expectations become the "solution" we want for others. In family dynamics this can be volatile...as you know.
Having said that...has the son communicated that he is missing his dad? If this man is like me, such an honest comment would sting him to his core, and cause him to consider his priorities...taking whatever action is needed to make things better. Quitting the ministry is not out of the question, even for one who is called into ministry. In fact, if he is failing to minister first at home, he is neglecting the very call he aims to be committed to.
One other point...if it is clear that this pastor needs to spend more time with his son, then it will be clear to others in the church. If they continue to expect more time from the pastor when they know his family is missing him, then they are just as guilty as he. For instance, why doesn't his fishing buddy see this situation? The problem with many American churches is that they are affluent beehives for non-essential pet projects and activities.
Churches as peaceful communion of saints builds up people...churches as ambitious organizations devour people!
Blessings,
MC