is it wrong to say people are not saved?
Yes.
Only Jesus knows who is truly saved and who is not. Even if you are right about what you say, it is not your place (or mine) to say who is saved. That is the work of Christ.
John 21:22 (NIV)
Jesus answered, "If I want him to remain alive until I return, what is that to you? You must follow me."
As far as backsliding goes, this is a different matter. We are not to judge a person as to whether or not they are saved. However, if a person is blatantly sinning, we should follow the counsel of Paul.
1 Corinthians 5:9-13 (NIV)
"9 I have written you in my letter not to associate with sexually immoral people-- 10 not at all meaning the people of this world who are immoral, or the greedy and swindlers, or idolaters. In that case you would have to leave this world. 11 But now I am writing you that you must not associate with anyone who calls himself a brother but is sexually immoral or greedy, an idolater or a slanderer, a drunkard or a swindler. With such a man do not even eat. 12 What business is it of mine to judge those outside the church? Are you not to judge those inside? 13 God will judge those outside. 'Expel the wicked man from among you.'"
Clearly, there is a difference here. To judge a person's salvation is to condemn them. Only Christ has the authority to judge a person's faith. However, if a brother or sister is sinning, we can judge their fruit. We can say, "My brother, you are sinning."
Christ gave proper procedures for addressing sin in the Church.
Matthew 18:15-17 (NIV)
15 "If your brother sins against you, go and show him his fault, just between the two of you. If he listens to you, you have won your brother over. 16 But if he will not listen, take one or two others along, so that 'every matter may be established by the testimony of two or three witnesses.' 17 If he refuses to listen to them, tell it to the church; and if he refuses to listen even to the church, treat him as you would a pagan or a tax collector.
Luke 17:3 (NIV)
So watch yourselves. "If your brother sins, rebuke him, and if he repents, forgive him.
Our goal in addressing sin is not to prove ourselves or to have power over another. It is to lovingly bring a person back to Christ. Only a person walking in humility is able to do such a thing. First, we need to check ourselves and see if we have a log in our own eye. Once we have been humbled, then we can help another to see.
The key word in the above passage is "repents." This is very important. Christ said we should continually forgive a person if they sin so long as they repent. If they do not repent, we should rebuke them. If a person is struggling with sin, then we should help them to find victory.
For example, let's say a brother you know is committing fornication on a regular basis and you confront him with this sin. The brother says to you, "I know it's wrong, I just can't stop." Obviously, the brother is struggling with sin. You should then encourage him to seek help. However, if you come to this brother and he says, "Who are you to judge me?" Obviously, this brother does not see his sin. He is not seeking to turn from his sin. In this case, you should gather together other brothers to confront him as Jesus said and show him his sin. If this brother still continues to deny his sin, then this should be followed by rebuke as Paul said. We should not even eat with this brother.
Of course, any situation like this that arises should be preluded with immense prayer.
There is a difference between judging the fruit of a person, and judging a person. We are not to judge a person's salvation. We just don't know. A person could be committing all sorts of sins and be saved. Christ will allow this to happen in order to discipline. When a person is left to their sins, they suffer the consequences of sin. As such, they may come back to Christ years later.
We cannot be the judge of who is saved or not. We can, however, clearly see the fruit of a person in the Church. If a person says, "I am a Christian" and then says, "it is okay to sin." This person must be rebuked. Do not even eat with such a person. It is not okay to sin. However, if a person says, "I am not a Christian" and says, "it is okay to sin" we must heed Pauls words. It is not our place to judge the fruit of a person outside the Church.
I hope this helps,
May Christ bring you clarity on this,
In love,
Blake