This is a big question and the answer is not as straightforward as you might think. On the one hand, the New Testament encourages believers to live in unity with one another (e.g. Eph 4:3). On the other hand, the New Testament warns against those who spread false teaching (e.g. 2 John 1:10-11). Imagine there was a church that claimed to be following the gospel but that taught that adultery was okay, or that Jesus was a liar. I don’t think any true Christian would want to be associated with such a church. These, of course, are extreme examples but they illustrate the point. The question for Christians is not whether a line should be drawn between believers and non-believers, but where that line should be drawn.
The problem faced by Christians throughout history is where to draw that line. If you believe that a different church is advocating wicked behaviour or repugnant doctrine then of course you will wish to be separate. Separate for the sake of your own church, to keep them from error, and separate for the sake of others to make it clear that this error is unacceptable. The problem is that Christians throughout history have often been too quick to condemn differences rather than tolerating a spectrum of views on issues that carry little weight.
However not all the separate churches were born out of doctrinal disagreement. Sometimes people will set up a new church for more mundane reasons, such as to have a different style of service.
As Christians we should seek for unity and to avoid needless separation. However if we take seriously the gospel message then there will always be some divisions between those who follow the gospel and those who do not.