This passage, so often misapplied by believers and unbelievers alike, must be understood within the context of where it came from - the so-called Sermon on the Mount. Jesus was speaking specifically to His closest disciples in this sermon, laying out for them how they were to act when He was bodily gone.
Thousands of others heard the sermon, too, but did not have the connection with Jesus that His closest disciples did.
The verse at issue seems to me the fullest expression of what is called by believers and unbelievers alike the Golden Rule, which is spoken by Jesus a few verses later in Matthew Chapter 7. But it is so often used by unbelievers to judge believers' actions as Christians, to call down the label of hypocrite - and used meanly by believers on each other as a hickory accountability stick.
Only when verse 1 is cited/read in conjuction with verses 2-14, does its deeper meaning appear: settle your own spiritual affairs with understanding and obedience to the Father before dealing with those of fellow believers.