This idea comes from King James Version of the Bible, which says:
But thou, when thou prayest, enter into thy closet, and when thou hast shut thy door, pray to thy Father which is in secret; and thy Father which seeth in secret shall reward thee openly.
Matthew 6:6 [KJV]
However, Jesus is not talking about closets. The verse is better translated as follows:
But when you pray, go into your room and shut the door and pray to your Father who is in secret. And your Father who sees in secret will reward you.
Matthew 6:6 [ESV]
In any case, the physical location is not important. Jesus is making the point that we shouldn’t pray just to be seen to be praying — our prayers should be geniune. It doesn’t matter where we pray because God can hear us.