The original context for the phrase is found in Galatians:
“James, Cephas and John, those esteemed as pillars, gave me and Barnabas the right hand of fellowship when they recognized the grace given to me. They agreed that we should go to the Gentiles, and they to the circumcised.” (Galatians 2:9)
This marks the first report by Paul to the elders in Jerusalem of his work, and the other apostles showed their approval by extending “the right hand of fellowship” to Paul and Barnabas.
Subsequently the act was adopted by many churches as a symbol of welcoming new members to share in the work of a church. It is often given immediately after baptism, or when moving cities.