How can we know that the believer has been saved from guilt and the penalty of sin?

In many places the Bible states explicitly that the believer has been saved. Paul expresses it to the believers in Ephesus:

But God, being rich in mercy, because of the great love with which he loved us, even when we were dead in our trespasses, made us alive together with Christ—by grace you have been saved— and raised us up with him and seated us with him in the heavenly places in Christ Jesus, so that in the coming ages he might show the immeasurable riches of his grace in kindness toward us in Christ Jesus. For by grace you have been saved through faith. And this is not your own doing; it is the gift of God.
Eph 2:4-8

This passage raises some powerful ideas:

• Above all things we have God’s love. God’s love is the reason for everything, his whole purpose. That has to be remembered at all times, even when we feel the burden of guilt in our hearts — because God’s love is greater than our hearts.
• God loved us even before we became aware of him.
• If we come to him through faith in Jesus we have been saved. The fact that our salvation is stated in the past tense is astounding. It means that the only condition attached is that we be open to accept this gift of God’s grace and don’t reject it.

For God so loved the world, that he gave his only Son, that whoever believes in him should not perish but have eternal life
John 3:16

Now the penalty of sin is death and whilst it’s obvious that believers still experience death, it is not a permanent death.  In fact the Bible describes believers who have died as ‘those who have fallen asleep’ (1 Thess 4:15).