You will still get password prompt to decrypt private key even if it is loaded into ssh-agent until the corresponding SSH public key is added into remote ~/.ssh/authorized_keys.
To reproduce:
# We are about to ssh to localhost, therefore, unauthorized everyone.
rm ~/.ssh/authorized_keys
eval $(ssh-agent)
Agent pid 9290
ssh-add
Enter passphrase for /home/uvsmtid/.ssh/id_rsa:
Identity added: /home/uvsmtid/.ssh/id_rsa (/home/uvsmtid/.ssh/id_rsa)
1st attempt: password is prompted
ssh localhost
Enter passphrase for key '/home/uvsmtid/.ssh/id_rsa':
uvsmtid@localhost's password:
NOTE: See password prompt for private key
(and only then prompt for remote login).
Why? Isn't the private key is already loaded by ssh-add?
Copy public key to remote host:
ssh-copy-id localhost
2nd attempt: password is NOT prompted
ssh localhost
NOTE: No password for private key anymore.
The key is served by ssh-agent.
Confusing enough.
Remote SSH login password would be enough in this case.
I can speculate that this prevents adding your public key (which is paired with encrypted private key) without knowing encryption password for corresponding private key (without owner of the keys) for better security, but... it is still just a public key.
Anyway, it is one-time-per-remote-login procedure.