Why is the SSH server asking for a password even if I specify a valid key?

Question

I can't set up password-less login on a Ubuntu server. Even if I specify a key, I get prompted for a password.

The same key can correctly log me in another server. So I don't think it's a permission issue on the client.

Server output:

root@server:~# /usr/sbin/sshd -d -p 2222
debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: private host key: #3 type 4 ED25519
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='2222'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from CLIENTIP port 48264 on SERVERIP port 2222
debug1: Client protocol version 2.0; client software version OpenSSH_9.3p1 Ubuntu-1ubuntu3.2
debug1: match: OpenSSH_9.3p1 Ubuntu-1ubuntu3.2 pat OpenSSH* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13
debug1: permanently_set_uid: 104/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> none [preauth]
debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user ubuntu service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "ubuntu"
debug1: PAM: setting PAM_RHOST to "CLIENT HOSTNAME"
debug1: PAM: setting PAM_TTY to "ssh"

Client output:

simone@client:~/.ssh$ ssh -vvv mmo
OpenSSH_9.3p1 Ubuntu-1ubuntu3.2, OpenSSL 3.0.10 1 Aug 2023
debug1: Reading configuration data /home/simone/.ssh/config
debug1: /home/simone/.ssh/config line 1: Applying options for mmo
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname SERVERIP is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/simone/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/simone/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to SERVERIP port 2222.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/simone/.ssh/id_rsa type 0
debug1: identity file /home/simone/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.3p1 Ubuntu-1ubuntu3.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13
debug1: compat_banner: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13 pat OpenSSH_6.6.1* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to SERVERIP:2222 as 'ubuntu'
debug3: put_host_port: SERVERIP:2222
debug1: load_hostkeys: fopen /home/simone/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: no algorithms matched; accept original
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:NeYvqJmThTo4GAxqqkz3qMbaN4KIJbUUPiGapVhoHgI
debug3: put_host_port: SERVERIP:2222
debug3: put_host_port: SERVERIP:2222
debug1: load_hostkeys: fopen /home/simone/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: checking without port identifier
debug3: record_hostkey: found key type ED25519 in file /home/simone/.ssh/known_hosts:1
debug3: load_hostkeys_file: loaded 1 keys from SERVERIP
debug1: load_hostkeys: fopen /home/simone/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'SERVERIP' is known and matches the ED25519 host key.
debug1: Found key in /home/simone/.ssh/known_hosts:1
debug1: found matching key w/out port
debug1: check_host_key: hostkey not known or explicitly trusted: disabling UpdateHostkeys
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug3: ssh_get_authentication_socket_path: path '/run/user/1000/keyring/ssh'
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 2 keys
debug1: Will attempt key: /home/simone/.ssh/id_rsa RSA SHA256:lHWoOcJ9LHEs3BLMMf/gBpeD9jyy6+OiC5PatiocLQc explicit agent
debug1: Will attempt key: simone@client RSA SHA256:Wg678UTK8aU1CFEw028Xomk9oPArJQ9nZItw2dIQoDA agent
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/simone/.ssh/id_rsa RSA SHA256:lHWoOcJ9LHEs3BLMMf/gBpeD9jyy6+OiC5PatiocLQc explicit agent
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: simone@client RSA SHA256:Wg678UTK8aU1CFEw028Xomk9oPArJQ9nZItw2dIQoDA agent
debug1: send_pubkey_test: no mutual signature algorithm
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
ubuntu@SERVERIP's password: 

My ~/.ssh/config file:

Host mmo
    HostName SERVERIP
    User ubuntu
    Port 2222
    IdentityFile ~/.ssh/id_rsa

I can correctly login with username and password. I've tried different keys, including one key that was for sure working a few months ago.

These are the permissions on the server:

root@vultr:/home/ubuntu/.ssh# ls -la
total 72
drwx------  3 ubuntu ubuntu  4096 Feb 18 13:26 .
drwxr-xr-x 26 ubuntu ubuntu 36864 Feb 18 13:23 ..
-rw-r--r--  1 ubuntu ubuntu  2563 Feb 18 13:26 authorized_keys
-rw-------  1 ubuntu ubuntu   141 Dec  4  2015 config
-rw-------  1 ubuntu ubuntu  1675 Jul 30  2022 id_rsa
-rw-r--r--  1 ubuntu ubuntu   394 Jul 30  2022 id_rsa.pub
drwxrwxr-x  2 ubuntu ubuntu  4096 Jul 30  2022 key-backup
-rw-r--r--  1 ubuntu ubuntu  6856 Jul 30  2022 known_hosts

My sshd_config file has these lines:

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      /home/ubuntu/.ssh/authorized_keys

Any ideas?

Answer 1

Your new (9.3) client disables ssh-rsa for RSA key(s) — because that method is now considered unsafe; it enables only the newer and better rsa-sha2-* methods —BUT your ancient (OpenSSH 6.6) server ONLY supports ssh-rsa and has never even heard of rsa-sha2. This is what the log message no mutual signature algorithm means.

Specify -oPubkeyAcceptedAlgorithms=ssh-rsa -- or =+ssh-rsa to continue allowing other key types but you don't seem to have any others. Alternatively configure this (without the -o) in ~/.ssh/config or /etc/ssh/ssh_config. (Below 8.5 you needed the older name PubkeyAcceptedKeyTypes.)

The following are actually dupes (which I have bookmarked) but for some reason Googling (as I would expect you to do) only finds me the first:

• OpenSSH v8 client talking to OpenSSH v6.7p1 server: "no mutual signature algorithm" because of disabled ssh-rsa? But why and how?
• Ubuntu Connection closed by [preauth]
• Different SSH versions and behaviors, public key failing.

Your client can't accept the host's RSA key for the same reason, or its ssh-dss (DSA) key which is similarly now insecure, but the host also offers ECDSA and Ed25519 keys which are still secure, and the Ed25519 key is selected and used because ssh-ed25519 was listed earlier in the client proposal. There are quite a few Qs about people unable to connect with recent SSH clients to ancient or low-quality servers with only ssh-rsa and/or ssh-dss keys, which need a similar fix.