Why is my SSH login slow?

Question

I'm seeing delays in SSH Logins. Specifically, there are 2 spots where I see a range from instantaneous to multi-second delays.

1. Between issuing the ssh command and getting a login prompt and
2. between entering the passphrase and having the shell load

Now, specifically I'm looking at ssh details only here. Obviously network latency, speed of the hardware and OSes involved, complex login scripts, etc can cause delays. For context I ssh to a vast multitude of linux distributions and some Solaris hosts using mostly Ubuntu, CentOS, and MacOS X as my client systems. Almost all of the time, the ssh server configuration is unchanged from the OS's default settings.

What ssh server configurations should I be interested in? Are there OS/kernel parameters that can be tuned? Login shell tricks? Etc?

Answer 1

Try setting UseDNS to no in /etc/sshd_config or /etc/ssh/sshd_config.

Answer 2

When I ran ssh -vvv on a server with a similar slow performance I saw a hang here:

debug1: Next authentication method: gssapi-with-mic

By editing /etc/ssh/ssh_config and commenting out that authentication method I got the login performance back to normal. Here's what I have in my /etc/ssh/ssh_config on the server:

GSSAPIAuthentication no

You can set this globally on the server, so it doesn't accept GSSAPI to authenticate. Just add GSSAPIAuthentication no to /etc/ssh/sshd_config on the server and restart the service.

Answer 3

For me, the culprit was IPv6 resolution, it was timing out. (Bad DNS setting at my host provider, I guess.) I discovered this by doing ssh -v, which showed which step was hanging.

The solution is to ssh with the -4 option:

ssh -4 me@myserver.com

Answer 4

With systemd, login may hangs on dbus communication with logind after some upgrades, then you need to restart logind

systemctl restart systemd-logind

Saw that on debian 8, arch linx, and on a suse list

Answer 5

You can always start ssh with the -v option which displays what is being done at the moment.

$ ssh -v you@host

With the information you gave I can only suggest some client side configurations:

• Since you write that you are entering passwords manually, I would suggest that you use public key authentification if possible. This removes you as a speed bottleneck.

• You could also disable X-forwarding with -x and authentication forwarding with -a (these might already be disabled by default). Especially disabling X-forwarding can give you a big speed improvement if your client needs to start an X-server for the ssh command (e.g. under OS X).

Everything else really depends on what kinds of delays you experience where and when.

Answer 6

Regarding the 2. point, here is an answer that don't require to modify the server nor require to have root/administrative privileges.

You need to edit your "user ssh_config" file which is:

vi $HOME/.ssh/config

(Note: you would have to create the directory $HOME/.ssh if it does not exist)

And add:

Host *
  GSSAPIAuthentication no
  GSSAPIDelegateCredentials yes

You can do so on a per host basis if required :) example:

Host linux-srv
  HostName 192.158.1.1
  GSSAPIAuthentication no
  GSSAPIDelegateCredentials yes

Make sure the IP address match your server IP. One cool advantage is that now ssh will provide autocomplete for this server. So you can type ssh lin + Tab and it should autocomplete to ssh linux-srv.

Answer 7

I found that PAM was reading the file /var/log/btmp, which had become huge as a result of people trying to brute-force my server. This was leading to login times of a minute. Clearing this file solved the problem.

Answer 8

I found that restarting systemd-logind.service only cured the problem for a few hours. Changing UsePAM from yes to no in sshd_config has resulted in fast logins, although motd is no longer displayed. Comments about security issues?

Answer 9

Check /etc/resolv.conf on the server to be sure that DNS server, listed in this file, works ok, and delete any non-working DNS.

Sometimes it is very helpful.

Answer 10

Note: This started as a "How to debug", tutorial, but ended up being the solution that helped me on an Ubuntu 16.04 LTS server.

TLDR: Run landscape-sysinfo and check if that command takes a long time to finish; it's the system information printout on a new SSH login. If that's the case, you may be able to disable the execution of /etc/update-motd.d/50-landscape-sysinfo, which does call that command, by chmod -x-ing it. Note that this command isn't available on all systems, the landscape-common package installs it.

---

Start a second ssh server on another port on the machine that has the problem, do so in debug mode, which won't make it fork and will print out debug messages:

sudo /usr/sbin/sshd -ddd -p 44321

connect to that server from another machine in verbose mode:

ssh -vvv -p 44321 username@server

My client outputs the following lines right before starting to sleep:

debug1: Entering interactive session.
debug1: pledge: network

Googling that isn't really helpful, but the server logs are better:

debug3: mm_send_keystate: Finished sending state [preauth]
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
debug3: PAM: opening session
---- Pauses here ----
debug3: PAM: sshpam_store_conv called with 1 messages
User child is on pid 28051

I noticed that when I change UsePAM yes to UsePAM no then this issue is resolved.

Not related to UseDNS or any other setting, only UsePAM affects this problem on my system.

I have no clue why, and I'm also not leaving UsePAM at no, because I do not know which the side-effects are, but this lets me continue investigating.

So please don't consider this to be an answer, but a first step to start finding out what's wrong.

---

So I continued investigating, and ran sshd with strace (sudo strace /usr/sbin/sshd -ddd -p 44321). This yielded the following:

sendto(4, "<87>Nov 20 20:35:21 sshd[2234]: "..., 110, MSG_NOSIGNAL, NULL, 0) = 110
close(5)                                = 0
stat("/etc/update-motd.d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
umask(022)                              = 02
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f15dce784b0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f15dce784b0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7ffde6152d2c) = 2385
wait4(2385, # BLOCKS RIGHT HERE, BEFORE THE REST IS PRINTED OUT # [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2385

The line /etc/update-motd.d made me suspicious, apparently the process waits for the result of the stuff that is in /etc/update-motd.d

So I cd'd into /etc/update-motd.d and ran a sudo chmod -x * in order to inhibit PAM to run all the files which generate this dynamic Message Of The Day, which includes system load and if packages need to be upgraded, and this solved the issue.

This is a server based on an "energy-efficient" N3150 CPU which has a lot of work to do 24/7, so I think that collecting all this motd-data was just too much for it.

I may start to enable scripts in that folder selectively, to see which are less harmful, but specially calling landscape-sysinfo is very slow, and 50-landscape-sysinfo does call that command. I think that is the one which causes the biggest delay.

After reenabling most of the files I came to the conclusion that 50-landscape-sysinfo and 99-esm were the cause for my troubles. 50-landscape-sysinfo took about 5 seconds to execute and 99-esm about 3 seconds. All the remaining files about 2 seconds altogether.

Neither 50-landscape-sysinfo and 99-esm are crucial. 50-landscape-sysinfo prints out interesting system stats (and also if you're low on space!), and 99-esm prints out messages related to Ubuntu Extended Security Maintenance

Finally you can create a script with echo '/usr/bin/landscape-sysinfo' > info.sh && chmod +x info.sh and get that printout upon request.

Answer 11

Besides the DNS issues already mentioned, if you're ssh'ing into a server with many NFS mounts, then there can be a delay between password and prompt as the quota command checks for your usage/quota on all filesystems not mounted with the noquota. On Solaris systems, you can see this in the default /etc/profile and skip it by running touch $HOME/.hushlogin .

Answer 12

This is probably only specific to the Debian/Ubuntu OpenSSH, which includes the user-group-modes.patch written by one of the Debian package maintainers. This patch allows the ~/.ssh files to have the group writable bit set (g+w) if there is only one user with the same gid as that of the file. The patch's secure_permissions() function does this check. One of the phases of the check is to go through each passwd entry using getpwent() and compare the gid of the entry with the gid of the file.

On a system with many entries and/or slow NIS/LDAP authentication, this check will be slow. nscd does not cache getpwent() calls, so every passwd entry will be read over the network if the server is not local. On the system I found this, it added about 4 seconds for each invocation of ssh or login into the system.

The fix is to remove the writable bit on all of the files in ~/.ssh by doing chmod g-w ~/.ssh/*.

Answer 13

I have recently found another cause of slow ssh logins.

Even if you have UseDNS no in /etc/sshd_config, sshd may still perform reverse DNS lookups if /etc/hosts.deny has an entry like:

nnn-nnn-nnn-nnn.rev.some.domain.com

That might happen if you have DenyHosts installed in your system.

It would be great if someone knew how to make DenyHosts avoid putting this kind of entry in /etc/hosts.deny.

Here is a link, to the DenyHosts FAQ, on how to remove entries from /etc/hosts.deny - see How can I remove an IP address that DenyHosts blocked?

Answer 14

To complete all the answers showing that DNS resolutions can slow your ssh login, sometimes, a firewall rules is missing. For example, if you DROP all the INPUT paquets by default

iptables -t filter -P INPUT DROP

then you'll have to accept INPUT for ssh port and DNS request

iptables -t filter -A INPUT -p tcp --dport 53 -j ACCEPT
iptables -t filter -A INPUT -p udp --dport 53 -j ACCEPT

Answer 15

ssh -vvv connection went really fine until it hung on the system trying to get the terminal for at least 20 Seconds:

debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
... waiting ... waiting ... waiting

After doing a systemctl restart systemd-logind on the server I had instant connection again!

This was on debian8! So systemd was the issue here!

Note: Bastien Durel already gave an answer for this issue, however it lacks the debug information. I hope this is helpful to someone.

Answer 16

We may find that the preferred name resolution method isn't the host file and then DNS.

For example, this would be the usual configuration:

[root@LINUX1 ~]# cat /etc/nsswitch.conf|grep hosts
#hosts:     db files nisplus nis dns
hosts:      files dns myhostname

First, hosts file is reached (option: files) and then DNS (option: dns), however we can find that another name resolution system has been added that is not operational and is causing us the slowness in trying to do the reverse resolution.

If the name resolution order isn't correct, you can change it at: /etc/nsswitch.conf

Extracted from: http://www.sysadmit.com/2017/07/linux-ssh-login-lento.html

Answer 17

I tried all the answers but none of them worked. finally I find out my problem:

first I run sudo tail -f /var/log/auth.log so I can see the log of ssh then in another session run ssh 172.16.111.166 and noticed waiting on

/usr/bin/sss_ssh_knownhostsproxy -p 22 172.16.111.166

after searching I located this line in /etc/ssd/ssh_config

ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

I commented it and the delay gone

Answer 18

This thread is already providing a bunch of solutions but mine is not given here =). So here it is. My problem (took about 1 minutes to ssh login into my raspberry pi), was due to a corrupted .bash_history file. Since the file is read at login, this was causing the login delay. Once I removed the file, login time went back to normal, like instantaneous.

Hope this will help some other people.

Answer 19

Work fine.

# uname -a
SunOS oi-san-01 5.11 oi_151a3 i86pc i386 i86pc Solaris
# ssh -V
Sun_SSH_1.5, SSH protocols 1.5/2.0, OpenSSL 0x009080ff
# echo "GSSAPIAuthentication no" >> /etc/ssh/sshd_config
# echo "LookupClientHostnames no" >> /etc/ssh/sshd_config
# svcadm restart ssh

UseDNS no do not work with OpenIndiana !!!

Read "man sshd_config" for all the options

"LookupClientHostnames no" if your server can not resolve

Answer 20

If none of the above answers works and you're facing dns reverse lookup problems you can also check if nscd (name service cache daemon) is installed and running.

If this is the problem it is because you have no dns cache, and each time you query for a hostname that is not on your hostfile you send the question to your name server instead of looking in your cache

I tried all the above options and the only change that worked was start nscd.

You should also verify the order to make dns query resolution in /etc/nsswitch.conf to use hosts file first.

Answer 21

Remarkably, a package update of bind on CentOS 7 broke named, now stating in the log that /etc/named.conf had a permissions problem. It had worked well for months with 0640. Now it wants 0644. This makes sense as the named daemon belongs to the 'named' user.

With named down everything was slow, from ssh logins to page serving from the local web server, sluggish LAMP apps etc, most probably because every request would time out on the dead local server before looking up to an external, secondary DNS that is configured.

Answer 22

If ssh taking time then we can follow the steps : There are 2 files

1. ssh_config
2. sshd_config --> will change in sshd_config

We can uncomment the line no 115

[root@mycentos ~]# vi /etc/ssh/sshd_config

UseDNS no -> default #UseDNS no --> remove # -> UseDNS no only

Save the sshd file and restart the sshd service

[root@mycentos ~]# systemctl restart sshd

If not,again take time then we can follow below change more : (Uncomment below lines )

1. GatewayPorts no

2. PermitTTY yes

3. UseDNS no -> line no 115

4. GSSAPIAuthentication no -> line no 79

Restart the sshd service and try to take the ssh

[root@mycentos ~]# systemctl restart sshd

Answer 23

I found that when I was having this issue, I had to update /etc/resolv.conf and just point to 8.8.8.8 as my internal DNS servers were not working properly. Even though I was connecting to my server via IP address...DNS check was still happening.

Answer 24

ssh -vv host showed two times Operation timed out to different ip adresses. drill host showed that fritz.box answered with three ip adresses, two of which were no longer existent.

Answer 25

For me I needed GSSAPI, and I didn't want to turn off reverse DNS lookups. That just didn't seem like a good idea, so I checked out the main page for resolv.conf. It turns out that a firewall between me and the servers I was SSHing to, was interfering with DNS requests, because they weren't in a form that the firewall expected. In the end, all I needed to do was add this line to resolv.conf on the servers that I was SSHing to:

options single-request-reopen​

Answer 26

For me there was an issue in my local /etc/hosts file. So ssh was trying two different IP (one wrong) which took forever to time-out.

Using ssh -v did the trick here:

$ ssh -vvv remotesrv
OpenSSH_6.7p1 Debian-5, OpenSSL 1.0.1k 8 Jan 2015
debug1: Reading configuration data /home/mathieu/.ssh/config
debug1: /home/mathieu/.ssh/config line 60: Applying options for remotesrv
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to remotesrv [192.168.0.10] port 22.
debug1: connect to address 192.168.0.10 port 22: Connection timed out
debug1: Connecting to remotesrv [192.168.0.26] port 22.
debug1: Connection established.