1

Debugging Weird sshd Connection Problems + What Happens When You Stop sshd


Posted by Artem Russakovskii on August 15th, 2008 in Linux

So the other day I was setting up public key authentication for one of my users, which is usually very straightforward: generate a private/public key pair, stick the private key into user's .ssh dir, set dir permissions to 0700, private key permissions to 0600, stick the public key into the authorized_keys file on the server, and the job's done. However, this time, no matter what I was doing, the public key was being rejected or ignored and the system was moving on to the keyboard-interactive authentication.

Debugging on the client side with -v didn't help much:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[email protected]:~/svn/b2/Fetch/LinkChecker> ssh -v [email protected]
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
...
lots of boring shit
...
debug1: Found key in /home/artem/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
...
more boring shit
...
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key:
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key:
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: /home/artem/.ssh/id_rsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /home/artem/.ssh/id_dsa
debug1: Next authentication method: keyboard-interactive
Password:

After breaking my head over possible reasons why the pile of junk that thinks it's smarter than me next to my feet doesn't work, kicking it a few times, and observing the same result, I turned to debugging the ssh daemon itself – sshd.

  • The -d option disables the daemon mode and enables debug mode, in which only 1 connection is accepted for the lifetime of the server, after which it simply quits.
  • -dd simply enables a more detailed output.
  • -e switches this debug output from a log file to STDOUT.

However, to free up port 22, I had to stop the daemon that was already running, or else a "Bind to port 22 on 0.0.0.0 failed: Address already in use." error appeared (duh). An interesting question though, especially for people doing this to remote boxes, what happens when one stops sshd? Ever thought of doing that but instead ran over to your mommy crying like a little girl? Well, fear no more, because I'll tell you exactly what happens:

  1. New users will have their connection refused.
  2. Your own connection will not be interrrupted. sshd works by spawning a new instance of itself for every incoming connection, so your own sshd process will stay in memory.

So where was I?

1
/usr/sbin/sshd -dd -e
1
2
3
4
...
Authentication refused: bad ownership or modes for directory /home/monkey
...
Failed publickey for monkey from 192.168.1.30 port 56287 ssh2

AhA!! (emphasis on the last 'a'). What have we here?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[email protected]:~> cd /home/
[email protected]:/home/> l
drwxrwx--- 29 monkey  users 4096 2008-08-06 23:14 monkey/
 
DeathStar:/home/ # chmod 755 monkey
drwxr-xr-x 29 monkey  users 4096 2008-08-06 23:14 monkey/
 
[email protected]:~/svn/b2/Fetch/LinkChecker> ssh -v [email protected]
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 192.168.1.30 [192.168.1.30] port 22.
debug1: Connection established.
debug1: identity file /home/artem/.ssh/id_rsa type 1
debug1: identity file /home/artem/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.6
debug1: match: OpenSSH_4.6 pat OpenSSH*
...

Connection established, all systems are go, the key has been accepted.

Inspired by http://linux.derkeiler.com/Mailing-Lists/Fedora/2005-08/1105.html

P.S. Don't forget to /etc/init.d/sshd start. 😉

● ● ●

Artem Russakovskii is a San Francisco programmer, blogger, and future millionaire (that last part is in the works). Follow Artem on Twitter (@ArtemR) or subscribe to the RSS feed.

In the meantime, if you found this article useful, feel free to buy me a cup of coffee below.



Share
  • http://omninoggin.com Thaya Kareeson

    Pretty good sshd ninja debugging skills. I never had the balls to try to kill sshd and see what happens to your current shell (I thought it would just die). It's a lot less obvious than "rm -f /bin/rm" working properly.