Debugging Weird sshd Connection Problems + What Happens When You Stop sshd

Artem Russakovskii — Fri, 15 Aug 2008 16:14:53 +0000

So the other day I was setting up public key authentication for one of my users, which is usually very straightforward: generate a private/public key pair, stick the private key into user's .ssh dir, set dir permissions to 0700, private key permissions to 0600, stick the public key into the authorized_keys file on the server, and the job's done. However, this time, no matter what I was doing, the public key was being rejected or ignored and the system was moving on to the keyboard-interactive authentication.

Debugging on the client side with -v didn't help much:

artem@DeathStar:~/svn/b2/Fetch/LinkChecker> ssh -v monkey@192.168.1.30
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
...
lots of boring shit
...
debug1: Found key in /home/artem/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
...
more boring shit
...
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key:
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key:
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: /home/artem/.ssh/id_rsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /home/artem/.ssh/id_dsa
debug1: Next authentication method: keyboard-interactive
Password:

After breaking my head over possible reasons why the pile of junk that thinks it's smarter than me next to my feet doesn't work, kicking it a few times, and observing the same result, I turned to debugging the ssh daemon itself – sshd.

The -d option disables the daemon mode and enables debug mode, in which only 1 connection is accepted for the lifetime of the server, after which it simply quits.
-dd simply enables a more detailed output.
-e switches this debug output from a log file to STDOUT.

However, to free up port 22, I had to stop the daemon that was already running, or else a "Bind to port 22 on 0.0.0.0 failed: Address already in use." error appeared (duh). An interesting question though, especially for people doing this to remote boxes, what happens when one stops sshd? Ever thought of doing that but instead ran over to your mommy crying like a little girl? Well, fear no more, because I'll tell you exactly what happens:

New users will have their connection refused.
Your own connection will not be interrrupted. sshd works by spawning a new instance of itself for every incoming connection, so your own sshd process will stay in memory.

So where was I?

1	/usr/sbin/sshd -dd -e

...
Authentication refused: bad ownership or modes for directory /home/monkey
...
Failed publickey for monkey from 192.168.1.30 port 56287 ssh2

AhA!! (emphasis on the last 'a'). What have we here?

artem@DeathStar:~> cd /home/
artem@DeathStar:/home/> l
drwxrwx--- 29 monkey  users 4096 2008-08-06 23:14 monkey/
 
DeathStar:/home/ # chmod 755 monkey
drwxr-xr-x 29 monkey  users 4096 2008-08-06 23:14 monkey/
 
artem@DeathStar:~/svn/b2/Fetch/LinkChecker> ssh -v monkey@192.168.1.30
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 192.168.1.30 [192.168.1.30] port 22.
debug1: Connection established.
debug1: identity file /home/artem/.ssh/id_rsa type 1
debug1: identity file /home/artem/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.6
debug1: match: OpenSSH_4.6 pat OpenSSH*
...

Connection established, all systems are go, the key has been accepted.

Inspired by http://linux.derkeiler.com/Mailing-Lists/Fedora/2005-08/1105.html

P.S. Don't forget to /etc/init.d/sshd start.

beer planet » stop

Debugging Weird sshd Connection Problems + What Happens When You Stop sshd