Debugging Weird sshd Connection Problems + What Happens When You Stop sshd
So the other day I was setting up public key authentication for one of my users, which is usually very straightforward: generate a private/public key pair, stick the private key into user's .ssh dir, set dir permissions to 0700, private key permissions to 0600, stick the public key into the authorized_keys file on the server, and the job's done. However, this time, no matter what I was doing, the public key was being rejected or ignored and the system was moving on to the keyboard-interactive authentication.
Debugging on the client side with -v didn't help much:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | artem@DeathStar:~/svn/b2/Fetch/LinkChecker> ssh -v monkey@192.168.1.30 OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007 ... lots of boring shit ... debug1: Found key in /home/artem/.ssh/known_hosts:1 debug1: ssh_rsa_verify: signature correct ... more boring shit ... debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Next authentication method: publickey debug1: Offering public key: debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering public key: debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Offering public key: /home/artem/.ssh/id_rsa debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Trying private key: /home/artem/.ssh/id_dsa debug1: Next authentication method: keyboard-interactive Password: |
After breaking my head over possible reasons why the pile of junk that thinks it's smarter than me next to my feet doesn't work, kicking it a few times, and observing the same result, I turned to debugging the ssh daemon itself – sshd.
- The -d option disables the daemon mode and enables debug mode, in which only 1 connection is accepted for the lifetime of the server, after which it simply quits.
- -dd simply enables a more detailed output.
- -e switches this debug output from a log file to STDOUT.
However, to free up port 22, I had to stop the daemon that was already running, or else a "Bind to port 22 on 0.0.0.0 failed: Address already in use." error appeared (duh). An interesting question though, especially for people doing this to remote boxes, what happens when one stops sshd? Ever thought of doing that but instead ran over to your mommy crying like a little girl? Well, fear no more, because I'll tell you exactly what happens:
- New users will have their connection refused.
- Your own connection will not be interrrupted. sshd works by spawning a new instance of itself for every incoming connection, so your own sshd process will stay in memory.
So where was I?
1 | /usr/sbin/sshd -dd -e |
1 2 3 4 | ... Authentication refused: bad ownership or modes for directory /home/monkey ... Failed publickey for monkey from 192.168.1.30 port 56287 ssh2 |
AhA!! (emphasis on the last 'a'). What have we here?
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | artem@DeathStar:~> cd /home/ artem@DeathStar:/home/> l drwxrwx--- 29 monkey users 4096 2008-08-06 23:14 monkey/ DeathStar:/home/ # chmod 755 monkey drwxr-xr-x 29 monkey users 4096 2008-08-06 23:14 monkey/ artem@DeathStar:~/svn/b2/Fetch/LinkChecker> ssh -v monkey@192.168.1.30 OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to 192.168.1.30 [192.168.1.30] port 22. debug1: Connection established. debug1: identity file /home/artem/.ssh/id_rsa type 1 debug1: identity file /home/artem/.ssh/id_dsa type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_4.6 debug1: match: OpenSSH_4.6 pat OpenSSH* ... |
Connection established, all systems are go, the key has been accepted.
Inspired by http://linux.derkeiler.com/Mailing-Lists/Fedora/2005-08/1105.html
P.S. Don't forget to /etc/init.d/sshd start. 
Artem Russakovskii is a San Francisco programmer, blogger, and future millionaire (that last part is in the works). Follow Artem on Twitter (@ArtemR) or subscribe to the RSS feed.
In the meantime, if you found this article useful, feel free to buy me a cup of coffee below.
beer planet is a blog about technology, programming, computers, and geek life. It is run by Artem Russakovskii - a local San Francisco geek who currently works at 
Artem loves commenting on other people's blogs and reading comments on his own, so go ahead and leave one.
Pretty good sshd ninja debugging skills. I never had the balls to try to kill sshd and see what happens to your current shell (I thought it would just die). It's a lot less obvious than "rm -f /bin/rm" working properly.
[...] public links >> publickey Debugging Weird sshd Connection Problems + What Happens When You… Saved by dividebynought on Thu 29-1-2009 Ssh connection without password Saved by zai on Wed [...]