Step 5.1 General Troubleshooting

The primary purpose of this guide is to provide basic, step-by-step instructions for quickly installing, configuring, and using OpenSSH. Of course, as with most new ventures , you will most likely run into unexpected problems along the way. OpenSSH runs on numerous platforms so of course it would be an incredible, if not impossible , task to address all known problems encountered on every platform. This section was never intended to address every known problem but rather is intended to be a guide to help with general troubleshooting of SSH issues and also to address specific common problems.

You may have other troubleshooting techniques to share or may learn of solutions to new problems you encounter “ if you would like to share your findings, you are encouraged to do so via e-mail to openssh@sans.org.

Action 5 1 1 Verify download and installation configuration of software

If you are encountering problems just getting OpenSSH to run, and/or are receiving error messages that indicate an install or configure problem, then of course the first place to start is to review the installation/configuration steps for the OpenSSH pre-requisite software and for OpenSSH itself (see Section I). Remember that certain software - such as Zlib and OpenSSL - must be installed before OpenSSH, since OpenSSH relies on them.

Before you get into specific install reviews, it's a good idea to make sure that you have a valid download file. It might appear obvious that you don't have a download problem if you were able to "unpack", install and compile the source code “ but stranger things have happened . Make sure that the size of the downloaded file (typically .tar, .Z, .gz, .zip) is equal to that provided (usually) by the download site. If you performed an FTP download, check the site to determine if it was necessary to perform a BIN (binary) transfer ” and if so, make sure that you did so (To perform a binary transfer, enter "BIN" at the "ftp>" prompt). You should also verify any checksums provided for downloaded files.

To ensure that the OpenSSH content is complete and has not been corrupted in transit or maliciously modified, the PGP signature should also be checked. Note that a trojaned version of OpenSSH has previously been placed on hacked mirror FTP sites (See CERT CA-2002-24 for details). Unfortunately, the where- abouts of the OpenSSH public key are not well publicized at the time of this writing. The public key currently used to sign OpenSSH belongs to Damien Miller (Personal Key) and may be retrieved from PGP key servers using id 86FF9C48. The following example commands use the open source GNU Privacy Guard (GPG) from http://www.gnupg.org/.

The key is also available from the mirror FTP sites, however, if you retrieve it from the same FTP server from which you downloaded the software, you run the risk that both have been maliciously altered . Once the key is imported, the tar bundle can be verified . It is not necessary to assign any trust to the key.

$> gpg --keyserver wwwkeys.pgp.net --recv-keys 86FF9C48 $> gpg --list-keys 86FF9C48 pub 1024D/86FF9C48 2001-02-26 Damien Miller (Personal Key) sub 2048g/AA2B1C41 2001-02-26 $> gpg --verify openssh-3.5p1.tar.gz.sig openssh-3.5p1.tar.gz gpg: Signature made Thu Oct 3 21:34:43 2002 EDT using DSA key ID 86FF9C48 gpg: Good signature from "Damien Miller (Personal Key) " Could not find a valid trust path to the key. Let's see whether we can assign some missing owner trust values. No path leading to one of our keys found. gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. gpg: Fingerprint: 3981 992A 1523 ABA0 79DB FC66 CE8E CB03 86FF 9C48

Many open-source tools provide a significant amount of verbose output when they are installed, so it can be easy to overlook error messages. Therefore, it's a good idea to review any install messages/logs for the pre-requisite software if you still have them. If these are not available, then you may still be able to validate the installations “ most of the open-source tools come with helpful installation/configuration documentation. Check for INSTALL and README files within the directories where you stored the source code for each package. These files frequently provide good detail and can be of great help. For example, notice the INSTALL and README files that are provided with the OpenSSL distribution:

# pwd /usr/local/openssl-0.9.6g # ls CHANGES MacOS apps include perl CHANGES.SSLeay Makefile bugs install.com rehash.time Configure Makefile.org certs libRSAglue.a rsaref FAQ Makefile.ssl config libcrypto.a shlib INSTALL NEWS crypto libssl.a ssl INSTALL.MacOS PROBLEMS demos makevms.com test INSTALL.VMS README doc ms times INSTALL.W32 README.ENGINE e_os.h openssl.doxy tools LICENSE VMS e_os2.h openssl.spec util

Action 5 1 2 Verify OpenSSH commands are in the UNIX search path

If you get a "not found" error when trying to invoke one of the OpenSSH executables (ssh, slogin , scp, sftp, etc), then make sure that the UNIX PATH environment variable includes the location of the executables; otherwise the OpenSSH commands' location will have to be provided when the executables are invoked. For example:

$ ssh server.example.com ssh: not found $ echo $PATH /usr/sbin:/usr/bin:/usr/dt/bin:/usr/openwin/bin:/bin:/usr/ucb:/usr/ccs/bin

To update the search PATH, first verify the location of the executables. For example, if you installed the executables in the /usr/local/bin directory, the following commands in a Bourne-, Korn-, or Z-Shell will ensure that this directory will be searched:

$ PATH=$PATH:/usr/local/bin; export PATH $ echo $PATH /usr/sbin:/usr/bin:/usr/dt/bin:/usr/openwin/bin:/bin:/usr/ucb:/usr/ccs/bin:/usr/local/bin

A subsequent SSH attempt is now successful:

$ ssh server.example.com sshuser@server.example.com's password: ******* Last login: Mon Feb 17 14:05:03 2003 from client.example.com $

Action 5 1 3 Utilize verbose option to OpenSSH commands

Sometimes the “v verbose option available with most of the OpenSSH commands can provide insight. This option provides much more detail than is normally received. For example, here is the previous connection attempt using the “v flag:

$ ssh -v server.example.com OpenSSH_3.5p1, SSH protocols 1.5/2.0, OpenSSL 0x0090607f debug1: Reading configuration data /usr/local/etc/ssh_config debug1: Rhosts Authentication disabled, originating port will not be trusted. debug1: ssh_connect: needpriv 0 debug1: Connecting to server.example.com [172.16.121.6] port 22. debug1: Connection established. debug1: identity file /home/sshuser/.ssh/id_rsa type 1 debug1: Remote protocol version 2.0, remote software version OpenSSH_3.4p1 debug1: match: OpenSSH_3.4p1 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_3.5p1 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-cbc hmac-md5 none debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: dh_gen_key: priv key bits set: 113/256 debug1: bits set: 1602/3191 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Host 'server.example.com' is known and matches the RSA host key. debug1: Found key in /home/sshuser/.ssh/known_hosts:1 debug1: bits set: 1595/3191 debug1: ssh_rsa_verify: signature correct debug1: kex_derive_keys debug1: newkeys: mode debug1: SSH2_MSG_NEWKEYS sent debug1: waiting for SSH2_MSG_NEWKEYS debug1: newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: done: ssh_kex2. debug1: send SSH2_MSG_SERVICE_REQUEST debug1: service_accept: ssh-userauth debug1: got SSH2_MSG_SERVICE_ACCEPT debug1: authentications that can continue: publickey,password,keyboard-interactive debug1: next auth method to try is publickey debug1: try pubkey: /export/home/sshuser/.ssh/id_rsa debug1: authentications that can continue: publickey,password,keyboard-interactive debug1: next auth method to try is keyboard-interactive debug1: authentications that can continue: publickey,password,keyboard-interactive debug1: next auth method to try is password sshuser@server.example.com's password

As you can see, the verbose flag provides helpful debugging information - in this example, the client " knows " about the server (i.e. this is not the first SSH connection attempt), the client and server have "negotiated" to use the SSHv2 protocol, and public key authentication has failed (possibly this type of authentication has not yet been configured or the permissions for the file containing the private key have not been set to a secure level), so the UNIX password authentication is tried next (default behavior for SSH authentication).

Action 5 1 4 Utilize OpenSSH server daemon log

By default, the OpenSSH server daemon (sshd) logs certain information, such as authentication failures, to the UNIX syslog facility. These messages can sometimes provide insight, especially if the problem is just an incorrect password or passphrase. For example, here is an entry indicating an authentication failure on an OpenSSH server:

Feb 19 13:37:50 server.example.com sshd[10024]: [ID 800047 auth.crit] fatal: Timeout before authentication for 172.16.121.16.

You can get much more verbose output in the OpenSSH server log file by restarting the sshd daemon with the -d (debug) command line option. For example, on the OpenSSH server:

# /usr/local/sbin/sshd -d & [1] 14027

If you use the “d flag, be prepared to wade through a lot of detail. The following is the verbose output from a client connection whereby the public key authentication failed and therefore UNIX password authentication was used (which is the default sshd behavior):

debug1: sshd version OpenSSH_3.4p1 debug1: read PEM private key done: type RSA debug1: private host key: #0 type 1 RSA debug1: read PEM private key done: type DSA debug1: private host key: #1 type 2 DSA debug1: Bind to port 22 on ::. Server listening on :: port 22. debug1: Bind to port 22 on 0.0.0.0. Server listening on 0.0.0.0 port 22. debug1: Server will not fork when running in debugging mode. Connection from 172.16.121.16 port 33672 debug1: Client protocol version 2.0; client software version OpenSSH_3.5p1 debug1: match: OpenSSH_3.5p1 pat OpenSSH* Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_3.4p1 debug1: list_hostkey_types: ssh-rsa,ssh-dss debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: client->server aes128-cbc hmac-md5 none debug1: kex: server->client aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent debug1: dh_gen_key: priv key bits set: 133/256 debug1: bits set: 1590/3191 debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT debug1: bits set: 1617/3191 debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent debug1: kex_derive_keys debug1: newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: waiting for SSH2_MSG_NEWKEYS debug1: newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: KEX done debug1: userauth-request for user sshuser service ssh-connection method none debug1: attempt 0 failures 0 Failed none for sshuser from 172.16.121.16 port 33672 ssh2 Failed none for sshuser from 172.16.121.16 port 33672 ssh2 debug1: userauth-request for user sshuser service ssh-connection method publickey debug1: attempt 1 failures 1 debug1: test whether pkalg/pkblob are acceptable debug1: temporarily_use_uid: 1009/150 (e=0) debug1: trying public key file /home/sshuser/.ssh/authorized_keys debug1: matching key found: file /home/sshuser/.ssh/authorized_keys, line 1 Found matching RSA key: a4:93:99:99:29:9f:41:9d:af:21:54:b6:8f:3e:ce:c6 debug1: restore_uidPostponed publickey for sshuser from 172.16.121.16 port 33672 ssh2 debug1: userauth-request for user sshuser service ssh-connection method keyboard-interactive debug1: attempt 2 failures 1 debug1: keyboard-interactive devs debug1: auth2_challenge: user=sshuser devs= debug1: kbdint_alloc: devices "Failed keyboard-interactive for sshuser from 172.16.121.16 port 33672 ssh2 debug1: userauth-request for user sshuser service ssh-connection method password debug1: attempt 3 failures 2Failed password for sshuser from 172.16.121.16 port 33672 ssh2Failed password for sshuser from 172.16.121.16 port 33672 ssh2 debug1: userauth-request for user sshuser service ssh-connection method password debug1: attempt 4 failures 3 Accepted password for sshuser from 172.16.121.16 port 33672 ssh2 debug1: monitor_child_preauth: sshuser has been authenticated by privileged process Accepted password for sshuser from 172.16.121.16 port 33672 ssh2 debug1: newkeys: mode 0 debug1: newkeys: mode 1 debug1: Entering interactive session for SSH2. debug1: fd 4 setting O_NONBLOCK debug1: fd 5 setting O_NONBLOCK debug1: server_init_dispatch_20 debug1: server_input_channel_open: ctype session rchan 0 win 65536 max 16384 debug1: input_session_request debug1: channel 0: new [server-session] debug1: session_new: init debug1: session_new: session 0 debug1: session_open: channel 0 debug1: session_open: session 0: link with channel 0 debug1: server_input_channel_open: confirm session debug1: server_input_channel_req: channel 0 request pty-req reply 0 debug1: session_by_channel: session 0 channel 0 debug1: session_input_channel_req: session 0 req pty-req debug1: Allocating pty. debug1: session_new: init debug1: session_new: session 0 debug1: session_pty_req: session 0 alloc /dev/pts/22 debug1: server_input_channel_req: channel 0 request x11-req reply 0 debug1: session_by_channel: session 0 channel 0 debug1: session_input_channel_req: session 0 req x11-req debug1: fd 12 setting O_NONBLOCK debug1: channel 1: new [X11 inet listener] debug1: fd 13 setting O_NONBLOCK debug1: channel 2: new [X11 inet listener] debug1: server_input_channel_req: channel 0 request shell reply 0 debug1: session_by_channel: session 0 channel 0 debug1: session_input_channel_req: session 0 req shell

Action 5 1 5 Run the OpenSSH Server in foreground verbose mode

Running the server in foreground debug mode ( “D “d ) is very useful for debugging a wide variety of connection problems. When run in the foreground, the server will need to be restarted for each connection attempt:

# /usr/local/sbin/sshd -D -d debug1: Seeding random number generator debug1: sshd version OpenSSH_3.0.1p1 debug1: private host key: #0 type 0 RSA1 debug1: read PEM private key done: type RSA debug1: private host key: #1 type 1 RSA debug1: read PEM private key done: type DSA debug1: private host key: #2 type 2 DSA debug1: Bind to port 22 on 192.168.102.12. Server listening on 192.168.102.12 port 22. debug1: Bind to port 22 on 216.55.177.197. Server listening on 192.168.102.12 port 22. debug1: Server will not fork when running in debugging mode. Connection from 10.10.102.84 port 1446 debug1: Client protocol version 2.0; client software version PuTTY-Release-0.53b debug1: no match: PuTTY-Release-0.53b Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_3.0.1p1 debug1: Rhosts Authentication disabled, originating port 1446 not trusted. debug1: list_hostkey_types: ssh-rsa,ssh-dss debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: client->server 3des-cbc hmac-sha1 none debug1: kex: server->client 3des-cbc hmac-sha1 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST_OLD received debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent debug1: dh_gen_key: priv key bits set: 193/384 debug1: bits set: 1595/3191 debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT debug1: bits set: 1599/3191 debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent debug1: kex_derive_keys debug1: newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: waiting for SSH2_MSG_NEWKEYS debug1: newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: KEX done