June 8, 2010

Building RPM packages as non-root user

Filed under: Security — Tags: , — martin @ 11:38 am

This came up below the posting about Packaging OpenSSH on CentOS:

For non-root building of RPMs, the user needs a configuration file named .rpmmacros in his home directory:

cat <<EOF > ~/.rpmmacros
%_topdir $HOME/rpm
%_tmppath $HOME/rpm/tmp

The directory structure for RPM needs to be prepared by hand:
mkdir ~/rpm
mkdir ~/rpm/BUILD
mkdir ~/rpm/RPMS
mkdir ~/rpm/RPMS/`uname -p`
mkdir ~/rpm/RPMS/noarch
mkdir ~/rpm/SOURCES
mkdir ~/rpm/SPECS
mkdir ~/rpm/SRPMS
mkdir ~/rpm/tmp

Now build, for example, OpenSSH:
$ cp openssh-5.5p1/contrib/redhat/openssh.spec ~/rpm/SPECS/
$ cp openssh-5.5p1.tar.gz ~/rpm/SOURCES/
$ cd ~/rpm/SPECS/
$ rpmbuild -bb --define 'skip_x11_askpass 1' --define 'skip_gnome_askpass 1' openssh.spec
$ cd ~/rpm/RPMS/`uname -i`
$ ls -l
total 1108
-rw-r--r-- 1 martin martin 359994 Jun 8 12:33 openssh-5.5p1-1.i386.rpm
-rw-r--r-- 1 martin martin 471360 Jun 8 12:33 openssh-clients-5.5p1-1.i386.rpm
-rw-r--r-- 1 martin martin 284453 Jun 8 12:33 openssh-server-5.5p1-1.i386.rpm


April 24, 2009

OpenSSH connection multiplexing

Filed under: Security, UNIX & Linux — Tags: , , — martin @ 6:44 am

The Challenge

I was in touch with a developer the other day who used SSH to programmatically connect to a remote machine where he would start some kind of processing job. Unfortunately, he was in trouble when he wanted to kill the remote process. Killing the local SSH client would leave his job active. He claimed that there used to be some sort of signal forwarding feature in OpenSSH on the machine where he had developed his application in OpenSSH 3.x days, but this feature seems to have been removed by now.

I wasn’t able to confirm anything of this, but this gentleman’s problem got me curious. I started to wonder: Is there some kind of sideband connection that I might use in SSH to interact with a program that is running on a remote machine?

The first thing I thought of were port forwards. These might actually be used to maintain a control channel to a running process on the other side. On the other hand, sockets aren’t trivial to implement for a /bin/ksh type of guy, such as the one I was dealing with. Also, this approach just won’t scale. Coordination of local and remote ports is bound to turn into a bureaucratic nightmare.

I then started to skim the SSH man pages for anything that looked like a “sideband”, “session control” or “signaling” feature. What I found, were the options ControlMaster and ControlPath. These configure connection multiplexing in SSH.

Proof Of Concept

Manual one-shot multiplexing can be demonstrated using the -M and -S options:

1) The first connection to the remote machine is opened in Master mode (-M). A UNIX socket is specified using the -S option. This socket enables the connection to be shared with other SSH clients:

localhost$ ssh -M -S ~/.ssh/controlmaster.test.socket remotehost

2) A second SSH session is attached to the running session. The socket that was opened before is specified with the -S option. The remote shell opens without further authentication:

localhost$ ssh -S ~/.ssh/controlmaster.test.socket remotehost

The interesting thing about this is that we now have two login sessions running on the remote machine, who are children of the same sshd process:

remotehost$ pstree -p $PPID

What About The Original Challenge?

Well, he can start his transaction by connecting to the remote machine in Master mode. For simplicity’s sake, let’s say he starts top in one session and wants to be able to kill it from another session:

localhost$ ssh -t -M -S ~/.ssh/controlmaster.mytopsession.socket remotehost top

Now he can pick up the socket and find out the PIDs of all other processes running behind the same SSH connection:

localhost$ ssh -S ~/.ssh/controlmaster.mytopsession.socket remotehost 'ps --ppid=$PPID | grep -v $$'
  PID TTY          TIME CMD
 4390 pts/0    00:00:00 top

This, of course, leads to:

localhost$ ssh -S ~/.ssh/controlmaster.mytopsession.socket remotehost 'ps --no-headers -o pid --ppid=$PPID | grep -v $$ | xargs kill'

Then again, our shell jockey could just use PID or touch files. I think this is what he’s doing now anyway.

Going Fast And Flexible With Multiplexed Connections

With my new developer friend’s troubles out of the way, what else could be done with multiplexed connections? The SSH docs introduce “opportunistic session sharing”, which I believe might actually be quite useful for me.

It is possible to prime all SSH connections with a socket in ~/.ssh/config. If the socket is available, the actual connection attempt is bypassed and the ssh client hitches a ride on a multiplexed connection. In order for the socket to be unique per multiplexed connection, it should be assigned a unique name through the tokens %r (remote user), %h (remote host) and %p (destination port):

ControlPath ~/.ssh/controlmaster.socket.%r.%h.%p
# Will create socket as e.g.: ~/.ssh/controlmaster.socket.root.remotehost.example.com.22

If there is no socket available, SSH connects directly to the remote host. In this case, it is possible to automatically pull up a socket for subsequent connections using the following option in ~/.ssh/config:

ControlMaster auto

So Where’s The Actual Benefit?

I use a lot of complex proxied SSH connections who take ages to come up. However, connecting through an already established connection goes amazingly fast:

# Without multiplexing:
localhost$ time ssh remotehost /bin/true
real    0m1.376s
# With an already established shared connection:
localhost$ time ssh remotehost /bin/true
real    0m0.129s

I will definitely give this a try for a while, to see if it is usable for my daily tasks.

Update, 2009/05/04: No, it isn’t. Disconnecting slave sessions upon logout of the master session are too much of a nuisance for me.

February 27, 2009

Packaging OpenSSH on CentOS

Filed under: Security, UNIX & Linux — Tags: , , , , — martin @ 8:29 am

March 30, 2010: It was pointed out to me that Redhat has backported chroot functionality into its OpenSSH 4.3 packages, so these directions may not be neccessary anymore.

My article on chrooted SFTP has turned out to be the most popular article on this blog. What a pity that its “companion article” on building current OpenSSH on CentOS 5 is such a bloody hell of a mess.

Fortunately, reader Simon pointed out a really simple method for building RPMs from current OpenSSH sources in a comment. We had the chance to try this out in a production deployment of chrooted SFTP the other day, and what can I say? It just works(tm)! Thanks a lot, dude! 🙂

# yum install gcc
# yum install openssl-devel
# yum install pam-devel
# yum install rpm-build

It certainly doesn’t hurt to make the GPG check a habit:

# wget http://ftp.bit.nl/mirror/openssh/openssh-5.2p1.tar.gz
# wget http://ftp.bit.nl/mirror/openssh/openssh-5.2p1.tar.gz.asc
# wget -O- http://ftp.bit.nl/mirror/openssh/DJM-GPG-KEY.asc | gpg –-import
# gpg openssh-5.2p1.tar.gz.asc
gpg: Signature made Mon 23 Feb 2009 01:18:28 AM CET using DSA key ID 86FF9C48
gpg: Good signature from "Damien Miller (Personal Key) "
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3981 992A 1523 ABA0 79DB FC66 CE8E CB03 86FF 9C48

Prepare, build and install the RPM. Disable the building of GUI components in the spec file. We don’t need these on a server:

# tar zxvf openssh-5.2p1.tar.gz
# cp openssh-5.2p1/contrib/redhat/openssh.spec /usr/src/redhat/SPECS/
# cp openssh-5.2p1.tar.gz /usr/src/redhat/SOURCES/
# cd /usr/src/redhat/SPECS
# perl -i.bak -pe 's/^(%define no_(gnome|x11)_askpass)\s+0$/$1 1/' openssh.spec
# rpmbuild -bb openssh.spec
# cd /usr/src/redhat/RPMS/`uname -i`
# ls -l
-rw-r--r-- 1 root root 275808 Feb 27 08:08 openssh-5.2p1-1.x86_64.rpm
-rw-r--r-- 1 root root 439875 Feb 27 08:08 openssh-clients-5.2p1-1.x86_64.rpm
-rw-r--r-- 1 root root 277714 Feb 27 08:08 openssh-server-5.2p1-1.x86_64.rpm
# rpm -Uvh openssh*rpm
Preparing... ########################################### [100%]
1:openssh ########################################### [ 33%]
2:openssh-clients ########################################### [ 67%]
3:openssh-server ########################################### [100%]
# service sshd restart

The RPM should install cleanly on CentOS 4. On CentOS 5, after installation, service ssh restart throws a warning that initlog is obsolete. I work around this by keeping a copy of the old /etc/init.d/sshd and restoring it after RPM installation.

October 20, 2008

OpenSSH: Going flexible with forced commands

Filed under: Security, UNIX & Linux — Tags: , — martin @ 9:32 am

As we all know, it is possible to use SSH not only for obtaining an interactive login session on a remote machine, but also for executing commands remotely. For instance, the following command will log on to myserver.example.com, execute “uname -a” and return to the local shell:

ssh myserver.example.com uname -a

(The local SSH client returns the exit code from the remote command, if you’re into this kind of detail.)

You might have some users (or scheduled automatisms) that you don’t want to be able to log on to that machine at all, but who should be permitted to execute only a given command. In order to achieve this, you can configure key-based authentication. Once this has been done, the key can be prefixed with a number of configuration options. Using one of these options, it is possible to enforce execution of a given command when this key is used for authentication.

In this example from ~/.ssh/authorized_keys, the user wants to look at the process list, so we set the command to “ps -ef”.

command="/bin/ps -ef"

Using this, when the user tries to log in, or tries to execute an arbitrary command, “/bin/ps -ef” is executed instead and the SSH session terminates.

In addition to enforcing a command, it is advisable to disable a number of advanced SSH features, such as TCP and X11 forwarding. Assignment of a pseudo terminal to the user’s SSH session may also be suppressed, by adding a number of additional configuration options next to the forced command:


Here’s what a full entry from ~/.ssh/authorized_keys might look like:

command="/bin/ps -ef",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAp0KMipajKK468mfihpZHqmrMk8w+PmzTnJrZUFYZZNmLkRk+icn+m71DdEHmza2cSf9WdiK7TGibGjZTE/Ez0IEhYRj5RM3dKkfYqitKTKlxVhXNda7az6VqAJ/jtaBXAMTjHeD82xlFoghLZOMkScTdWmu47FyVkv/IM1GjgX/I8s4307ds1M+sICyDUmgxUQyNF3UnAduPn1m8ux3V8/xAqPF+bRuFlB0fbiAEsSu4+AkvfX7ggriBONBR6eFexOvRTBWtriHsCybvd6tOpJHN8JYZLxCRYHOGX+sY+YGE4iIePKVf2H54kS5UlpC/fnWgaHbmu/XsGYjYrAFnVw== Test key

This is quite nice: We have successfully limited this user to requesting a process list.

This is called an SSH forced command.

So much for the introduction. 😀

Here’s what I’m really getting at today – What, if we want the user to not only execute a single command, but a number of commands, such as:

– Show process list (ps)
– Show virtual memory statistics (vmstat)
– Stop and start the print server (/etc/init.d/cupsys stop/start)

Following the approach described above, this would give us four key pairs, four entries in ~/.ssh/authorized_keys, and four entirely different invocations of SSH on the client side, each of them using a dedicated private key. In other words: An administrative nightmare.

This is where the environment variable $SSH_ORIGINAL_COMMAND comes in. (This nice facility was pointed out to me last week by G., who had read about it somewhere but wondered what it might be useful for.)

Until now, all we know is that with a forced command in place, the SSH server ignores the command requested by the user. This is not entirely true, though. The SSH server does in fact remember the command that was requested, stores it in $SSH_ORIGINAL_COMMAND and thus makes it available within the environment of the forced command.

With this in mind, it is possible to allow more flexibility inside forced commands, without the need to go crazy with countless key pairs. Instead, it is possible to just create a wrapper script that is called as the forced command from within ~/.ssh/authorized_keys and decides what to do, based on the content of $SSH_ORIGINAL_COMMAND:

# Script: /usr/local/bin/wrapper.sh 

		ps -ef
		vmstat 1 100
	"cups stop")
		/etc/init.d/cupsys stop
	"cups start")
		/etc/init.d/cupsys start
		echo "Sorry. Only these commands are available to you:"
		echo "ps, vmstat, cupsys stop, cupsys start"
		exit 1

It is important to be aware of potential security issues here, such as the user escaping to a shell prompt from within one of the listed commands. Setting the “no-pty” option already makes this kind of attack somewhat difficult. In addition, some programs, such as “top”, for example, have special options to run them in a “secure” read-only mode. It is advisable to closely examine all programs that are called as SSH forced commands for well-meant “backdoors” and to find out about securing them.

It’s up to you to decide based on your own situation, whether you want to run this wrapper as the root user or if you prefer to use password-less “sudo” commands to raise privileges where needed.

If you encounter problems while debugging $SSH_ORIGINAL_COMMAND, please make absolutely sure that you are authenticating with the correct key. I found it helpful to unset SSH_AUTH_SOCK in the window where I do my testing, in order to prevent intervention from identies stored in the SSH agent.

April 6, 2008

OpenSSH chrooted SFTP (e.g. for Webhosting)

Filed under: Security — Tags: , — martin @ 8:53 am

Over at Denny, I discovered that OpenSSH, since version 4.8, supports chrooted SFTP operation. This is, of course, a feature that many of us have been waiting for, so I immediately gave it a try, using a somewhat adventurous manually compiled OpenSSH on CentOS 5. (Update, February 27, 2009: See Packaging OpenSSH on CentOS for a more coherent installation method.) I also had a little help from the Debian Administration Blog.

In order to enable chrooted SFTP for some users, we’ll first create a separate group for users that will get the chroot treatment. I named this group chrooted, for no obvious reason. This group will be assigned as a supplementary group for chroot users.

The common application for this will be virtual WWW hosting, so I started with the assumption that a website called http://www.example.com will be accessed by SFTP. The directory /vhost/www.example.com will therefore serve as the user’s home directory. In order to make chrooting work along with key-based authentication, I found that it neccessary to make the user name and the name of his home directory identical, so the user was named http://www.example.com as well (login shell: /bin/false), along with a similar user private group http://www.example.com. This looks as if it may have a tendency to get awkward, but it really only is a first test. I’ll have to invest a bit more thought before this goes into production.

The directory /vhost/www.example.com was created and populated like this:

drwxr-xr-x 5 root            root            4096 Apr  5 22:01 .
drwxr-xr-x 3 root            root            4096 Apr  5 21:22 ..
drwxrwxr-x 2 www.example.com www.example.com 4096 Apr  6 08:45 htdocs
drwxr-xr-x 2 root            root            4096 Apr  5 21:22 logs
drwxr-xr-x 2 www.example.com www.example.com 4096 Apr  5 22:02 .ssh

With the user created and his directory populated, we’ll now edit sshd_config as follows:

#Replace the OpenSSH sftp-server backend with its internal SFTP engine:
#Subsystem      sftp    /opt/openssh/libexec/sftp-server
Subsystem       sftp    internal-sftp
# Configure special treatment of members of the group chrooted:
Match group chrooted
         # chroot members into this directory
         # %u gets substituted with the user name:
         ChrootDirectory /vhost/%u
         X11Forwarding no
         AllowTcpForwarding no
         # Force the internal SFTP engine upon them:
         ForceCommand internal-sftp

I actually had a quite hard time figuring out the proper constellation of user name, user home directory and ChrootDirectory. ChrootDirectory applies after the user has been authenticated. Before that, his home directory from /etc/passwd still applies. In order to enable the user to maintain his SSH key and to enable sshd to find the key, both environments must be congruent. However, the chroot destination must not be owned by the user for security reasons; the user’s home directory therefore belongs to root. Tricky, isn’t it? I must admit, though, that this would have been a lot more intuitive if I hadn’t strayed away from /home on the very first test. Doh! 😮

Here’s a sample session with the user “www.example.com”, authenticated by public key:

$ sftp www.example.com@
Connecting to
sftp> ls -la
drwxr-xr-x    5 0        0            4096 Apr  5 20:01 .
drwxr-xr-x    5 0        0            4096 Apr  5 20:01 ..
drwxr-xr-x    2 59984    59984        4096 Apr  5 20:02 .ssh
drwxrwxr-x    2 59984    59984        4096 Apr  6 07:32 htdocs
drwxr-xr-x    2 0        0            4096 Apr  5 19:22 logs
sftp> pwd
Remote working directory: /
sftp> cd ..
sftp> ls -la
drwxr-xr-x    5 0        0            4096 Apr  5 20:01 .
drwxr-xr-x    5 0        0            4096 Apr  5 20:01 ..
drwxr-xr-x    2 59984    59984        4096 Apr  5 20:02 .ssh
drwxrwxr-x    2 59984    59984        4096 Apr  6 07:32 htdocs
drwxr-xr-x    2 0        0            4096 Apr  5 19:22 logs
sftp> pwd
Remote working directory: /
sftp> ls -la .ssh
drwxr-xr-x    2 59984    59984        4096 Apr  5 20:02 .
drwxr-xr-x    5 0        0            4096 Apr  5 20:01 ..
-r--------    1 59984    59984         601 Apr  5 20:02 authorized_keys
sftp> bye

Pay attention to the UIDs, 0 and 59984: The SFTP subsystem, running under chroot, doesn't have access to /etc/passwd from the user's environment.

I am convinced that this is the most important update to OpenSSH for at least the past five years. This has the opportunity to entirely eradicate authenticated FTP from the internet, just as it has already happened with Telnet.

Thanks a lot to the OpenSSH developers for making this happen!

Quick and dirty manual compile of OpenSSH on CentOS 5

Filed under: UNIX & Linux — Tags: , , — martin @ 7:58 am

(Update, February 27, 2009 – Please click here, for goodness’ sake: Packaging OpenSSH on CentOS)

I wanted to try the new chroot feature of OpenSSH (see the companion post) but didn’t want to invest in building an OpenSSH RPM. Here are my notes from how I did a quick replacement of the stock SSH packages by a hand-rolled installation:

# yum install gcc
# yum install openssl-devel
# yum install pam-devel
# wget http://ftp.bit.nl/mirror/openssh/openssh-5.0p1.tar.gz
# wget http://ftp.bit.nl/mirror/openssh/openssh-5.0p1.tar.gz.asc
# wget -O- http://ftp.bit.nl/mirror/openssh/DJM-GPG-KEY.asc | gpg --import
# gpg openssh-5.0p1.tar.gz.asc
gpg: Signature made Thu 03 Apr 2008 12:02:00 PM CEST using DSA key ID 86FF9C48
gpg: Good signature from "Damien Miller (Personal Key) <djm@****.org>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3981 992A 1523 ABA0 79DB  FC66 CE8E CB03 86FF 9C48
# tar zxvf openssh-5.0p1.tar.gz
# cd openssh-5.0p1
# ./configure --prefix=/usr/local --sysconfdir=/etc/openssh --with-md5-passwords --with-pam
# make
# make install
# cp /etc/ssh/* /etc/openssh/
# sed 's/^\(GSSAPI.*\)$/#\1/' < /etc/ssh/sshd_config > /etc/openssh/sshd_config
# sed 's/^ *\(GSSAPI.*\)$/#\1/' < /etc/ssh/ssh_config > /etc/openssh/ssh_config
# cp /etc/pam.d/sshd /etc/pam.d/openssh
# service sshd stop
# yum remove openssh
# ln -s openssh /etc/pam.d/sshd
# /usr/local/sbin/sshd
# echo "echo Starting ssh daemon." >> /etc/rc.local
# echo "/usr/local/sbin/sshd" >> /etc/rc.local

No: I’m not quite conviced that this should go anywhere beyond a test system. 😉 If you have a quick way for building proper OpenSSH replacement RPMs, you’re welcome to share it.

September 4, 2003

SSH-Hostkey generieren

Filed under: UNIX & Linux — Tags: — martin @ 10:18 pm

ssh-keygen -t rsa1 -f /etc/openssh/ssh_host_key -N ""
ssh-keygen -t dsa -f /etc/openssh/ssh_host_dsa_key -N ""
ssh-keygen -t rsa -f /etc/openssh/ssh_host_rsa_key -N ""

Ob ich mir das jemals auswendig merken kann?

Create a free website or blog at WordPress.com.