July 29, 2017

Debian /boot old kernel images

Filed under: Uncategorized, UNIX/Linux/BSD — Tags: , — martin @ 10:59 am

So I was looking at yet another failed apt-get upgrade because /boot was full.

After my initial whining on Twitter, I immediately received a hint towards /etc/apt/apt.conf.d/01autoremove-kernels, which gets generated from /etc/kernel/postinst.d/apt-auto-removal after the installation of new kernel images. The file contains a list of kernels that the package manager considers vital at this time. In theory, all kernels not covered by this list should be able to be autoremoved by running apt-get autoremove.

However it turns out that apt-get autoremove would not remove any kernels at all, at least not on this system. After a bit of peeking around on Stackexchange, it turns out that this still somewhat newish concept seems to be ridden by a few bugs, especially concerning kernels that are (Wrongfully? Rightfully? I just don’t know.) marked as manually-installed in the APT database: “Why doesn’t apt-get autoremove remove my old kernels?”

The solution, as suggested by an answer to the linked question, is to mark all kernel packages as autoinstalled before running apt-get autoremove:

apt-mark showmanual | 
 grep -E "^linux-([[:alpha:]]+-)+[[:digit:].]+-[^-]+(|-.+)$" | 
 xargs -n 1 apt-mark auto

I’m not an APT expert, but I’m posting this because the post-install hook that prevents the current kernel from being autoremoved makes the procedure appear “safe enough”. As always, reader discretion is advised. And there’s also the hope that it will get sorted out fully in the future.

September 27, 2012

Postfix: Rewrite the sender address of all mail

Filed under: Internet, UNIX/Linux/BSD — Tags: — martin @ 12:17 pm

OMG, I used to do mail for kings and queens. With properly crafted mail setups, meticulously built from the finest bits and bytes you can imagine.

But things have gone somewhat downhill from there. Now that I play the most general UNIX dude, I get this request far too often (talk about once every few weeks), and I’m not sure whether I have a proper solution for it or not. It goes like this:

“I have applications on the system, and I don’t know how, but they send mail, and I want all mail to be rewritten to a single from address. And please don’t try to tell me what you think good design looks like, but just get the damn job done.

So, the first approach was to add a pcre map in sender_canonical_maps with something like this, that matches every sender, as requested:

/.*/ godknowswhat@example.com

Directly from there, we tried to optimize away the sender_canonical file and its regex and came to this in main.cf:

sender_canonical_maps = static:godknowswhat@example.com

This actually leads to the same behaviour as it replaces any given sender with godknowswhat@example.com. Which, after a while, brought us to our first mail loop when Postfix had delivery problems and rewrote the empty bounce sender address (<>).

So now we are back to our pcre map for good, interestingly with just a single byte changed:

/.+/ godknowswhat@example.com

This rewrites all sender addresses to godknowswhat@example.com, but if a bounce appears, the sender address is not rewritten and the bounce can be delivered or at least double-bounce if it runs into additional failure.

June 26, 2012

A transparent firewall using OpenBSD

Filed under: UNIX/Linux/BSD — Tags: , — martin @ 1:46 pm

I wanted to enforce a different security policy for a given part of the home network, but without the introduction of separate subnets and static routes all over the place. So I started to experiment with a transparent firewall, on OpenBSD.

To accomplish this, what is needed first, is a bridge interface. A bridge interface is comprised of multiple (frequently 2) ethernet interfaces and transparently forwards traffic between the two interfaces. Very much like a 2-port ethernet switch would, or like a wireless bridge does.

If you have worked with bridges on Linux, configuration on OpenBSD works confusingly different than what you already know.

First, we have the configuration file /etc/hostname.bridge0, that specifies the interfaces that will be bridged:

# /etc/hostname.bridge0:
add vr0
add vr1
blocknonip vr1

If you are not concerned about non-IP traffic compromising the security of your bridging firewall, you may omit the option blocknonip here. The interface vr1 connects to the “restricted” part of the network here and for this example, I don’t want non-IP protocols such as NetBIOS to slip through.

Next, one of the two underlying interfaces is configured as if the bridge didn’t even exist. This will be the IP address of the bridge, here in /etc/hostname.vr0:

# /etc/hostname.vr0:
inet6 2001:db0:1:2::3 64

And the additional interface for the bridge is only brought “up” in /etc/hostname.vr1:

# /etc/hostname.vr1:

After a reboot, the bridge will be forwarding packets, without the need to enable IP forwarding and without enabling the firewall.

Now I can start adding firewall rules in /etc/pf.conf:

# /etc/pf.conf:

# The interface on the open network

# The interface on the restricted network

# I don't believe in dropping packets
set block-policy return

# Leave the loopback interface unfiltered
set skip on lo

# Pass out what's already inside the firewall
pass out on vr0 all
pass out on vr1 all

# Pass in everything coming from the open network
pass in on $PUBLIC_IF

# Reject everything coming from the restricted network
block in on $RESTRICT_IF

# Restricted network may talk to the world
pass in on $RESTRICT_IF inet
pass in on $RESTRICT_IF inet6

# But not to the open network
block in on $RESTRICT_IF from any to
block in on $RESTRICT_IF from any to 2001:db8:1:2::/64

# Allow access to the site's DNS server 
pass in on $RESTRICT_IF proto {tcp,udp} from any to port 53

# Allow access to this single SSH service
pass in on $RESTRICT_IF proto tcp from any to port 22
pass in on $RESTRICT_IF proto tcp from any to 2001:db8:1:2::b port 22

# Pass ICMP in all directions
pass proto {icmp, icmp6}

And that’s about it. What I have here is really just a rough example of what can be done. Bridging over 3 or more interfaces is possible, if you enjoy the complexity.

With basic firewalling out of the way, it is also possible to filter by MAC address. For this, we need to tag some traffic, e.g. a “trusted client”, directly on the bridge:

ifconfig bridge0 rule pass in on vr1 src 00:1c:c6:8b:ae:3b tag TRUSTEDCLIENT

(Append this, excluding the leading “ifconfig bridge0”, to /etc/hostname.bridge0 to make it permanent.)

In /etc/pf.conf, we can then filter by tag:

# Allow all traffic from trusted MAC addresses

Or, more specific:

pass in on $RESTRICT_IF proto tcp all port 22 tagged TRUSTEDCLIENT

Combining MAC filters with other filter criteria is of course possible. Please make up your own mind about whether and how you want to use this.

A final word on performance: With both interfaces working in promiscuous mode, stress on the firewall’s CPU can be expected to be rather high. This 100 Mbps bridge with a 500 MHz AMD Geode CPU will reach a CPU load of 70% while moving no more than 85 Mbps on a single TCP connection.

January 7, 2012

Securitygewixe reloaded

Filed under: Security, UNIX/Linux/BSD — Tags: , — martin @ 11:13 am

Die WordPress-Referrer haben mir einen alten Beitrag über OpenBSD-Security nach oben gespült, in dem Linus Torvalds in seiner unnachahmlichen Art damit zitiert wird, daß er die OpenBSD-Entwickler für Securitywixer hält. Leider muß ich ihm zwischenzeitlich zustimmen.

Mich persönlich hat OpenBSD vor 2 Jahren beim Übergang von 4.6 zu 4.7 mit einer weitreichenden Änderung an der Firewallkonfiguration abgehängt, bei der Rewrite- und Filterregeln vereinigt wurden. Theoretisch betrachtet eine Vereinfachung. Praktisch leider mit dem Haken, daß es keinen Migrationspfad gab, um Regeln Zug um Zug umzustellen, denn mit Einführung der neuen Regeltypen waren die alten Regeln nicht mehr verfügbar. Die einzige Möglichkeit war, das gewachsene Regelwerk im Blindflug komplett umzustellen.

Die OpenBSD-Community zuckte mit den Schultern und verwies auf mein Testsystem, auf dem ich die Regeln ja testen kann. Was den Blindflug bei der Inbetriebnahme nicht minderte. Schade. Seitdem verzichte ich bei Installationen, die auch ohne Kopfschmerzen mal ein oder zwei Updates überstehen sollen, auf OpenBSD und greife lieber zu Debian Linux.

November 24, 2011

Dovecot LDA behind Procmail

Filed under: UNIX/Linux/BSD — Tags: , — martin @ 12:17 pm

I have millions of archived mails in my ~/Maildir/ and wanted to change local delivery from a fairly extensive Procmail configuration to the Dovecot LDA, in order to keep the Maildir properly indexed.

In order to do this, I globally enabled LDA and Sieve in the Dovecot configuration, migrated my ~/.procmailrc into a Sieve ruleset (manually, because the converters don’t cut it) and then replaced my trusty old ~/.procmailrc with:

:0 w

This way, the global default of using Procmail remains unharmed, maintaining compatibility with other users. On delivery, all messages to me are redirected into the Dovecot delivery system, including Sieve filters.

July 24, 2010

How to get rid of the plus signs in Bonnie++ output

Filed under: UNIX/Linux/BSD — Tags: — martin @ 9:58 pm

I’m confident that many of you have seen this countless times when using the popular “standard” harddisk benchmarking tool, Bonnie++:

# bonnie++ -u root -d /mnt/1
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
grml             8G 51447  98 129399  39 56253  16 43505  97 136526  16 985.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

The manpage explains this as follows:

If a test completes in less than 500ms then the output will be displayed as “++++”. This is because such a test result can’t be calculated accurately due to rounding errors and I would rather display no result than a wrong result.

You will most likely see this for the lower “Create” section of Bonnie++. While the tests from the upper “Input/Output” section always make an educated guess and use twice the size of RAM, the “Create” tests have no way of guessing the proper number of files. Look for the word “files” near the left: Bonnie++ always defaults to this 16 x 1024 files for testing. Indeed, munging 16000 files in less than half a second sounds like no big task for a fast system.

Bonnie++ takes an option that instructs it to use more files for the “Create” tests, e.g. -n 128 tells it to test with 128 x 1024 files. In this second run, I’ll also disable the “Input/Output” tests by setting -s0:

# bonnie++ -u root -d /mnt/1 -s 0 -n 128
Version 1.03d       ------Sequential Create------ --------Random Create--------
grml                -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                128 46336  84 +++++ +++ 31277  50 32574  65 +++++ +++ 12803  23

We now get values for file creation and deletion, but reads are still excessively fast. Since Bonnie++ discarded the measurement because it finished within less than 500 ms, it may be possible to estimate that the system is capable of performing at least (128 x 1024 x 2 = 262144) read operations per second.

Further increasing the number may not be possible due to lack of inodes on your filesystem. Also, such an extremely high number of files and file operations may decrease overall file system performance.

The system used for testing was a dual AMD Opteron 285 with a hardware RAID 0 of 3 UW320 disks @ 15k RPM.

December 11, 2004

SSH um die Ecke(n) bringen

Filed under: Notizen, Sicherheit, UNIX/Linux/BSD — martin @ 11:41 am

Die Ausgangslage

Auf einer bestimmten Baustelle muß ich, wenn ich an meine Mails kommen will, über zwei DMZ-Gateways (eins von der Baustelle und unser eigenes) sowie unseren im Rechenzentrum stehenden Webserver gehen, um schließlich auf dem Server im Büro zu landen. Als ich noch mit dem textbasierten Mutt gearbeitet habe, war das eine ganz einfache Sache, denn ich konnte mich immer der Nase nach per SSH über ein System nach dem anderen durchhangeln und dann auf der Kommandozeile des Zielsystems mutt starten:

martin@workstation:~$ cat bin/bring-mich-office.sh
ssh -A -t dmzbox.baustelle.invalid \
 ssh -A -t webbox.example.com \
 ssh -A -t dmzbox.example.com \
 ssh officebox.example.com
martin@workstation:~$ bin/bring-mich-office.sh
martin@officebox:~$ mutt -y

Die Frickelvariante

Seit ich unter die Mausschubser gegangen bin, gestaltet sich das ganze deutlich schwieriger, denn über diese ganze Kette müssen jetzt die Ports 143 und 25 für IMAP und SMTP getunnelt werden. Dies soll geschehen, ohne daß diese Ports auf den in der Mitte der Kette befindlichen Systemen ansprechbar sind. Wenn auf den Zwischenstationen jedermann auf die weitergeleiteten Ports zugreifen könnte, hätte ich riesige unerwünschte Löcher in Firewalls gebohrt, und das will sicherlich niemand.

Um diesen Tunnel vollautomatisch aufbauen zu können, habe ich mich dann eine ganze Weile mit einer wüsten Konstruktion herumgeschlagen, bei der zunächst über alle beteiligten Funkhäuser ein “verlegter” SSH-Port durchgetunnelt wurde. In einer zweiten SSH-Sitzung habe ich über diesen dann meine beiden Applikationsports getunnelt:

# Erste Verbindung
ssh -A -L 10022:localhost:10022 dmzbox.baustelle.invalid \
 ssh -A -L 10022:localhost:10022 webbox.examle.com \
 ssh -A -L 10022:localhost:10022 dmzbox.example.com \
 ssh -L 10022:localhost:22 officebox.example.com sleep 20 &
# Gedenkpause
sleep 10
# Zweite Verbindung
ssh -t -C -p 10022 -L 10025:localhost:25 -L 10143:localhost:143

Ich denke, ich muß nicht näher erläutern, warum das nicht nur furchtbar aussieht, sondern auch ganz schön bescheiden funktioniert hat. Bei fünf ineinandergestöpselten Port-Weiterleitungen ist die Wahrscheinlichkeit, daß alles klappt, wirklich nicht besonders hoch.

December 3, 2003

Was ist UNIX?

Filed under: UNIX/Linux/BSD — martin @ 10:12 pm

Wer kennt sie nicht, die modischen Diskussionen darüber, ob Mac OS und Linux nun UNIX-Betriebssysteme sind, oder nicht? Dabei ist die Sache doch in Wirklichkeit ganz einfach und nachzulesen bei der Open Group, die der Markeninhaber von “UNIX” ist.

UNIX ist eine Familie von Betriebssystemen, die in ihrer Funktionalität der Single UNIX Specification (IEEE 1003.1, früher auch bekannt unter dem Namen POSIX) nicht nur folgen, sondern auch gemäß dieser Spezifikation zertifiziert sind, und die zur Benutzung der Marke “UNIX” lizensiert wurden. Dazu gehören z.B. Solaris von Sun, AIX vom IBM, HP-UX und Tru64 von Compaq, IRIX von Silicon Graphics und auch UnixWare von SCO. Linux, die BSD-Familie und MacOS dagegen gehören nicht zu diesen “branded UNIX”-Systemen, ganz unabhängig davon, wie energisch viele “Fans” und vermeintliche Experten das Gegenteil betonen.

October 14, 2003

In-place editing, ein für alle mal.

Filed under: Internet, Notizen, perl, UNIX/Linux/BSD — Tags: — martin @ 6:17 pm

So bearbeitet man mit Perl als sed-Ersatz Dateien und läßt dabei ein Backup zurück:

perl -i.bak -pe 's/lull/lall/g' file1 file2 file3 ... fileN

Die genannten Dateien werden so geändert, und die alte Version bleibt mit der Extension .bak jeweils zurück.

Blog at WordPress.com.