My name is Philipp C. Heckel and I write about nerdy things.
This site moved here recently from!

Posts Categorized / Linux

  • Aug 05 / 2018
  • 0
Linux, Security

Using Let’s Encrypt for internal servers

Let’s Encrypt is a revolutionary new certificate authority that provides free certificates in a completely automated process. These certificates are issued via the ACME protocol. Over the last 2 years or so, the Internet has widely adopted Let’s Encrypt — over 50% of the web’s SSL/TLS certificates are now issued by Let’s Encrypt.

But while there are many tools to automatically renew certificates for publicly available webservers (certbot, simp_le, I wrote about how to do that 3 years back), it’s hard to find any useful information about how to issue certificates for internal non Internet facing servers and/or devices with Let’s Encrypt.

This blog posts describes how to issue Let’s Encrypt certificates for internal servers. At Datto, we issued a certificate for each of our 65,000+ BCDR appliances using this exact mechanism.

Continue Reading

  • Mar 18 / 2018
  • 0
Linux, Virtualization

USB disk causes blinking cursor at boot; how to “fix” the MBR bootstrap code

Have you ever rebooted your computer only to see a black screen with a blinking cursor? If you have a USB drive attached, chances are the blinking cursor is caused by invalid bootstrap code in the Master Boot Record (MBR) on that drive which has caused the normal boot execution to stop without returning control to the BIOS. If you have physical access to the machine, simply remove the USB drive and/or change the boot order to pick the OS disk first.

If you have no physical access, things are a bit more tricky: This exact thing happened to me at work the other day. Unfortunately, it didn’t happen to my computer, but to a few dozen of our customer backup appliances during their scheduled upgrade/reboot. Now, while dozens out of over 60k isn’t that much, our customers rely on these devices, so it’s not acceptable to have them not boot properly.

In this short post, I’ll demonstrate how to reproduce the blinking cursor problem, and how to “fix” the MBR to ensure the computer still boots, regardless of the boot order.

Continue Reading

  • May 28 / 2017
  • 6

Creating a BIOS/GPT and UEFI/GPT Grub-bootable Linux system

Good old Master Boot Record (MBR) unfortunately cannot address anything beyond 2TB, so partitioning large disks and making them bootable is impossible using MBR. The GUID Partition Table (GPT) solves this problem: It supports disks up to 16EB. However, installing grub does not work without a special BIOS boot partition. If you also want to support booting the same system via UEFI, another partition, the EFI System Partition (ESP), is necessary.

This should post shows you how to partition a disk with GPT and make a bootable Linux system via BIOS/Legacy and UEFI.

Continue Reading

  • Jan 08 / 2017
  • 19
Administration, Linux

How-To: Using ZFS Encryption at Rest in OpenZFS (ZFS on Linux, ZFS on FreeBSD, …)

An upcoming feature of OpenZFS (and ZFS on Linux, ZFS on FreeBSD, …) is At-Rest Encryption, a feature that allows you to securely encrypt your ZFS file systems and volumes without having to provide an extra layer of devmappers and such. To give you a brief overview of what the feature can do, I thought I’d write a short post about it.

The current ZFS encryption implementation is not (yet) merged into the upstream repository (as of January 2017). There is a pretty big pull request which is still being reviewed, but because the feature is so incredibly cool (and because my colleague Tom Caputi at Datto developed it), I thought a sneak preview is absolutely necessary.

Continue Reading

  • Jan 01 / 2017
  • 0
Administration, Linux

zfsu: ZFS utils for offsite backup, retention and maintaining a slow mirror

My laptop runs ZFS as its root file system (see this blog post) — meaning that I can snapshot my root file system and I can send it to another machine as a backup very easily. Unfortunately, while ZFS provides the raw functionality, there is no great tool to manage offsite backups and retention. To ease this pain, I wrote/forked and packaged a few helper scripts which I called zfsu, a collection of ZFS utilities.

It consists of the following tools: zfsu tx (aka zfstx) maintains a mirror of a ZFS pool over the network. zfsu ret (aka zfsret) is a simple script to apply local retention (destroy snapshots) of a file system and its snapshots. zfsu res (aka zfsres) is a script to resilver a slow mirror, e.g. a HDD disk if mirrored with a SSD.

Continue Reading

  • Dec 31 / 2016
  • 9

How-To: Move your existing Linux install to ZFS on Root

Ever since I joined Datto two years ago, ZFS has been part of my work every day. And every day, I am amazed how great it is. So naturally, I wanted to move my existing Linux Mint 18 installation to boot off of ZFS. Why, you may wonder? Well that’s easy. Because now I can snapshot my root file system, I can roll back if I need to, and I can restore individual files in a heartbeat.

It took a bit of fiddling in the beginning, but once you know how it works, it’s a piece of cake. This short post shows you how to move your existing Linux installation to ZFS on root (preferably Ubuntu 16.04+ based, may work for others).

Continue Reading

  • Dec 04 / 2015
  • 4
Administration, Code Snippets, Linux, Scripting, Security

Snippet 0x0D: Let’s Encrypt – 5 min guide to set up cronjob based certificate renewal

Let’s Encrypt was officially released to the open public today. That means the Internet can finally get free, trusted SSL/TLS certificates. This quick guide shows how to set up Let’s Encrypt with auto-renewal through a cronjob — using the simp_le client, an alternative client developed by one of the same authors who develop the official client.

Continue Reading

  • May 04 / 2015
  • 3
Linux, Programming, Security

OpenSSH ‘AuthorizedKeysCommand’ with ‘fingerprint’ argument (Patch for OpenSSH 6.6p1)

Many of us developers or system administrators use OpenSSH’s public key authentication (aka password-less login) on a daily basis. The mechanism works based on public key cryptography: By adding a RSA/DSA public key to the authorized_keys file, the user with the matching private key can login without a password. The mechanism works great for a couple of hundred, thousands and even 100k thousand users (tested, login takes ~2sec).

But what if there are more keypairs, say, a million users, or a more flexible approach is desired? Maybe with an LDAP or a database backend? Think of GitHub and how they do their ssh ... login! This blog post shows you how to do that by patching OpenSSH’s AuthorizedKeysCommand option to support an additional fingerprint argument.

Continue Reading