How To Verify and Change the Certbot Authenticator
I have never acquired a paid SSL certificate in my life back when it was still SSL and even during my various jobs I never had a reason to obtain a TLS certificate with money, and I may never have to. Prior to the Snowden/Wikileaks events the movement to encrypt the entire internet was already well underway by people who grasped what we were up against and I had heard rumblings about completely free TLS certificates. I can’t recall the exact order in history to be honest as I was very far removed from all of that but one of the results of all of this was the creation of Let’s Encrypt, the certbot project and the ACME protocol.
Fast forward to today and we’re living & breathing Let’s Encrypt and certbot at work. I recently had to consolidate multiple instances of the same project which were setup many years apart to use the exact same configuration just to retain a little bit of our sanity during daily operations. Part of that was that the certbot configuration was different across these environments and I found myself struggling to remember which environment used which certbot authenticator to obtain it’s TLS certificate. I used to like the webroot authenticator and having certbot write the ACME verification challenge into a directory in the filesystem from which the rest of the website was served from as I heavily rely on hugo for everything website related. It was just another static file served by nginx.
I have since come around to using the standalone authenticator though alongside the --http01_port=xxxx flag to have certbot serve the ACME challenge out of RAM in order to eliminate a whole swath of ACL related issues in the future.
In order to check which authenticator is currently used, you need to read the certbot config file from the hostname you’re after in /etc/letsencrypt/renewal/, which should look something like this:
# renew_before_expiry = 30 days
version = 1.12.0
archive_dir =
cert =
privkey =
chain =
fullchain =
# Options used in the renewal process
[renewalparams]
account =
http01_port = 9090
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
Key things to look out for here is the authenticator value and the related metadata given your authenticator of choice.
Changing Authenticators
Given that you have now looked at your authenticator in horror on some old host and want to change it I have found this great post.
The recommended way to change authenticators is to re-run the certbot command to issue a certificate for the exact same hostname with but including the --force-renewal flag as well. For example
$ certbot certonly --force-renew --standalone -d <your-hostname> --non-interactive --agree-tos --email <your-email-to-be-notified-about-stuff> --http01_port=9090
07.01.2024
E-Mail Server Update
Quite a while ago I wrote about my use of the OSX Server application as I was one of the few remaining people on the internet dumb enough to run my own E-Mail server. Operating this service mostly for myself has taught me a lot which allows me to confidently approach certain problems while helping to develop & run Guardian’s many backend services today for our customers. The broken Mac mini resides on my desk to this day as a reminder of where this journey started.
After shopping around many E-Mail server options, I ultimately landed on Mailcow and I could not be happier about it! Using Docker to solve the problem of making the various components talk to each other was quite intimidating to me at first, as I was afraid about the networking aspect of it all. It turns out though that the maintainers of the project have done a phenomenal job of getting all the various pieces to talk to each other and effortlessly upgrade to newer versions. I suspect the project’s use of Docker is how it was imagined to be used by it’s creators and Mailcow’s setup has become my personal measuring stick whenever I see others trying to leverage Docker as a growth-hacking-problem-abstraction-layer.
Out with the old
As part of my yearly tradition of tinkering with my own hardware over the holiday season I decided that this year was going to be the year to upgrade the servers hardware. I settled on jumping from an Intel i5 6600 to an AMD Ryzen 5 5600G, alongside a new motherboard, new faster RAM etc…
As part of the upgrade I had also decided to jump from running everything bare metal with the desktop variant of Ubuntu 22.04 LTS to Proxmox VE 8 as the primary operating system in order to leverage ZFS and mirrored boot drives as an operational safety net. I have been a ZFS advocate for a few years now and have been fortunate enough to have gained a lot of real world experience with this exact setup. ZFS is the real deal and I will go out of my way to recommend it whenever applicable!
The Mailcow server itself is running the server variant of Ubuntu 22.04 LTS inside a VM managed by Proxmox. All in all things are bit more folded into itself as I added yet another level of abstraction but the Linux Kernel virtualisation layer (KVM) is simply amazing, so none of that ended up giving me any trouble at all!
As part of the takedown preparation of the old server I ran a backup of the E-Mail inboxes itself with imapsync (link) followed by a complete backup with Mailcow’s backup-and-restore (link) script. The output of that script are a bunch of .tar.gz files as well as the Mailcow configuration file which I rsync’d to the new Mailcow VM. The same backup-and-restore script was run in the VM to restore a one-to-one copy of the old on the new server and I was back up and running within a bit over an hour end-to-end.
No E-Mail client that interacts with my server appears to have noticed anything about the move, which has done nothing but push my confidence in the Mailcow toolchain even further. If you think about running your own E-Mail server for whatever reason, Mailcow would be my recommendation 10/10 times!
One key thing that I am still missing in Mailcow is the XAPPLEPUSHSERVICE which was a Apple created custom Dovecot extension in OSX Server, to enable you to send push notifications about new E-Mail to iOS’s Mail.app. So far the only other provider being able to do this has been Fastmail to my knowledge, but I am not entirely sure whether they support it anymore as I have not been able to get it to work recently with my Fastmail account.
31.12.2023
Being A Good Citizen on the Internet
I must admit that I am severely impressed. Whenever I think of Backblaze at this point I think of a size-able company, maintaining EXABYTES (that number makes no sense whatsoever) of data from individuals as well as massive corporations. They run two physical locations with lots of staff and develop their own hardware & software to make all of this happen while being entirely independent of the few big “cloud” providers that have cornered the market.
This mentality is something I have always looked up to and am actively pursuing with my work at Guardian, which is now part of DNSFilter. If you want it to be done correctly you either resort to buying the most expensive option, which you probably can’t afford and is not guaranteed to solve your problem, or do it yourself. For a lot of problems those are the two options if you want to avoid mediocre, middle-of-the-road crap. Backblaze has mostly chosen to do the latter, which is something that I can really respect.
Given their responsibilities and reputation I was quite impressed to see a seemingly real human being reach out to me via E-Mail, kindly asking me to update an older link on my tiny website to their B2 product. I was going to write back telling them that I would do that but that they should ensure that they have forwards setup for the old links, but as I tested that I quickly noticed that they were already way ahead of me.
This is how you do PR work correctly for a company of any size! No ego & no weird lawyerly phrasing, just a regular message from somebody reaching out kindly asking to avoid a redirect for anybody clicking the link on my website. Backblaze has once again proven that the organization as a whole has not fallen into the
“big corporation that is required to increase shareholder value by any means”
trap and is aware that they’re just a citizen of the internet like any of us. And they’re making an effort trying to be a good internet citizen, too! I am still dearly missing a few (basic-ish) features which I would expect from their product to further integrate it into Guardian itself, but this interaction has once again proven that Backblaze is a great company to be a customer of and has really re-assured my trust in them with regards to everything that they do. I would love to one day be able to tour one of their facilities, given that I have followed their lead with building backup servers out of cheap components and being successful at it.
None of this was related to their technical strategy, past decisions or current offering, but solely on a normal human interaction.
Funny how effective a little bit of politeness & self-awareness is in this anonymized, digital world…
06.08.2023
A Blog Series: ¿What does Tzeejay Listen To?
The majority of my working time is accompanied by music. Usually long running live electronic sets but I throw other stuff in there as well on the regular. The majority of it plays through Youtube, simply because Apple Music makes my blood boil in anger and Spotify is not an alternative to me for many reasons. Them seeking to completely ruin Podcasts & their stance on tracking and privacy are just the start of it all. I pay for Youtube Premium (yeah I know it’s rich reading this given the last sentence, but there is no alternative to Youtube that is actually being used or has relevant anything on it) since I watch a lot of car related videos at night so being able to enjoy some good music without any ads is the most obvious choice to me, even though I am not crazy about streaming everything all the time. And neither am all in on anything Google owned apart from Youtube.
This series is an attempt to give myself something simple to blog about and get into a habit of throwing things that are a little longer onto my own website and actually use it. So far it had been a little sad on here.
Expect short little posts describing the music I am linking to and why you may like it as well.
Maz Live @ Warung Tour Porto Alegre
24.11.2022
¿What does Tzeejay Listen To? Maz Live @ Warung Tour Puerto Alegre
I had actually never heard of this artist before stumbling onto this video but from the first beat it kinda took me for a ride and was perfect to work to. 2 hours might be a little long to some, but you can skip to almost any section in the video and get into the rhythm of the beats right away.
There is also an interesting mix of Afro House in there at the end with (I believe) Portuguese lyrics. All around great set
Youtube link
24.11.2022
WWDC 2022
If you have been following me on Twitter for a little longer you are aware that I am quite outspoken about things like App Store policies, Apple’s seemingly great service revenue and their interaction and treatment of developers. If things are right they should be praised and if things are wrong they should be critiqued. Nothing good is created or can exist in a vacuum without feedback.
Over the past 3 years relations between developers and Apple have gone from indifferent at best to outright hostile. WWDC always moves the general sentiment to being more favorable and friendly, this year however has built on top of the very good remote WWDC we got to enjoy last year. I have not participated in digital lounges or other events due to various reasons but have heard from many that they have really enjoyed it.
With various people that I know of, listen to Podcasts to or know personally being in person at WWDC in Cupertino has really made me wish to be able to be there as well. It filled me with an enormous amount of envy. Envy to be there and experience the energy in the “room”, envy to make new friends, envy to share ideas, envy to learn & envy to see existing friends.
The cool kids got to fly to Cupertino. I wish I was part of the group of cool kids.
Thank you Apple for a great WWDC 2022. It feels like you have figured out what you want this to be going forward and I am excited to keep being involved in developing software for these platforms.
10.06.2022