-ENOTTY (Posts about english)

Apple knows better (how big it is)

Tomasz Torcz — Fri, 16 Jan 2026 10:38:18 GMT

Last year, I've spent few months exchanging emails and packet dumps with Zoom's support. They provide videoconferencing software which gave me connectivity problems . Specifically, Zoom client, on my work Macbook, when connected to my home network, failed. To make reporting harded, Zoom's webpage did not open in Chrome on that laptop neither.

Perplexingly, my private Fedora laptop, on the same network, had no problems whatsoever. I'll spare the details of weeks-long investigation (no, it wasn't DNS' fault; nor Cloudflare's).

The problem source…

MacOS X DHCP client ignores Maximum Transfer Unit (option 26) from DHCP server!

I couldn't believe it, but there are tons of similar reports over the net. Apparently, Apple deems MTU information not trustworthy. Well, thank you 🍏, there went weeks of my time.

Why do I even have non-standard MTU in my network?: Since my ISP was acquired, I had to deal with blast-from-the-past networking. Plain, reliable Ethernet connection was replaced with PPPoE, lowering the MTU to 1492 bytes, bringing distaste and headaches.
Why other webpages and software continued working?: Most communications use TCP/IP protocol, which knows how to deal with decreased MTU. Modern pages (and Zoom) try to use QUIC protocol. It works over UDP. UDP has no mechanisms of Path MTU discovery and too big packets are just dropped. PMPTU for QUIC is still at a draft stage.
Why it started to happening recently?: Due to other circuimstances, I had disabled IPv6 connectivity for my work laptop around summer last year. I didn't noticed at the time, but before that, Zoom worked fine over IPv6. MTU information from RA is good enough for Apple, whereas from DHCP is a no-no.

Kudos to Zoom Support, for spotting the MTU problem in the end.

030/100 of #100DaysToOffload

I Voted, F43 edition

Tomasz Torcz — Tue, 06 Jan 2026 16:44:33 GMT

I've cast my votes in Fedora Engineering Steering Comittee. The voting closes tomorrow.

As usual, I've read interview with the candidates, and then decided on my preferences. Red Hat employees get a minus, new faces get plus. There are exceptions, it's not a hard rule!

That's one of the way I contribute to Fedora. Sometimes I blog about elections. I've also been a Fedora packages for almost 18 years!

028/100 of #100DaysToOffload

Fedora 42→43, more eventful than usual

Tomasz Torcz — Fri, 12 Dec 2025 10:39:03 GMT

I've found time for dist-upgrade of my home server, finally. As usual, there was one thing needing manual intervention: PostgreSQL update. But this time it was more complicated.

Between Fedora 42 and 43, PostgreSQL jumped from v16 to v18. And postgresql-setup --upgrade handles adjacent versions upgrades only. Fortunately, Fedora ships other version-suffixed packages for this database.

It is possible (and needed!) to use postgresql-server17 and postgresql17-upgrade packages as an intermediate step in the upgrade. Commands are documented in bz#2411778#c1. It should be included in F43 Common Bugs list, but it isn't. (And the list itself was moved from Wiki into Discourse…)

Note to self: the upgrade always fails with my customised postgresql.conf. Remember to plant the default config for the duration of postgresql upgrade.

026/100 of #100DaysToOffload

Backups with btrbk

Tomasz Torcz — Sun, 02 Nov 2025 19:52:18 GMT

Storage setup of my home server is btrfs raid1 over two, dm-crypt'ed 16TB HDDs cached with bcache on NVMe. It works fine, however for PostgreSQL database and my homedir I prefer full NVMe speed.

Therefore I've put those two directories on (dm-crypt'ed) btrfs subvolumes directly on NVMe. Thanks to DUP profile there's a protection against bitrot. But it's still a single device which may just die. Regardless of full backups, I was doing daily rsync into main drives, but there's faster and more capable way.

Enter btrbk, which operates on btrfs subvolumes. It uses btrfs' native send capability to copy the subvolume between filesystems effectively.

Additionally, it's very easy to keep some number of historic subvolume snapshots. They utilize copy-on-write, minimizing space usage. This let me recover files quickly or compare filesystem's state over last few days.

The config it bit tricky, that's why I'm posting this. My full backup config below, divided in three sections for explanations.

timestamp_format        long
snapshot_preserve       14d
snapshot_preserve_min   2d      # defaults to 'all'

The source definition. preserve option combination is needed to have daily snapshots kept for last two weeks and have older snapshots removed.

target_preserve         7d
target_preserve_min     latest  # defaults to 'all'

What to do with subvolumes copies at the target directory. Above combination of options keeps last seven days of snapshots copies.

send_compressed_data    yes

volume /run/btrbk-work
        target /home/poligon/backs/btrbk_snaps

        subvolume home_zdzichu

        subvolume var_lib_pgsql

Job definition. /run/btrbk-work is a directory where I temporarily mount NVMe drive root volume with subvolumes beneath. /home/poligon/backs/btrbk_snaps is the directory on my main (raid1) pool where subvolume copies are stored. And the last two lines are specific subvolumes to copy.

That works for me. btrbk is run by cron.daily/ from a short script ensuring everything is mounted where it should be.

020/100 of #100DaysToOffload

Actalis EAB ACME works – it was base64url issue

Tomasz Torcz — Wed, 20 Aug 2025 07:55:32 GMT

As an addendum to yesterday's note: I've got Actalis issuer to work with cert-manager.

cert-manager expects some credentials to be in so-called base64url encoding, which is stated in a note in the documentation. Fix was easy, I had to remove = from provided HMAC Keys. The docs have sed invocation to use.

002/100 of #100DaysToOffload

Leaving BuyPass as an ACME provider

Tomasz Torcz — Tue, 19 Aug 2025 18:26:33 GMT

For a long time I've been using BuyPass as TLS certificates provider for ACME. Unfortunately they decided to disengage from this area of services.

There are quite a few ACME providers. Some even look like they could replace BuyPass, which had two strong traits: it is based in Europe and was providing certificates valid for half a year. It looked like Actalis would be a good replacement. They're from Italy and have 1 year certificates, but available in paid plans only.

After some tinkering with cert-manager I was unable to make it work. Some cryptic, discouraging messages like "ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" and "failed to verify ACME account" err="failed to decode external account binding key data: illegal base64 data at input byte 43" made me look further.

Next shot, ZeroSSL worked straight away. Worth noting – official cert-manager documentation has a tutorial on using ZeroSSL. There are some limitation, but it's fine to me. There's nothing more to write, it just works.

For private services (meant to be accessed only from my devices), I'm using FreeIPA as an ACME provider, of course.

Why not Let's Encrypt? Only because it is not hipster enough.

This is post 001/100 of 100DaysToOffload challenge. I intend to write short posts about nothing in particular, just collected thoughts. Language will vary: Polish, English, maybe Arabic if I get back to learning it.

Fixing FreeIPA ACME

Tomasz Torcz — Thu, 08 Feb 2024 20:17:56 GMT

I've managed to fix one of the recent problems with my FreeIPA. To be more specific – after few months breakage, ACME issuer is working again for me.

Solution was dead simple after I've found the relevant line in the logs:

2024-02-07 21:59:24 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-3]
SEVERE: Servlet.service() for servlet [ACME] in context with path [/acme] threw exception
org.jboss.resteasy.spi.UnhandledException: com.netscape.certsrv.base.UnauthorizedException:
Authorization failed: Authorization failed on resource: group=Certificate Manager Agents, operation: {1}

I've fired up an LDAP client (I'm using JXplorer), went to /ipaca/groups/Certificate Manager Agents and added two records. Type uniqueMember with value uid=acme-kaitain.pipebreaker.pl,ou=people,o=ipaca, plus another one for the second replica. FreeIPA started to issue certificates again, although there are still some strange trackbacks in the logs.

Why the group membership dissapeared? I don't know.

Woes with FreeIPA and SIDs

Tomasz Torcz — Sat, 20 Jan 2024 17:23:22 GMT

As cool as FreeIPA is – I'm using it for over 10 years at home – sometimes I fell it breaks just by being looked at.

TLDR: If you have accounts or groups with UID or GID not falling in any defined range ID ranges, fix that by any of: 1) changing the UID/GID; 2) extending defined ID range; 3) creating a new ID range covering those UIDs/GIDs.

Then generate new SIDs with ipa config-mod --enable-sid --add-sids.

Few months ago my FreeIPA installation stopped allowing users to login. It was mostly a nuisance for me, nothing critical:

on most important computers I have local accounts
services authenticating with plain LDAP continued to work
so what didn't work? Login into FreeIPA's web interface. But admin accounted worked, so…

When I eventually started to debug, I found following logs in krb5kdc.log:

krb5kdc[3123](info): AS_REQ : handle_authdata (2)
krb5kdc[3123](info): AS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19),
aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26),
camellia128-cts-cmac(25)}) 192.168.x.x: HANDLE_AUTHDATA: zdzichu@PIPEBREAKER.PL for krbtgt/PIPEBREAKER.PL@PIPEBREAKER.PL, No such file or directory

Some searching the web later, I've concluded No such file or directory refers to SID, a Security Identifier. SIDs are some Microsoft's Active Directory properties. I see no reason to have them on Linux-only FreeIPA installs.

The Upgrade section in the documentation is wrong:

The upgrade does not turn on the feature. If the admin wants to enable SID generation, he needs to update the packages and run the new command

It's not true, admin desires don't matter. As Alexander Bokovoy wrote on the mailing list:

FreeIPA generates SIDs by default since FreeIPA 4.9.8. It is configured to do so on new installations even when integration with AD is not considered, due to tightened requirements to process constrained delegation in Kerberos.

So SIDs are really required in FreeIPA now. I don't want them, but whatever. There is a command which is supposed to generate missing data:

ipa config-mod --enable-sid --add-sids

It finished without errors, but login still didn't work.

More searching. The commands fires up a job (in LDAP server?) so let look into dirsvr's errors.log:

[12/Jan/2024:14:07:51.524937136 +0100] - ERR - sidgen_task_thread - [file ipa_sidgen_task.c, line 194]: Sidgen task starts ...
[12/Jan/2024:14:07:51.566941696 +0100] - ERR - find_sid_for_ldap_entry - [file ipa_sidgen_common.c, line 521]: Cannot convert Posix ID [135] into an unused SID.
[12/Jan/2024:14:07:51.585948214 +0100] - ERR - do_work - [file ipa_sidgen_task.c, line 154]: Cannot add SID to existing entry.
[12/Jan/2024:14:07:51.597088076 +0100] - ERR - sidgen_task_thread - [file ipa_sidgen_task.c, line 199]: Sidgen task finished [32].

OK, we're getting somewhere. I had some an obsolete group defined with GID 135. Let's remove it and re-run the command.

[12/Jan/2024:14:24:29.879953617 +0100] - ERR - find_sid_for_ldap_entry - [file ipa_sidgen_common.c, line 521]: Cannot convert Posix ID [1037] into an unused SID

I know this UID, this is one of the users I've migrated to FreeIPA. Obviously I cannot remove it.

Turns out, FreeIPA has something called ID ranges. If you have accounts or groups with UID or GID not falling in any defined range, sidgen script just fails.

I had to go to IPA Server -> ID Ranges -> Add in web console and make changes. I did not want to mess with already defined ranges, so I created a new one. An “unix” range. Base 1000, size 1000, had to guess some numbers for RID ranges before it was accepted.

Not that I'm so smart to figure it out. There's Knowledgebase solution 394763 for exactly this problem.

And just like that, kerberized login to FreeIPA started working again.

Knowing what was wrong, I've started noticing (and understanding) other people have the same kind of issues.

One problem down, two to go. Replication between my replicas is not working. Again with unhelpful message:

[12/Jan/2024:14:33:32.812986996 +0100] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=kaitain.pipebreaker.pl-to-okda.pipebreaker.pl" (okda:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later.

I did all the steps from KB 2988311 but it didn't help.

Second problem, ACME on FreeIPA ceased to work for me.

I hadn't had willpower to debug these two, yet.

How systemd exponential restart delay works

Tomasz Torcz — Tue, 02 Jan 2024 20:25:44 GMT

Since the beginning, systemd had a Restart= directive to do just that – restart a service when it fails or exists. Some tuning was provided by RestartSec= (how long to wait between restarts), StartLimitBurst= (how many times to try) and few minor directives.

Starting with systemd v254, we have two new knobs:

RestartSteps= the number of steps to take to increase the interval of auto-restarts
RestartMaxDelaySec= the longest time to sleep before restarting a service as the interval goes up

Together, they provide ability to exponentially extend wait-time between restarts. First restart is quite quick, but then system waits more and more. Here what it means.

Let assume a service with following settings:

Restart=always
RestartSec=100ms (the default)
RestartMaxDelaySec=10s
RestartSteps=5

Upon a failure, systemd will wait (rounded a bit):

100ms until first restart
250ms until next restart (first step)
630ms until next restart (second step)
1.58s until next restart (third step)
3.98s until next restart (fourth step)
10.0s until next restart (RestartSteps=5 – fifth step) and following restarts

Second example. Given:

RestartSec=1s
RestartMaxDelaySec=10s
RestartSteps=3

subsequent restarts will be done after waiting:

100ms
2.15s (step 1)
4.64s (step 2)
10.0s (step 3, as in RestartSteps=3).

Hope this helps. You can find the exact formula in src/core/service.c.

Migrated home server to the UEFI boot

Tomasz Torcz — Fri, 17 Feb 2023 11:58:31 GMT

I've migrated my home server to boot using UEFI. It means suprising number of things:

I did something useful during my unplanned PTO days
Fedora's GRUB instructions are useful and precise
I was really wrong in my rant couple years ago. Although I had to switch firmware into Windows 8 (no CSM) mode today.
the server is ready for modernization later this year, the upgrade of fives: some AMD Ryzen 7xxx (AM5 socket), DDR5, PCIe5. Packed in a SSUPD Meshroom S case. Getting back to the integrated GPU and hopefully lowering power consumption (now it idles at around 90W🙁)
I can unsubscribe from BIOS Boot SIG, as this was my last legacy-booting computer. The SIG mailing list is completely empty, apparently all the ruckus with needing BIOS booting within Fedora has no real standing.

Obvious webdevel things dump

Tomasz Torcz — Tue, 15 Feb 2022 14:29:29 GMT

Having just spent hours debugging simple Flask application, I need to went:

1. tzdata package in fedora-minimal container image is broken. RPM database lists files which are not installed (actually, they were removed during base container build). It's getting fixed.

Symptom: FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/zoneinfo/zone.tab'

Workaround: microdnf reinstall tzdata.

2. HTML checkboxes are weird. When they are set, they appear in HTTP request data with value. When unset, they are not in the request (i.e. one cannot look for checked = False). Thus, following is enough to find if the checkbox has been checked:

if "checkbox_name" in request.form:
 …

3. default-on checkboxes with state retained over POST… This one was tricky. Apparently, initialising Flask-WTForms with request data on the first load overwrites default state. Solution: use form data in POST handler only.

class SomeForm(FlaskForm):
  checkbox_name = BooleanField("Some label", default=True)

  @app.route("/", methods=["GET", "POST"])
  def main():
      # doing "form = SomeForm(request.form)" here would obliterate default checkbox state; don't do it

      if request.method == "POST":
          form = SomeForm(request.form)
          …
      else:
          # not a POST? start with default, empty form
          form = SomeForm()
          …

The lights are blinking, but a part of me is broken

Tomasz Torcz — Sat, 12 Feb 2022 13:34:33 GMT

Software neglect forces perfectly good hardware obsolete. I was suprised it could strike such basic devices as an ethernet switch.

Few years ago I bought a TP-Link gigabit switch for home network. TL-SG2216 model ticked all the boxes: 16 ports, 2 SFP slots (if I ever get FTTH), VLANs, IPv6 support and remote management, 5 years warranty. Although ssh required strange dances since the beginning (ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -oCiphers=aes256-cbc -oHostKeyAlgorithms=+ssh-dss …), the HTTPS interface worked fine. Until it broke last year.

Firefox was adamant – SSL_ERROR_NO_CYPHER_OVERLAP. Chromium threw similar tantrum: ERR_SSL_VERSION_OR_CIPHER_MISMATCH. HTTPS management UI on my switch ceased to be secure enough for modern browsers. The world rushed forward, obsoleting and disabling old, unsecure algorthms and protocols. In the meantime my switch stood still. Eventually the world surpassed what was possible with TP-Link.

From my perspective, the web UI broke. I bought a switch with HTTPS management, and this feature stopped working. So, in the last days of my warranty coverage, I reported the issue to TP-Link. Long story short, it was denied on a technicality. I should have opened the issue through the distributor who sold me the switch, not directly with TP-Link. I've got this information after the warranty lapsed, which ended the story.

But frankly, I do not think TP-Link would be able to salvage this situation. I put only a tiny blame on them.

They designed the switch couple years ago. It was good enough and worked with the ecosystem of its time. They provided couple minor firmware upgrades, and after a decade on the market, EOLed the switch.

It is the world which changed.

Couple decades ago, network hardware once deployed would be working for years, practically forever. Security risks weren't so serious as we have now. No one would deprecate and remove protocols in the name of security. How long 3DES (hell, even single DES?) was with us?

Today, security is paramount. Our lifes are entwined with TCP/IP services. Protocols are phased out and improved when needed. Web browsers automatically update to newer versions. And this is good. But manufactures need to catch up. Product development cannot end when market availability starts. Fixes, updates are required to do more work to retain functionality. Even bigger changes like adding new TLS protocol needs to happen during the device lifetime. We need regulators (like EU) to enforce that.

We also need to vote with our wallets. Maybe pay a bit more, but buy from companies providing better support for their products, through their shelf-life and beyond. Couple of years ago I was dissapointed [2] with Motorola not delivering on promise of upgrades for Moto G. I will not buy Motorola Android phone again.

What options do I have with the switch?

I could replace the switch with something modern, but it's just spending money and generating electro-waste.

I could get a little VM with obsolete operating system and an old browser just to manage this switch. This is unsafe, makes me shudder and is too cumbersome to even consider.

Finally, I can manage the device using plain, unencrypted HTTP. Given it's accessible in my LAN only, this is the way to go. I will be sad inside, but that's the only loss.

sslscan output below shows the sad state of TP-SG2216 web management:

Testing SSL server distrans.pipebreaker.pl on port 443 using SNI name distrans.pipebreaker.pl

SSL/TLS Protocols:
SSLv2     disabled
SSLv3     disabled
TLSv1.0   enabled
TLSv1.1   disabled
TLSv1.2   disabled
TLSv1.3   disabled

Supported Server Cipher(s):
Preferred TLSv1.0  128 bits  RC4-SHA
Accepted  TLSv1.0  128 bits  RC4-MD5
Accepted  TLSv1.0  112 bits  DES-CBC3-SHA
Accepted  TLSv1.0  56 bits   TLS_RSA_WITH_DES_CBC_SHA

I am sorry if you read this at “We Make Fedora” :(

Tomasz Torcz — Sat, 27 Nov 2021 16:00:00 GMT

Chances are you are reading this post on a blog aggregator (“planet”) named We Make Fedora (lack of link deliberate). We Make Fedora was apparently organised by Daniel Pocock. He copied all the blog sources from Planet Fedora, but mixed-in some shady, anonymous sources. This way, “conspiracy theory”-like nameless posts build on credibility of real Fedora developer's blogs.

I do not want neither my name nor mine posts to appear alongside such content. Examples of recent posts:

borderline slanderous posts about Chris Lamb, former Debian leadership (on allegely Fedora-related site!) personal matters
insinuations about trafficking of underage girls by open source communities, on one occasion implicating Justin W. Flory from Fedora
comparison between Adolf Hitler's activities (!!!) and German Free Software Foundation of Europe
equaling Google's Youth Hacking 4 Freedom with nazi Concentration Camps

Holocaust is a very painful memory for European society. It should never be forgotten, the memory should be treated with seriousness in places like Yad Vashem Institue. It should never be used in an anonymous slanderous content.

Again, I do not want my name used to legitimise above content. I've asked Daniel to remove my blog from his aggregator three times:

on Tue, 24 Aug 2021 12:00:33 +0200, in a direct email
on Mon, 6 Sep 2021 09:17:35 +0200, in an email to Daniel and fedora-devel (apparently still in moderation queue)
on Mon, 6 Sep 2021 17:08:42 +0200, in a reply to Daniel

Despite above efforts, my posts still appear on We Make Fedora. I don't want to see my name on a site with such dubious content. I going to block access to my RSS/ATOM feeds for the aggregator. Unfortunately anyone can utilize public feeds to create any site he wants.

ACME & FreeIPA – super easy

Tomasz Torcz — Thu, 25 Nov 2021 11:05:24 GMT

This post will be short. Recent FreeIPA versions contain ACME server implementation, which makes TLS certificate issuance a breeze.

FreeIPA: FreeIPA is a solution giving you LDAP for user accounts, CA for issuing certificates and Kerberos for SSO, delivered with nice WebUI in an integrated package.
ACME: Automated Certificate Management Environment is a protocol designed to automate process of getting a TLS certificate. It was popularised by Let's Encrypt, BuyPass, Venafi and others.

When you have FreeIPA, you have your own Certificate Authority. It is most sensible to use it for securing your internal endpoints. Your clients should already trust this CA. Your internal network may not be reachable by external ACME providers. And you may want to hide your internal hostnames from appearing in global Certificate Transparency Databases.

Ready to start? Make sure package pki-acme is installed on you FreeIPA server. Next, enable ACME functionality:

$ ipa-acme-manage enable

The ipa-acme-manage command was successful

Done! Now configure the client – for my k8s I'm using awesome cert-manager. Start with definition of a ClusterIssuer with your FreeIPA URL (put you own email and server address, of course):

---
kind: ClusterIssuer
apiVersion: cert-manager.io/v1
metadata:
  name: pipebreaker-freeipa
  spec:
    acme:
      email: tomek@pipebreaker.pl
      server: https://ipa-ca.pipebreaker.pl/acme/directory
      privateKeySecretRef:
        # Secret resource that will be used to store the account's private key.
        name: issuer-pbrk-account-key
      solvers:
        - http01:
            ingress: {}

Second (and last) step is to annotate each ingress which should get a TLS certificate automatically provided:

---
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  annotations:
    cert-manager.io/cluster-issuer: pipebreaker-freeipa
[…]

And that's basically it. After few moments certificate should be issued:

When using cert-manager, make sure it's version 1.6.0 or later. There was a fix for ambiguity in spec which solved some interoperability problems with FreeIPA implementation. The fix may be backported for older cert-manager releases.

Makeshift Kubernetes external load balancer with haproxy

Tomasz Torcz — Fri, 30 Jul 2021 12:56:58 GMT

Some time ago I've replaced Google Analytics with Plausible. It works great, except for one tiny thing. The map of visitors was empty. Due to various layers of Network Adress Translations in k3s networking setup, the original client IP address information was not reaching analytics engine.

There are solutions – there is a PROXY Protocol exactly for that case. And Traefik, which handles ingress in k3s, supports PROXY. Only a bit of gymnastic was needed.

Legacy IPv4 traffic entry point to my bare-metal cluster has a form of a small in-the-cloud virtual machine. It routes incoming TCP/443 traffic over the VPN into the cluster. The VM itself is not a part of kubernetes setup – I cannot run any pods on it. I've decided to use Ansible to configure it.

The outcome lives in k8s-haproxy-external-lb and gives me following map:

(greetings Australia, have you found information about the red LED on Sonoff?)

There are few moving parts, but with Python, Kubernetes and Ansible, the result is suprisingly simple:

there's a persistent pod running on the k8s cluster, watching EndPoints exposed by Traefik. When a change occurs (traefik pod restart, replica count modification, etc.) – ansible playbook is triggered. This pod may be seen as a k8s controller.
ansible playbook collects Traefik pod's IP addresses and ports. JSON parsing in ansible is a bit suboptimal: … | first | first | first looks bad but works.
still using ansible, haproxy configuration file is created, put on the edge nodes, and the service is restarted. I've selected haproxy because:
- it implements PROXY protocol
- I did similar thing on OpenShift
haproxy passes received traffic directly to Traefik. This happens at the TCP level. TLS is terminated at Traefik, and certificates do not leave kubernetes cluster.

Some minimal preparations were needed. Communication between edge node and kubernetes pod network had to be established. This was done in an instant, thanks to Wireguard. SSH keypair for ansible had to be put in a k8s Secret and distributed among edge nodes. Finally, small fix was needed in ansible itself: local connection plugin was not happy when run in a container, as random user without an entry in /etc/passwd.

Traefik had to be configured to trust PROXY protocol information and generate X-Forwarded-For headers. Plausible utilized information in those headers without additional tinkering.

Configuration details are described at https://github.com/zdzichu/k8s-haproxy-external-lb.

PSA: kernel 5.12.11 is safe for bcache, again

Tomasz Torcz — Wed, 16 Jun 2021 10:32:18 GMT

With the release of Linux kernel 5.12.11, bcache is safe to use, again. The patch bcache: avoid oversized read request in cache missing code path has been merged.

Due to changes in 5.12 kernels, bcache was prone to cause a BUG_ON() when submitting large I/O requests. The result was a kernel panic or a system freeze. Problem was reported in couple of places: Fedora bug#1965809, bcache mailing list #1, bcache mailing list #2.

At some point UI loses usefulness

Tomasz Torcz — Thu, 29 Apr 2021 18:47:21 GMT

When it comes to configurability, modern software often hits a sweet spot. We are given nice, usable User Interface (UI) helping with configuration – by hinting, auto-filling and validating fields. Additionaly, the configuration itself is stored in text format, making it easy to backup and track changes. For example in git version control system.

Recently I encountered at least two cases, where the above features conflict.

Argo CD

Argo CD is a wonderful tool to implement GitOps with you Kubernetes cluster.

Kubernetes is configured by plain text files in YAML format. That's a perfect form to track in git. Argo CD provides synchronization service: what you have in git repository is applied to kubernetes. Synchronization could be automatic or you can opt to sync manually. In later case, Argo CD provides a nice diff view, showing what's currently configured and how should it be.

Argo CD also has a nice concept of responsibility boundaries: it cares only about YAML sections and fields present in the git repo. If you add new section on the running cluster, it won't be touched. It may be a single field, for example – replicas:

Above can be utilized when you manage Argo CD by Argo CD. install.yaml file defines configuration resources likes ConfigMaps and Secrets, yet it doesn't provide actual data: sections. When you configure Argo CD installation – using nice web UI, no less - data: sections are created and configuration is stored into k8s cluster.

Those sections are not part of what is stored in git repository, so they will neither be touched nor rewritten.

But what happens when we want to store the Argo CD configuration in the repository, and gitops it to the Moon and back?

If we add data: sections, they will be synced. But we will lose ability to use nice UI directly! As UI makes changes on the running cluster, Argo CD will notice live configuration differs from git repository one. It will overwrite our new configuration, undoing changes.

If we want to gitops configuration, we basically must stop using UI and manually add all changes to the text files in the repository!

Grafana

Grafana is another cool project. It is a graphing/dashboarding/alerting solution, which looks pretty and is quite powerful, yet easy to use. Mainly because user interface is a pleasure to use; all changes are visible instantly and we are free to experiment.

Behind the scenes dashboards are just text (JSON) files. Great, text, let's store it in git! Well…

First of all, generated JSON tend to be dynamic. If you do some manipulations in the UI, sections in final file may move relative to each other. Even if the content does not change.

Second, those documents tend to be verbose. Like, really. It is not recommended to edit them manually, better use some templating language. For example grafonnet, which is a customisation of jsonnet - templating for JSON.

The reader probably sees where it's all going. Decision to use grafonnet makes the whole nice UI almost useless, as it spits JSON only. Again, to have better control, history and visibility we must forego one of the main selling points of the software.

Solutions? Are there any?

Frankly, I don't see anything perfect. We have some workarounds, but they feel cumbersome.

For Argo CD we can disable self-healing of an app. That is, disable automatic synchronisation. That way we can still use the UI to do the configuration. Argo CD will notice out-of-sync status between git and live cluster. It will also provide helpful diff: showing exactly how changes made in the UI are reflected in the text configuration.

When we're happy with the changes, we have to extract them from diff view and commit to the git repository. Cumbersome. And we lose active counter-measurements against configuration drift.

Grafana problem we fight with sandbox instance. Any (templated) dashboard can be loaded, then customised with clickety click and exported to JSON. Now the tedious part begins: new stuff from JSON need to be identified, extracted, translated back into templating language and hand merged into grafonnet dashboard definition.

The improved dashboard should be imported into sandbox again and verified. If it is all right, it could be promoted to more important environments. Cumbersome².

I'm very interested in better solutions. If you have comments, ideas, links, please use comments section below!

KubeCon NA 2020 talks to watch, part 1

Tomasz Torcz — Sat, 09 Jan 2021 12:47:23 GMT

Pandemic situation forced most of the conferences to go on-line. On the one hand, it's not the same experience as in-person attendance. On the other hand - I can participate in events I wouldn't be able to travel to. Therefore I took part in KubeCon + CloudNativeCon North America 2020 last November. Moving to online format reduced price from $1000+ to just $75, which made it easier to justify 😊. It was a bit unusual to be at work in the morning, then move to couch and stay at the conference past midnight.

The online setup was quite good. There were virtual "booths" one would expect at expo – with demos, links to more materials and exhibitor's crew available for chat. There were additional number of channels on CNCF Slack. I followed the announcement one, sponsors one (heaps of interesting information there!) and some run by specific companies.

During the Conference there were some "meet the maintainer" events and accompanying gatherings. Those mainly had a form of Zoom (the owners of Keybase) video meetings where one could chat with the developers. I liked these!

The main course of conference are talks. There were plenty. Sometimes there were a dozen or so parallel tracks, so I did not have a chance to watch everything. I'm slowly working through backlog of things I missed. The talks itself were pre-recorded, but after the talk there was a live Q&A session with the speaker. Sadly, the Q&A is not available in recordings below. I guess this was one of the exclusive perks for attendees.

Below you'll find part 1 of my selection of most interesting talks. Second part will be coming later, but you can find all descriptions and links to the videos at https://kccncna20.sched.com/.

PKI the Wrong Way: Simple TLS Mistakes and Surprising Consequences - Tabitha Sable, Datadog

The Quest for the Ultimate Kubernetes Homelab - Dan Garfield, Codefresh

Stop Writing Operators - Joe Thompson, HashiCorp

Clean Up Your Room! What Does It Mean to Delete Something in K8s - Aaron Alpar, Kasten

How to Multiply the Power of Argo Projects By Using Them Together - Hong Wang

Stress and Mental Health in Technology - Dr. Jennifer Akullian, Growth Coaching Institute

The Open Source Revolution: How Kubernetes is Changing the Games Industry - Dominic Green

Admission Control, We Have a Problem - Ryan Jarvinen, Red Hat

High Performance KubeVirt in Action - Huamin Chen, Red Hat & Marcin Franczyk, Kubermatic

This one is from last year, but interesting: How the Department of Defense Moved to Kubernetes and Istio - Nicolas Chaillan

An observation: as for virtual conference, this one had a hefty carbon footprint! Imagine how many planes were flown to deliver these:

k3s is tiny and cute

Tomasz Torcz — Mon, 14 Dec 2020 11:17:01 GMT

After KubeCon I've decided to give k3s a try. And I'm impressed!

K3s is a small distribution of Kubernetes (k8s), Linux container orchestrator system. It's really tiny while being functional. One starts with a single 52MiB binary and after few seconds there's a functional installation with half a dozen of system pods. It's a far cry from OKD and its resource hunger. Of course, compared to OKD, there's much less functionality in k3s, but enough for most cases (including mine).

First, I'm writing now, because only recently cgroupsv2 support was added to k3s. Previously it just didn't work on modern systems, like Fedora.

Second, the etcd database, widely perceived as a mandatory part of k8s, is optional in k3s! By default embedded SQLite is used – enough for simple scenarios. I'm particularly happy for Postgresql support. Yes, you can have your Kubernetes working with pgsql.

But K3s is not a single-node solution only. Adding worker nodes is simple; High-Availability solutions for control-plane looks sound (haven't tried yet, but it's on my TODO).

For networking one can easily encrypt inter-node traffic using WireGuard. It's a matter of single switch for provided flannel network backend. Higher level needs? k3s ships with Traefik (which supports ACME for getting TLS certificates) and klipper-lb.

Unfortunately at the lowest level, networking still depends on iptables. This was horrible choice in the beginning of Kubernetes, already 15 years obsolete when it was selected. With known performance problems: nf-hipac tried to solve the problems when, in 2002? Huawei replaced iptables with Linux IP Virtual Server in 2017 to have k8s scale. But the solution is still not default in upstream Kubernetes.

Anyway, for small cluster k3s with iptables should work fine, but it has a potential to demolish your carefully tuned firewall configuration. So beware. For the positive aspect, go and read klipper-lb entire source code. This is world championship in simplicity and getting things done with existing infrastructure.

Keeping cluster up-to-date can be automated with system-upgrade-controller, which downloads new version and restarts the services. Simple.

k3s is provided by our Rancher friends at SUSE. I strongly recommend giving it some attention (at the moment INSTALL_K3S_COMMIT=fadc5a8057c244df11757cd47cc50cc4a4cf5887 works for me). Besides, I vaguely remember one needs k8s cluster to build some Fedora content. ;)

Unrelated blog summary

This year I've strived to write at least one note per month. Well… I've managed to write 9 notes only. In 2021 I will do better!

Trying OKD4: home lab needs to grow

Tomasz Torcz — Sun, 26 Jul 2020 12:27:19 GMT

I've spent last week tinkering with OKD4, which is open source base for OpenShift. OpenShift in turn is Red Hat's distribution of Kubernetes. Such opensource/commercial differentiation is quite popular among Red Hat products. OKD/OpenShift relation is like AWX/Ansible Tower, Spacewalk/Satellite, WildFly/JBoss, oVirt/RHEV etc.

I already had previous version deployed - 3.11, then called Origin, not OKD. My cluster consists of two old ThinkPads and a virtual machine, and I was planning to redeploy OKD4 on them. But first some PoC on virtual machines.

So I started with minimal viable cluster – 3 schedulable master nodes. (There's a Code Ready Containers version, too – 1 node cluster, but it's non-upgradable). Requirements table looks scary – 4 CPU cores and 16GiB per master – but that's probably an overkil, right?

As it turned out, I won't be able re-use my legacy hardware to host new cluster. Oh boy, OKD4 is massive.

That's the cluster dashboard just after the installation. For all practical purposes it's an empty platform, I haven't deployed any of my stuff yet. Almost 5 CPU cores used and 10 gigs of memory. Huh.

My VMs setup is below minimal requirements, each has 10GiB of RAM and 3 CPU cores. I'm used to think that you can't skimp on memory, but you can assign fewer CPU cores – at worst everything would be slower. That won't fly with OKD. Components define CPU needs with requires: sections. If you don't deliver, you'll get admission errors and the pods will not run.

So my home lab would need to be expanded with 64GiB of RAM and some 16 threads CPU, just for OKD. Speaking of CPU…

I'm seriously taken-aback with CPU usage of this empty cluster. I don't know exactly what's using so much. There is Prometheus scraping metric, one CronJob and a handful of healthchecks, that's all. Everything should be event-driven and just be idle when not running any workload. Right now OKD wastes tremendous amount of power just to do nothing.

Enough complaining. I like how OKD is managed and configured.

OKD owns the master nodes, including operating system (Fedora CoreOS). Everything is managed from within the cluster, and configured like normal Kubernetes resources - via YAML. LDAP connectivity, custom logo, apiserver's TLS certificates… create a ConfigMap, update a field in resources definition and it just happens.

I much prefer eventual consistency, asynchronously achieved by k8s operators, to using openshift-ansible. The latter touches everything and tends to break when something isn't 100% right. Which is cumbersome when each run takes 40+ minutes.

With OKD you have clean bootstrapping (with ignition config served by bootstrap k8s node – how cool is that?) and you receive basic working cluster, which you customize using kubernetes YAMLs.

Documentation is also very nice and seem to cover every common (and some less common) case.