Cytat na dziś

Świetnie spostrzeżnie usłyszałem dzisiaj w pracy. Odnośnie utrzymywania dwóch całkowicie odrębnych zespołów – jeden do opracowania i wdrożenia aplikacji, drugi do jej utrzymywania na produkcji.

To tak, jakby mieć dwie reprezentacje piłkarskie. Jedną szkolić i trenować, a drugą wysyłać na zawody i kazać grać na podstawie dokumentacji.

No czo ten Infoshare znowu

Po ostatniej porażce postanowiłem odpuścić sobie Infoshare w roku powrotu do przyszłości. Ale po ostatnim tech.3camp chyba uległem magii marketingu, bo rozważam udanie się do Amber Expo w czerwcu.

Po pierwsze to organizator przyznał, że ostatnimi laty Infoshare za daleko oddaliło sie od twardych tematów technicznych. I że starają się to nadrobić w tym roku. Hmm. Plus dla nich. Oby się udało.

Druga sprawa: imprezy towarzyszące. Zwiedzanie zardzewiałej fanaberii za ćwierć miliarda mam już za sobą i nie widzę potrzeby powtórki. Natomiast ostatniego dnia szykuje się niezły kąsek. Sztuka wystawiona w naszym czarnym zigguracie, specjalnie dla uczestników konferencji. To jest coś, co mnie mocno przekonuje.

Who wrote systemd?

When it comes to systemd middleware, Lennart Poettering often takes the blame and has sole authorship attributed. But there are many more developers (git shows 593 authors in total) – missing their portion of berating, thus unappreciated and unhappy. Over the Winter Holidays I’ve run LWN's “who wrote” scripts to gather more insight into systemd’s developer base.

Developers with most changesets

Developers with the most changesets
Lennart Poettering Red Hat 7104 38.5%
Kay Sievers Red Hat 3711 20.1%
Zbigniew Jędrzejewski-Szmek hobbyist 1446 7.8%
Tom Gundersen Red Hat 948 5.1%
Greg KH Linux Foundation 624 3.4%
Michal Schmidt Red Hat 369 2.0%
Thomas Hindoe Paaboel Andersen GNOME 253 1.4%
David Herrmann hobbyist 233 1.3%
Martin Pitt Canonical 231 1.3%
Harald Hoyer Red Hat 207 1.1%
Dave Reisner Arch 148 0.8%

Lennart leads, but with less than 40% of all changesets. It seems he has written less than half of systemd. Sorry, Lennart :)
Kay Sievers earned second place mostly by maintaining udev for past couple of years. Majority of his commits were made while he was employed by Novell. Kay now works at Red Hat.
Third most active developer, Zbigniew Jędrzejewski-Szmek, commited less than 8% during the history of systemd. Nevertheless, Zbigniew steadily increases his productivity. OpenHub’s graphs for systemd shows he wrote 12% of changesets in 2014.
Clearly, systemd is driven by top five developers. There is a short drop afterwards. Eleventh person on list contributed less than 1% of all changesets. Red Hat as an affiliation appears few times. Further analysis is in the last section – ”Direct comitters”.

Developers with most changed lines

Developers with the most changed lines
Kay Sievers 565610 34.3%
Lennart Poettering 438513 26.6%
David Herrmann 94753 5.7%
Tom Gundersen 70541 4.3%
Greg KH 66951 4.1%
Zbigniew Jędrzejewski-Szmek 61506 3.7%
Michal Schmidt 16948 1.0%
Patrik Flykt 7386 0.4%

Per-line statistics looks a little bit different. Here Kay Sievers leads the pack with one-third of all lines changed. It is again mostly caused by the long history of udev, which was merged into systemd two and a half years ago – in April 2012.
Big individual counts by David Herrmann and Tom Gundersen can be attributed to non-init components they developed in systemd project. David wrote virtual console implementation, hacked on kdbus and logind. Tom created network configuration element of systemd.
Speaking of the networking, Patrik Flykt’s high position comes from refactoring ConnMan's DHCP client library into systemd-networkd. Recently, the library was utilised by NetworkManager 1.0.

Employers with the most hackers

Employers with the most hackers
Red Hat 46 7.1%
Novell 19 2.9%
Intel 14 2.2%
IBM 10 1.5%
Samsung 9 1.4%
Canonical 5 0.8%
Fujitsu 4 0.6%
ProFUSION 4 0.6%
Mandriva 3 0.5%
Dell 2 0.3%
Axis Communications 2 0.3%
Linux Foundation 2 0.3%
OLPC 2 0.3%
SGI 2 0.3%
HP 2 0.3%

While LWN’s scripts try to map each author to his parent company, it’s less robust than other kinds of statistics. People change employers while contributing to the same project. I do not see “hobbyists” in the table above, while over 10% of systemd has been created by them. Some authors contribute patches in their free-time, without relation to the employer. Nevertheless, it is interesting to see more or less recognizable company names in the list.

Direct commiters

There are 26 people with commit access to systemd git. Biggest group – 9 people – work for Red Hat. This is not surprising. RH have choosen systemd as a foundation for their enterprise offering. They had to build expertise and assure sustainable development. Sure way for that is to hire people contributing to systemd.

Next groups come in threes. Three people without clear company ties – I call them hobbyist. Three Debian/Ubuntu developers, with at least one person employed by Canonical. There are also three names from Intel.

Two developers can be associated with GNOME project. There are single developers from openSUSE, Mageia, Arch and CoreOS. There are also single committers from companies: Pantheon Systems and Aldebaran Robotics.

Bear in mind, above only talks about people with direct commit access to systemd’s git. There are companies contributing many patches through the mailinglist, but without dedicated commiter. I can recall few names from Samsung, which has contributed quite a number of patches. Tangentailly, there were only 2 patches from Jolla, which produces mobile phones and tablets with systemd onboard.


Red Hat pays the bills for biggest number of systemd authors. But please note, some of them were hired by RH specifically because of their earlier contributions to systemd. Community around project contains representatives from other major distributions, some of them even with direct commit access. Tom Gundersen, for example, did majority of his work as an Arch developer.

Some parts of systemd came from specific needs of external users. For example: CoreOS sponsored development of systemd-networkd; Pengutronix contributed watchdog support; Pantheon improved scalability with thousand of units, etc.

Archived comments:

quest 2014-12-30 22:03:35

Wow. What a long post!

First off. I didn't know Poettering wasnn't sole developer, but after his recent letter where he complained for people "trying to hire hitman to kill him", I think the other are somewhat lucky.

and second. "author" is only noun. It shouldn't be used as verb >

zdz 2014-12-30 23:15:11

Thanks, I've replaced "author" with synonyms.

Linuxhippy 2014-12-30 23:56:46

As most of the time, Redhat is one of the few companies truly investing in open-source instead of just making bucks with it (canonical which e.g. has their desktop stuff under a license which makes it unuseable for anybody else)

Jristz 2014-12-31 01:07:33

@quest English is a language not normed, in contrast with spanish, french, bulgarian, etc; therefor any complain on what is correct and what is not in english grammar is either correct and wrong at same time, the main objetive is simply to be underestandable for the english readers, since without a norm solid writted like the examples I give, everithing that is underestandable is valid, even using author as a verb.

lRem 2014-12-31 01:15:57

> Google 0%

Impressive ;)

tg 2014-12-31 02:24:00

Nice post.

Minor comments: Dave Reisner works for Google (but I don't know if he contributes on their time) and David Herrmann works (now) for Red Hat.

Out of interest, how does the changed lines statistics handle non-code changes? If stuff like hwdb/*.hwdb and src/libsystemd-terminal/unifont.hex is included it would skew the numbers a lot I guess.

Jristz 2014-12-31 03:17:41

Dave Reiser is Arch developer, right, but they work indirectly to Google since the place where he work was buyed by google around 2009

Adam Williamson 2014-12-31 05:50:20

It's perfectly fine to use 'author' as a verb, e.g. "this changed was authored by Bob". Perfectly good English. I don't really like *any* free dictionary sites, but is the best of a bad bunch, and it lists the verb use of 'author':

really, though, if you want a good dictionary, pay for an OED subscription; it's worth it.

Jono Loves Bacon 2015-01-01 23:05:32

>> Google 0%

>Impressive ;)

Why would you expect Google to contribute to this project?

zdz 2015-01-02 13:39:01

There are 34 patches from Filipe Brandenburger, using his email address. It's hard to say if those were made during his paid hours.

SDSSS 2015-01-02 18:06:05

Well, the top two developers (Poettering and Sievers) have been chided more than once for their attitude, in general, and their sloppiness, in particular.

Anonymous 2015-01-08 00:55:31

Where are you getting the employer data from? I didn't think the LWN database was generally available, and relying on the email domain is not reliable.

Failure modes of incorrect Type= in systemd units

Proper supervision of daemons requires knowledge which cannot be inferred automatically. Some aspects of service behaviour have to be described by administrator. Widely used upstart system manager has 3 type-like job properties: expect daemon, fork and stop. Their description starts with a warning:

This stanza is extremely important: read this section carefully!
The warning is no less true with systemd's Type=.

Wrong type specification impacts service monitoring, failure detection and dependencies handling. There are six service types in systemd. Three basic ones (simple, forking, oneshot) and three more sophisticated (dbus, idle, notify).

Failure to set proper type may not bite you immediately. Service may run for minute or two and then get killed mysteriously. Services depending on wrongly specified one won't be started, even if it appears to run fine; but systemd won't consider the service ready and will not start dependent units.

Type highlights

  • simple - default type; daemon has to stay in foreground. Ready: as soon as the binary is executed (which may be too soon, service may not be ready to serve requests yet).
  • forking - daemon must fork. Ready: after first process exits. If PIDFile= is defined, readiness is delayed until PID is written to the pidfile.
  • oneshot - suitable for running scripts; systemd waits until process finishes. Ready: when main process exits.
  • dbus - for D-Bus services. Ready: when specified BusName= is acquired.
  • idle - like “simple” , but with delayed execution.
  • notify - most flexible and robust type; by defining simple communication protocol between daemon and systemd, precise information about state can be provided. Requires simple patching, for scripts you can use systemd-notify helper. Ready: when service says so.

Basic type determination

Suitable type can be guessed correctly in 75% cases by just running the daemon binary. If it stays in foreground, connected to the terminal, the type is “simple”. If it forks into the background, the type is “forking” (suprisingly!)

 # /usr/sbin/daemon
Copyright 2014 Foo Baz Bar Corp.
Serving Requests…
→ Type=simple
# /usr/sbin/otherdaemon
→ Type=forking

If you need to run a program/script which does its jobs and exits (does not stay running), use oneshot. Please note: if you can choose how the daemon behaves, it generally better to go with "forking". This makes systemd declare "ready" after daemon is actually able to accept connections.

Failure modes

Most of the failures shown below happen after TimeoutStartSec=. By default it’s set to 90 seconds, so lets stick with this value.

Incorrectly selected Type=
simple forking oneshot notify
Correct type simple killed after 90 s1 stays in “activating” state forever2 killed after 90 s3
forking killed almost immediately4 stays in “activating” state forever killed after 90 s
oneshot becomes “failed” after executing5 killed after 90 s killed after 90 s
notify probably works fine killed after 90 s stays in “activating” state forever
  1. systemd expects process to fork and parent to exit. It waits until timeout.
  2. it never gets to the “ready” state, so no units depending on it will start; start timeout is ignored for “oneshot” units.
  3. systemd will wait for ready notification. Until timeout.
  4. ”main” process exits
  5. daemons should not exit, but “oneshot” units should

I hope that explains why sometimes systemd kills your service after a minute. Of course you should read man systemd.service, it contains much more details.