At some point UI loses usefulness

Argo CD

Argo CD is a wonderful tool to implement GitOps with you Kubernetes cluster.

Kubernetes is configured by plain text files in YAML format. That's a perfect form to track in git. Argo CD provides synchronization service: what you have in git repository is applied to kubernetes. Synchronization could be automatic or you can opt to sync manually. In later case, Argo CD provides a nice diff view, showing what's currently configured and how should it be.

Argo CD also has a nice concept of responsibility boundaries: it cares only about YAML sections and fields present in the git repo. If you add new section on the running cluster, it won't be touched. It may be a single field, for example – replicas:

Above can be utilized when you manage Argo CD by Argo CD. install.yaml file defines configuration resources likes ConfigMaps and Secrets, yet it doesn't provide actual data: sections. When you configure Argo CD installation – using nice web UI, no less - data: sections are created and configuration is stored into k8s cluster.

Those sections are not part of what is stored in git repository, so they will neither be touched nor rewritten.

But what happens when we want to store the Argo CD configuration in the repository, and gitops it to the Moon and back?

If we add data: sections, they will be synced. But we will lose ability to use nice UI directly! As UI makes changes on the running cluster, Argo CD will notice live configuration differs from git repository one. It will overwrite our new configuration, undoing changes.

If we want to gitops configuration, we basically must stop using UI and manually add all changes to the text files in the repository!

Grafana

Grafana is another cool project. It is a graphing/dashboarding/alerting solution, which looks pretty and is quite powerful, yet easy to use. Mainly because user interface is a pleasure to use; all changes are visible instantly and we are free to experiment.

Behind the scenes dashboards are just text (JSON) files. Great, text, let's store it in git! Well…

First of all, generated JSON tend to be dynamic. If you do some manipulations in the UI, sections in final file may move relative to each other. Even if the content does not change.

Second, those documents tend to be verbose. Like, really. It is not recommended to edit them manually, better use some templating language. For example grafonnet, which is a customisation of jsonnet - templating for JSON.

The reader probably sees where it's all going. Decision to use grafonnet makes the whole nice UI almost useless, as it spits JSON only. Again, to have better control, history and visibility we must forego one of the main selling points of the software.

Solutions? Are there any?

Frankly, I don't see anything perfect. We have some workarounds, but they feel cumbersome.

For Argo CD we can disable self-healing of an app. That is, disable automatic synchronisation. That way we can still use the UI to do the configuration. Argo CD will notice out-of-sync status between git and live cluster. It will also provide helpful diff: showing exactly how changes made in the UI are reflected in the text configuration.

When we're happy with the changes, we have to extract them from diff view and commit to the git repository. Cumbersome. And we lose active counter-measurements against configuration drift.

Grafana problem we fight with sandbox instance. Any (templated) dashboard can be loaded, then customised with clickety click and exported to JSON. Now the tedious part begins: new stuff from JSON need to be identified, extracted, translated back into templating language and hand merged into grafonnet dashboard definition.

The improved dashboard should be imported into sandbox again and verified. If it is all right, it could be promoted to more important environments. Cumbersome².

I'm very interested in better solutions. If you have comments, ideas, links, please use comments section below!