for ns in $(kubectl get ns -o jsonpath="{.items[*].metadata.name}"); do
  for cj in $(kubectl get cronjobs -n "$ns" -o name); do
    kubectl patch "$cj" -n "$ns" -p '{"spec" : {"suspend" : true }}';
  done
done

And to enable them again, we can patch them to spec.suspend: false.

for ns in $(kubectl get ns -o jsonpath="{.items[*].metadata.name}"); do
  for cj in $(kubectl get cronjobs -n "$ns" -o name); do
    kubectl patch "$cj" -n "$ns" -p '{"spec" : {"suspend" : false }}';
  done
done

Keep in mind that resuming overdue cronjobs immediately schedules new jobs, so depending on your jobs this might cause some additional load.

$ dig +short TXT whoami.ds.akahelp.net
"ns" "203.0.113.1"

$ ansible localhost -m setup
localhost | SUCCESS => {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "10.24.94.195",
            "172.17.0.1"
        ],
        ...
        "ansible_interfaces": [
            "enp0s31f6",
            "lo",
            "docker0"
        ],
        ...
        "ansible_docker0": {
            "device": "docker0",
            "ipv4": {
                "address": "172.17.0.1",
                "broadcast": "172.17.255.255",
                "netmask": "255.255.0.0",
                "network": "172.17.0.0"
            },
            ...
        },
        ...
        "ansible_enp0s31f6": {
            "device": "enp0s31f6",
            "ipv4": {
                "address": "10.24.94.195",
                "broadcast": "10.24.94.255",
                "netmask": "255.255.255.0",
                "network": "10.24.94.0"
            },
            ...
        },
        ...
        "ansible_lo": {
            "device": "lo",
            "ipv4": {
                "address": "127.0.0.1",
                "broadcast": "",
                "netmask": "255.0.0.0",
                "network": "127.0.0.0"
            },
            ...
        },
        ...
    }
}

In theory we have all the facts right there and for the human eye it is quite easy to spot which IP belongs to which interface. But (as far as I know) there is no obvious way in Ansible to query the interface based on an IP. We could use the shell module and call some ip a | sed | foo command to figure out the network interface but after searching the web for a while I finally figured out a way solve this directly in Ansible:

# debug.yml
- name: Debug
  hosts: localhost
  tasks:
    - name: set_fact | figure out network device of private network
      set_fact:
        private_interface: "{{ hostvars[inventory_hostname]['ansible_' + item]['device'] }}"
      when:
        - hostvars[inventory_hostname]['ansible_' + item].ipv4 is defined
        - hostvars[inventory_hostname]['ansible_' + item]['ipv4']['address'] | ipaddr('10.24.94.0/24')
      with_items: "{{ ansible_interfaces }}"
    - name: debug | print network interface
      debug:
        msg: Interface {{ private_interface | default("not") }} found

Wait, what? Yep.

$ ansible-playbook debug.yml

PLAY [Debug]

TASK [Gathering Facts]
ok: [localhost]

TASK [set_fact | figure out network device of private network]
ok: [localhost] => (item=enp0s31f6)
skipping: [localhost] => (item=lo)
skipping: [localhost] => (item=docker0)

TASK [debug | print network interface]
ok: [localhost] => {
    "msg": "Interface enp0s31f6 found"
}

PLAY RECAP
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

We use the ansible_interfaces fact to get a list of all interfaces, then we loop through all interfaces and use the fact that the interface specific Ansible facts are called ansible_$interface. For each interface we use the when conditions to check if the interface has an IP in the desired subnet. In that case the fact private_interface is set to the current interface, otherwise the task is skipped. This way we can now use the new fact in other tasks or templates. In the when conditions we could also use other filters, for example if we want to match a specific IP instead of a subnet, etc.

Is this a bit hacky? Hell yeah! Does it work? Yes. Will it break in some situations? Probably, for example with several interfaces in the subnet. But depending on the situation it can be a feasible solution.

In December 2011 I bought rented my first server from Hetzner Online, since then there hasn’t been a month without an invoice from them… accumulating to a surprising amount of money. In November 2018 I booted the first four instances on the new Hetzner Cloud which was released earlier in the year… one of those instances is still running smoothly along. I can’t really put a number on it, but it is safe to say that a major part of my engineering knowledge is based on the stuff I have built and broken on this infrastructure over the years.

So, after almost a decade as a very happy customer I am beyond delighted to join the Hetzner Cloud team on the 1st of April. I can’t wait to see how this new chapter turns out.

.tanka:
  image: grafana/tanka:0.13.0
  before_script:
    - cd tanka
    - tk env set --server-from-context='centralserver-k3s' environments/default/

Tanka Diff:
  extends: .tanka
  stage: tanka-diff
  script:
    - tk diff environments/default || if [ $? -eq 16 ]; then echo 0; else echo $?; fi
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event" || ($CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH)
      changes:
        - tanka/**/*
    - if: $CI_PIPELINE_SOURCE == "web"

Tanka Apply:
  extends: .tanka
  stage: tanka
  script:
    - tk apply --dangerous-auto-approve environments/default
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - tanka/**/*
      when: manual
    - if: $CI_PIPELINE_SOURCE == "web"

Until recently it worked fine like this, but suddenly it stopped working and started to produce the following error:

Executing "step_script" stage of the job script
Error: unknown subcommand `sh`
Usage:
  tk [command]
Available Commands:
[...]
Cleaning up file based variables
ERROR: Job failed: exit code 1

So, this is rather weird… it sounds like GitLab CI tries to run something like tk sh ..., why would it do that? There is no tk sh command in my .gitlab-ci.yml, it doesn’t seem to make any sense. After searching a bit I found this helpful explanation: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/65110#note_198073241. Looking at the Dockerfile of Tanka confirms that it calls tk in the entrypoint. As explained in the linked issue the combination of the entrypoint and the way the Docker executor starts the container leads to a start command like tk sh -c [rest of shell detection...]. With this knowledge it is even reproducible locally:

$ docker run --rm -it grafana/tanka:0.13.0 sh -c 'echo Hello World'
Error: unknown subcommand `sh`

Usage:
  tk [command]

Available Commands:
[...]

Overriding the entrypoint with an empty string, as described in the issue, fixes the problem:

$ docker run --rm -it --entrypoint "" grafana/tanka:0.13.0 sh -c 'echo Hello World'
Hello World

The same fix works in the .gitlab-ci.yml file:

.tanka:
  image:
    name: grafana/tanka:0.13.0
    entrypoint: [""]
  before_script:
    - cd tanka
    - tk env set --server-from-context='centralserver-k3s' environments/default/

This leaves me with one question: Why did it work before? I think when it worked the job was running with the Kubernetes executor and then it broke when I switched it to the Docker executor. So, my guess is that the Kubernetes executor already overrides the entrypoint by default.

Even though the initial error message was rather confusing the technical explanation makes a lot sense and the fix works fine. Also it is worth mentioning that this is not really a Tanka issue! This will happen with every Docker image with an ENTRYPOINT in the Dockerfile, the fix should also work for all of them.

Imports should be grouped in the following order:

• Standard library imports.
• Related third party imports.
• Local application/library specific imports.

You should put a blank line between each group of imports.

Taking care of that manually is just tedious work! This is something the editor or IDE should take care of instead. VS Code has a feature called Organize Imports to do exactly that, but unfortunately by default we have to run it manually. Instead, it would be much nicer if it was triggered each time we save a file. We can achieve that by adding the following to VS Codes settings.json:

{
  ...
  "editor.codeActionsOnSave": {
    "source.organizeImports": true
  }
  ...
}

And just like that we get sorted and grouped imports each time we save a file.