Writing yet another Kubernetes templating tool


    If you are working with Kubernetes environment then you probably make use of several existing templating tools, some of them being a part of package managers such as Helm or Ksonnet, or just templating languages (Jinja2, Go template etc.). All of them have their own drawbacks as well as advantages and we are going to go through them and write our own tool that will try to combine the best features.


    So, why not Helm?


    There are a number of articles criticizing Helm (e.g. just one of them: Think twice before using Helm). The main issue with Helm is that it works with string representations and Kubernetes manifests are (json) objects. The real hell for a Helm chart developer begins when (s)he needs to calculate indents for a yaml manifest, sometimes it looks like this (it's a real example from my chart):


    spec:
      jobTemplate:
        spec:
          template:
            spec:
              containers:
              - name: my-awesome-container
                resources:
    {{ toYaml .Values.resources | indent 14 }}


    But Helm today is de-facto the standard for Kubernetes applications packaging. The main advantage of Helm is large community and a big number of public repositories with charts. And recently Helm developers have announced a Helm hub. So Helm today is like Docker — it's not the only one but it has community and support.


    There are promising changes coming with Helm 3 release but nobody knows when it could be.


    To conclude, Helm advantages:


    • Large community and a number of public charts
    • (Relatively) human-friendly syntax. At least it's yaml + go template ;)

    Drawbacks:


    • Working with strings and not objects
    • Limited number of operators and functions you can use

    OK, then maybe Ksonnet?


    If you are comparing Helm to Ksonnet the latter has a huge advantage, namely it works with objects. Ksonnet is a tool based on JSON templating language Jsonnet. Another cool feature about Ksonnet is that it has Kubernetes-API-compatible Jsonnet libraries that you can import into your template and work with Kubernetes objects like in any OOP language:


    local k = import "k.libsonnet";
    local deployment = k.apps.v1beta1.deployment;
    
    local appDeployment = deployment
      .new(
        params.name,
        params.replicas,
        container
          .new(params.name, params.image)
          .withPorts(containerPort.new(targetPort)),
        labels);

    Looks impressive, doesn't it?
    It is a little less neat when you are working not with API objects but with just json objects imported from yaml/json file:


    {
      global: {},
      components: {
        "deployment-nginx-deployment-dkecx"+: {
          spec+: {
            replicas: 10,
            template+: {
              spec+: {
                containers+: [
                  {
                    name: "nginx",
                    image: "nginx:latest",
                    ports: [
                      {
                        containerPort: 80,
                      },
                    ],
                  },
                ],
              },
            },
          },
        },
      },
    }

    But still it is something and it's better than working with strings in Helm. The disadvantage of Ksonnet is that it has smaller community and less packages than Helm (though you can import Helm charts into your Ksonnet project, but you will be working with them as json objects, not as jsonnet-library objects). And as a result of a smaller community and contribution there is lack of some features when you trying to write your own chart. One of them I experienced myself: you know that in Helm you can build up a ConfigMap from a directory containing a number of config files this way:


    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: conf
    data:
      {{- (.Files.Glob "foo/*").AsConfig | nindent 2 }}

    You can imagine my frustration when I found out that there is no such a feature in Ksonnet. There are workarounds though. But the point is that it's just example of the situation when you are happily writing your chart and then suddenly a lack of some feature stops you on the halfway.
    In total, advantages of Ksonnet:


    • Working with objects
    • Kubernetes-API-compatible Jsonnet libraries
    • Helm chart import support

    Drawbacks:


    • Smaller community and smaller number of Ksonnet-native packages
    • Lack of some functionality you can use in Helm
    • New syntax => increased learning time => increased bus-factor
    • Syntax can sometimes get ugly and less human-readable (especially when making workarounds for lacking features)

    Let's think of an ideal templating tool


    Here some criteria for the "ideal" templating tool:


    • It should work with objects, not strings
    • It should have an ability to work with Kubernetes-API-compatible objects
    • It should have a decent set of functions for working with strings
    • It should work nicely with json and yaml formats
    • It should be human-friendly
    • It should be simple
    • It should have an ability to import existing Helm charts (because this is reality and we want to make use of Helm community)

    That's enough for now. I went through this list in my head and thought to myself: okay, why not try Python? Let's see if it fits into our criteria:


    • Work with objects, not strings. Yes, we can use dict and list types for that.
    • Have an ability to work with Kubernetes-API-compatible objects. Yes, from kubernetes import client
    • Have a decent set of functions for working with strings. Plenty!
    • Work nicely with json and yaml formats. Very nicely.
    • Human-friendly. No shit.
    • Simple. Yes.
    • Ability to import existing Helm charts. That, we are going to add ourselves.

    Ok, looks promising. I decided to write simple templating tool atop of Official Python client library for kubernetes and now let me show you what came out of it.


    Meet Karavel


    There is nothing special or complicated about this tool. I just took Kubernetes library (which gave me an ability to work with Kubernetes objects) and wrote some basic functionality for existing Helm charts (so that one could fetch them and add into their own chart). So, lets have a tour.
    First of all, this tool is accessible at Github repo and you can find a directory with examples there.


    Quick start with Docker image


    If you want to try it out, the simplest way is to use this docker image:


    $ docker run greegorey/karavel -h
    usage: karavelcli.py [-h] subcommand ...
    
    optional arguments:
      -h, --help  show this help message and exit
    
    list of subcommands:
      subcommand
        template  generates manifests from template
        ensure    ensure helm dependencies

    Of course, if you want to template charts you need to mount your chart's directory:


    $ cd example
    $ docker run -v $PWD:/chart greegorey/karavel template .

    So, let's have a look at the chart structure. It is very similar to one of Helm:


    $ cd example
    $ tree .
    .
    ├── dependencies
    ├── prod.yaml
    ├── requirements.yaml
    ├── templates
    │   ├── custom-resource.py
    │   ├── deployment.py
    │   └── service-helm.py
    └── values.yaml
    
    2 directories, 6 files

    Like Helm, it has requirements.yaml file with the same layout:


    dependencies:
      - name: mysql
        version: 0.13.1
        repository: https://kubernetes-charts.storage.googleapis.com/

    Here you just list your Helm dependencies you want to import into your chart. The dependencies go to the dependencies directory. To fetch or update them use the ensure command:


    $ karavel ensure .

    After that your dependencies directory will look like this:


    $ tree dependencies
    dependencies
    └── mysql-0.13.1
        └── mysql
            ├── Chart.yaml
            ├── README.md
            ├── templates
            │   ├── NOTES.txt
            │   ├── _helpers.tpl
            │   ├── configurationFiles-configmap.yaml
            │   ├── deployment.yaml
            │   ├── initializationFiles-configmap.yaml
            │   ├── pvc.yaml
            │   ├── secrets.yaml
            │   ├── svc.yaml
            │   └── tests
            │       ├── test-configmap.yaml
            │       └── test.yaml
            └── values.yaml
    
    4 directories, 13 files 

    Now after we ensured our dependencies let's have a look at templates. First, we create a simple nginx deployment:


    from kubernetes import client
    
    from karavel.helpers import Values
    
    def template():
        values = Values().values
        # Configure Pod template container
        container = client.V1Container(
            name='nginx',
            image='{}:{}'.format(values.nginx.image.repository, values.nginx.image.tag),
            ports=[client.V1ContainerPort(container_port=80)])
        # Create and configurate a spec section
        template = client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(labels={'app': 'nginx'}),
            spec=client.V1PodSpec(containers=[container]))
        # Create the specification of deployment
        spec = client.ExtensionsV1beta1DeploymentSpec(
            replicas=3,
            template=template)
        # Instantiate the deployment object
        deployment = client.ExtensionsV1beta1Deployment(
            api_version='extensions/v1beta1',
            kind='Deployment',
            metadata=client.V1ObjectMeta(name='nginx-deployment'),
            spec=spec)
    
        return deployment # [deployment], (deployment, deployment) are valid

    So, for the template to be valid you need to have template() function which returns either single Kubernetes object or a list/tuple of them. You can find the list of API objects for Python client here.
    As you can see, the code is clean, simple, readable. You can wonder where from values.nginx.image.repository comes? It gets values from the value files you pass when templating the chart, just like in Helm: karavel template -f one.yaml --values two.yaml. We will have a look at them later.


    Okay, what about Helm charts?


    Now, we created our own Deployment. But what if we want to import Helm chart or a part of a chart? Let's take a look at templates/service-helm.py:


    from kubernetes import client
    
    from karavel.helm import HelmChart
    from karavel.helpers import Values
    
    def template():
        values = Values().values
        # Initialize the chart (== helm template --values)
        chart = HelmChart(name='mysql', version='0.13.1', values=values.mysql.helm)
        # Get the desired object from chart
        service = chart.get(name='svc', obj_class=client.V1Service)
        # Create custom objects to add
        custom_ports = [
            client.V1ServicePort(
                name='my-custom-port',
                protocol=values.mysql.protocol,
                port=values.mysql.port,
                target_port=39000,
            )
        ]
        # Add custom objects to the service
        service.spec['ports'] = custom_ports
        # Change Helm-generated label
        service.metadata['labels']['release'] += '-suffix'
        # Delete Helm-generated label `heritage: Tiller`
        del service.metadata['labels']['heritage']
    
        return service # [service], (service, service) are valid

    Simple, huh? Note this line: service = chart.get(name='svc', obj_class=client.V1Service) — we created object of class V1Service form Helm yaml file. If you don't want/need to do that — you can always work with just dict.


    What if I want to create custom resource?


    Well, there is a small issue with that. Kubernetes API doesn't add CRD objects into swagger json definition at /openapi/v2, and Python-client objects are build upon this definition. But you can still easily work with dict objects. Like this:


    from kubernetes import client
    
    def template():
        resource = {
            'apiVersion': 'stable.example.com/v1',
            'kind': 'Whale',
            'metadata': client.V1ObjectMeta(
                name='my-object',
            ),
            'spec': {
                'image': 'my-whale-image:0.0.1',
                'tail': 1,
                'fins': 4,
            }
        }
    
        return resource # [resource], (resource, resource) are valid

    Still looks nice, doesn't it?


    Can I have values for different environments, e.g. dev/prod?


    Yes, you can!
    Let's look at values.yaml first:


    nginx:
      image:
        repository: nginx
        tag: 1.15-alpine
    
    mysql:
      port: 3307
      protocol: TCP
      helm:
        releaseName: my-release
        namespace: prod
        imageTag: '5.7.14'
        service:
          type: NodePort

    Note the helm key inside mysql dict: we used it when specifying values for helm chart chart = HelmChart(name='mysql', version='0.13.1', values=values.mysql.helm). Some Helm charts need releaseName for application naming and namespace for RBAC policies. These two values are passed to Helm as --namespace and NAME arguments in helm template.


    Now, you can specify additional file for prod env, and template all our examples:


    $ karavel template -f values.yaml -f prod.yaml .
    ---
    # Source: templates/custom-resource.py
    apiVersion: stable.example.com/v1
    kind: Whale
    metadata:
      name: my-object
    spec:
      fins: 4
      image: my-whale-image:0.0.1
      tail: 1
    
    ---
    # Source: templates/deployment.py
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: nginx-deployment
    spec:
      replicas: 3
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - image: nginx:1.14-alpine
            name: nginx
            ports:
            - containerPort: 80
    
    ---
    # Source: templates/service-helm.py
    apiVersion: v1
    kind: Service
    metadata:
      annotations: null
      labels:
        app: prod-release-mysql
        chart: mysql-0.13.1
        release: prod-release-suffix
      name: prod-release-mysql
    spec:
      ports:
      - name: my-custom-port
        port: 3308
        protocol: TCP
        targetPort: 39000
      selector:
        app: prod-release-mysql
      type: NodePort
    

    After that you can do kubeclt apply and deploy these objects into the cluster.


    Cool! What about encoding and base64?


    import base64


    What about using Vault for secrets?


    import hvac


    Fetching urls?


    import importlib


    Secure hash functions?


    import Crypto


    You got it. With Python you can do a lot of things with your Kubernetes manifests.


    Is it NIH syndrome?


    No :)
    I am happily using Helm in my current projects. There are things that I miss though. I used Ksonnet in some of my projects as well.
    I would like to think of this tool as a proof-of-concept that we can have templating tools better than Helm and it's not very difficult to develop them using Python. If there is a community interest/need in such a tool we can together continue to develop it. Or we can wait for Helm 3 release ;)


    Conclusion


    I have showed you Python-based templating tool for Kubernetes which has Kubernetes-API-compatible objects support and support for importing Helm charts. Any comments and discussion from community are welcome, and again welcome to the repo.


    Thank you for reading this and have a nice day!


    References


    Share post

    Similar posts

    AdBlock has stolen the banner, but banners are not teeth — they will be back

    More
    Ads

    Comments 1

      0
      а может кто-нибудь посоветовать что-то типа ytt, но чтобы по темплейту генерировались не только YAML-файлы, а и произвольные текстовые файлы?

      Only users with full accounts can post comments. Log in, please.