.NET and Kubernetes setup

Without starting to re-explain all Kubernetes concepts, it is important to keep in mind Kubernetes is a workload manager at the end. Therefore its role is mainly to create light environments (a Pod) hosting some workload in a controlled environment.

But these light environments (containers at the end) are execute on some host (often a Node).

The configuration of the workload is done using descriptor(s) - YAML or JSON files in general. The granularity can vary. The minimal one is a Pod which defines a "virtual machine" in the "old" world, ie a set of containers - think applications.

Then you can add on top of that some control:

A common Pod will look like:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
    securityContext: (1)
        ...
    containers:
    - name: app
        securityContext: (2)
            ...
        image: oci.myregistry.com/myapp:1.2.3
        resources: (3)
            ...
        volumeMounts: (4)
            ...
    volumes: (5)
        ...

Most constraints come from the security contexts.

The container security context mainly defines the security within one container: what is the active user/group of the container, can it be root (user 0:0), does it have access to some host capabilities etc... In short it defines if your container can corrupt the host and therefore other containers as well as its own environment (filesystem for example).

securityContext:
  allowPrivilegeEscalation: [true|false],
  capabilities:
    add: []
    drop: []
  priviledged: [true|false]
  procMount: 
  readOnlyRootFilesystem: [true|false]
  runAsUser: 1234
  runAsGroup: 1234
  runAsNonRoot: [true|false]
  seLinuxOptions: {}
  seccompProfile:
    type: RuntimeDefault
    ...

The pod security context overlap on some aspects with container security but also defines some specific security - mainly for what is shared between pod containers like filesystem permissions:

securityContext:
  appArmorProfile: {}
  fsGroup: 1234
  fsGroupChangePolicy: OnRootMismatch
  runAsNonRoot: [true|false]
  runAsUser: 1234
  seLinuxOptions: {}
  seccompProfile: {}
  supplementalGroups: [4567]
  sysctls: []

Here again some host features are enabled or not (sysctls is a common suspicious one) but the filesystem is also setup (fsGroup to define the group of the filesystem mounted from a PersistentVolume(Claim) for example) and the very important fsGroupChangePolicy to ensure that if the PersistentVolume differs from the expectation (fsGroup), it is aligned by Kubernetes (kubelet) to make it functional for the container.

When you setup the security in your descriptors you have multiple options, one is to configure everything - which is very complicated to do and maintain, another one is to do it iteratively, but the likely most efficient way is to use a linter/validator of descriptor. Most are very easy to integrate with Helm, BundleBee or plain descriptors.

The most known linters are:

As most of the time you can fine more (like kube-lint, datree etc...).

Once validated, you will often end up with the same kind of security context definition:

podSecurityContext: # "securityContext" at pod level
  fsGroup: 1001
  fsGroupChangePolicy: OnRootMismatch
  seccompProfile:
    type: RuntimeDefault
containerSecurityContext: # "securityContext" at container level
  capabilities:
    drop:
    - ALL
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1001
  allowPrivilegeEscalation: false

And here starts the issues. Several applications will run like that without any issue, some others will just fail at startup and others at runtime.

capabilities:
  add:
    - SYS_TIME

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
    containers:
      - name: app
          image: oci.myregistry.com/myapp:1.2.3
        volumeMounts:
          - mountPath: /tmp
            name: tmp
          ...
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 100Mi
    ...

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
    containers:
      - name: app
          image: oci.myregistry.com/myapp:1.2.3
        env:
          - name: TMPDIR
            value: /writable/path
        volumeMounts:
          - mountPath: /writable/path
            name: tmp
        ...
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 100Mi

#define TEMP_DIRECTORY_PATH "/tmp/"

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
    containers:
      - name: app
          image: oci.myregistry.com/myapp:1.2.3
        env:
          - name: DOTNET_RUNNING_IN_CONTAINER
            value: "true"

env:
  # home for the `dotnet` SDK CLI
  - name: DOTNET_CLI_HOME
    value: /mnt/testing-tools/work/dotnet
  # disable telemetry (microsoft metrics)
  - name: DOTNET_CLI_TELEMETRY_OPTOUT
    value: "true"
  # no banner when calling the cli the first time
  - name: DOTNET_NOLOGO
    value: "true"
  # no integrity check - docker image check is sufficient
  - name: DOTNET_SKIP_WORKLOAD_INTEGRITY_CHECK
    value: "true"
  # for nuget migration which is ran the first time and can't be disabled, ensure the directory is writable
  - name: XDG_DATA_HOME
    value: /mnt/work/xdg
  - name: TMPDIR
    value: /mnt/work/tmp
  # simulate a "common" CI (dotnet CLI does some auto config with that, it is redundant with previous cases for now but not bad to flag it)
  - name: CI
    value: "true"

The .NET within Kubernetes journey is quite smooth:

However, there are still a few pitfalls like switching the default user to a controlled one, aligning filesystem permissions and the most vicious one, ensuring /tmp is writable within containers if you use mutex or related API.

But long story short, .NET runs very well within Kubernetes, let's just hope they use the same TMPDIR variable than for Path.GetTempPath() for mutex code in v10.