Running CPU only jobs

Running large CPU-only jobs

Nautilus is primarily used for GPU jobs. While it's possible to run large CPU-only jobs, you have to take certain measures to prevent taking over all cluster resources.

You can run the jobs with lower priority and allow other jobs to preempt yours. This way you should not worry about the size of your jobs and you can use the maximum number of resources in the cluster. To do that, add the opportunistic priority class to your pods:

    spec:
      priorityClassName: opportunistic

Another thing to do is to avoid the GPU nodes. This way you can be sure you're only using the CPU-only nodes and jobs are not preventing any GPU usage. To do this, add the node antiaffinity for GPU device to your pod:

    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: feature.node.kubernetes.io/pci-10de.present
                operator: NotIn
                values:
                - "true"

You can use a combination of 2 methods or either one.