A problem that a lot of sysadmins and developers have is, how do you run a single task on a CPU without it being interrupted? It’s a common scenario for real-time and virtualised workloads where any interruption to your task could cause unacceptable latency.
For example, let’s say you’ve got a virtual machine running with 4 vCPUs, and you want to make sure those vCPU tasks don’t get preempted by other tasks since that would introduce delays into your audio transcoding app.
Running each of those vCPU tasks on its own host CPU seems like the way to go. All you need to do is choose 4 host CPUs and make sure no other tasks run on them.
How do you do that?
I’ve seen many people turn to the kernel’s
isolcpus for this. This
kernel command-line option allows you to run tasks on CPUs without
interruption from a) other tasks and b) kernel threads.
isolcpus is almost never the thing you want and you should
absolutely not use it apart from one specific case that I’ll get to at
the end of this article.
So what’s the problem with isolcpus?
1. Tasks are not load balanced on isolated CPUs
When you isolate CPUs with
isolcpus you prevent all kernel tasks from
running there and, crucially, it prevents the Linux scheduler load
balancer from placing tasks on those CPUs too. And the only way to get
tasks onto the list of isolated CPUs is with
taskset. They are
effectively invisible to the scheduler.
Continuing with our audio transcoding app running on 4-vCPUs example
above, let’s say you’ve booted with the following kernel command-line:
isolcpus=1-4 and you use
taskset to place your four vCPU tasks on to
those isolated CPUs like so:
taskset -c 1-4 -p <vCPU task pid>
The thing that always catches people out is that it’s easy to end up with all of your vCPU tasks running on the same CPU!
Why? Well because
isolcpus disabled the scheduler load balancer for
CPUs 1-4 which means the kernel will not balance those tasks equally
among all the CPUs in the affinity mask. You can work around this by
manually placing each task onto a single CPU by adjusting its affinity.
2. The list of isolated CPUs is static
A second problem with
isolcpus is that the list of CPUs is configured
statically at boot time. Once you’ve booted, you’re out of luck if you
want to add or remove CPUs from the isolated list. The only way to
change it is by rebooting with a different
cset to the rescue
My recommended way to run tasks on CPUs without
by isolating them from the rest of the system with the cgroups subsystem
cset shield command, e.g.
cset you can update and modify the list of CPUs included in the
cgroup dynamically at runtime. It is a much more flexible solution for
Sometimes you really do want isolcpus
OK, I admit there are times when you really do want to use
For those scenarios when you really cannot afford to have your tasks
interrupted, not even by the scheduler tick which fires once a second,
you should turn to
isolcpus and manually spread tasks over the CPU
But for most uses,
cset shield is by far the best option that’s least
likely to catch you by surprise.