it is a general problem of systems with more than one CPU to assign the
proper CPU to a task.
The most obvious problem is the cache. If a task is migrating all the
time, the cache has to be reloaded again and again.
The next problem is the location of the data in RAM on NUMA machines. It
is better to leave a thread on a CPU if the RAM connected to it also has
the data stored. This could block a task from running if to many running
tasks have their data stored on the same CPU.
The last, but hardly used parameter, is the CPU temperature. I noticed
that FreeBSD tends to use always the same CPU to start a task. This
makes one CPU real hot while the other stays cool. Taking the CPU
temperature into account for starting at least new threads would also
have an advantage of systems with an less then ideal cooling system.
Spreading tasks all over the system with the coolest CPU being the one
to be started next will make systems a bit cooler.