[load-balancer] migrate fn, handle affine change.

Handle the case, exercised in the set_migrate_fn_test, where the
affinity of a thread changes when a migrate function is set. When
determining if we could stay on the old cpu until delayed migration
happened, in the old path we compared against available_mask in the
new path we also included affinity in that decision which was incorrect.

Also we should be comparing against last_cpu, not initial_cpu. It is not
correct to schedule a migration on a new thread so we add in the bypass
if the last_cpu is unset, which is the new thread case.

This brings us inline with scheduler.cc:650 which is the main flow.

Test: Ran k ut with the load-balancer enabled.
Change-Id: Ia83f47412b3bca861abfb2a4d19c8796a57bd5bd
Reviewed-on: https://fuchsia-review.googlesource.com/c/fuchsia/+/405375
Reviewed-by: Nick Maniscalco <maniscalco@google.com>
Testability-Review: Nick Maniscalco <maniscalco@google.com>
Commit-Queue: Ed Coyne <edcoyne@google.com>
diff --git a/zircon/kernel/lib/load_balancer/include/lib/load_balancer_percpu.h b/zircon/kernel/lib/load_balancer/include/lib/load_balancer_percpu.h
index 0c34b49..365e1a9 100644
--- a/zircon/kernel/lib/load_balancer/include/lib/load_balancer_percpu.h
+++ b/zircon/kernel/lib/load_balancer/include/lib/load_balancer_percpu.h
@@ -159,10 +159,11 @@
     return initial_cpu;
   }
 
-  if (unlikely(thread->has_migrate_fn() && initial_cpu_available)) {
+  if (unlikely(last_cpu != INVALID_CPU && last_cpu != lowest_cpu &&
+               thread->has_migrate_fn() && (mp_get_active_mask() & ToMask(last_cpu)))) {
     // Stay where we are, the migrate_fn_ will migrate us later.
     thread->scheduler_state().set_next_cpu(lowest_cpu);
-    return initial_cpu;
+    return last_cpu;
   }
 
   return lowest_cpu;