[roll] Roll fuchsia [kernel][lockup_detector] Implement N-Way detection.

Change the lockup detector so that all CPUs are always checking each
other, instead of relying on CPU-0 to check all of the non-boot-CPUs.

Basically, this CL smashes the "heartbeat check-in" operation together
with the "check lockup conditions" operation.  To prevent multiple
CPUs from accidentally reporting the same condition multiple times, we
introduce a per-cpu concept of "active checker".  Basically, this
behaves like a try-acquire only spinlock.  When CPU-X is being checked
by CPU-Y, X's per-cpu data structure is updated to indicate that Y is
the current active checker.  If CPU-Z comes along and wants to check X
concurrently, it will see that Y is performing the check already, and
simply skip the check this time.

If Y happens to hang during this operation (clearly, this should never
happen, but who can say what might go wrong in the future), it should
be caught by yet another CPU, until we eventually run out of
functioning CPUs, at which point in time, the HW WDT (if present) will
kick in.

In addition to this top level change, the CL also spends a bit of time
re-organizing things in the data structures just a bit.  We currently
have two "conditions" which might trigger the lockup detector; the
"heartbeat" condition, and the "critical_section" condition.

1) Classes were created to hold the logic for each of the conditions.
   This will hopefully make it a bit easier to keep the global/static
   state for each condition separate and easier to understand.
2) Instances of these classes can now be used to observe the
   condition's state for CPU-X while the active CPU has the "checker"
   role for X.  Once the per-cpu bookkeeping for the condition has
   been updated, and checker role has been dropped, reports can then
   be generated (if needed) from the state captured in the condition's
   instance, allowing other threads to continue to perform checks,
   even if the initial CPU encounters trouble during the reporting
   process.
3) Per-cpu state variables for each condition have been organized into
   independent structures in the top level per-cpu structure.  Once
   again, this mostly about organizing state variable for the various
   conditions to make it more clear which variable belongs to which
   condition, and to start a pattern which should be easy to replicate
   as more conditions are added over time.

Finally, change the reporting rate limiters to be
per-cpu-per-condition limiters, and reduce their throttle time to once
every second, instead of once every 10 seconds.  We currently want to
get some data from CI/CQ bots which are hitting the HW WDT, and the
original value of 10 seconds is just too large to see anything but the
first report in the case that the WDT fires.

There are still a lot of things which can be changed here, including
stuff like:
1) Having CPUs pet the WDT during their check-in.  As long as there is
   still at least one CPU functioning, we have a chance of harvesting
   data from a CPU in the case that it appears to be locked up.
2) Panicking if a CPU appears to be so locked up that it will never
   recover.  This is important to implement if #1 is implemented.
   Otherwise, we could lose a CPU but not reboot because the WDT
   continues to get pet.
3) Filling out any such panic induced crash logs with as much useful
   data as we can obtain from the still running CPU(s).  Without this,
   we will only be able to get useful data from CI/CQ devices which
   encounter lockups because they are monitored on serial at all
   times.  Devices in the field are not, so it is essential to get
   data recorded in the crashlog followed by a deliberate reboot, so
   that the data can be uploaded and examined.

Test: Forced lockup conditions on various CPUs under x64 KVM QEMU and
      made sure they were reported properly.  Also ran existing kernel
      lockup detector unit tests.

Original-Bug: 66284
Original-Bug: 70724
Original-Reviewed-on: https://fuchsia-review.googlesource.com/c/fuchsia/+/493668
Original-Revision: f213ab07075a1066501b758daf94ae085120c440
GitOrigin-RevId: 0aef60566943c534f193fd6218ee512bdcbfb6fe
Change-Id: Id20bcdfe1817ac1e0aa312a1c6d8962075a92720