Major refactor of snmalloc (#343)

# Pagemap
 
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.

This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.


# Backend

There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.

# Thread local state

The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.

# Difference

This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.

src/test/func/sandbox/sandbox.cc

@@ -1,4 +1,4 @@
-#ifdef SNMALLOC_PASS_THROUGH
+#if defined(SNMALLOC_PASS_THROUGH) || true
 /*
  * This test does not make sense with malloc pass-through, skip it.
  */
@@ -16,25 +16,14 @@ using namespace snmalloc;
 
 namespace
 {
-  /**
-   * Helper for Alloc that is never used as a thread-local allocator and so is
-   * always initialised.
-   *
-   * CapPtr-vs-MSVC triggering; xref CapPtr's constructor
-   */
-  bool never_init(void*)
-  {
-    return false;
-  }
   /**
    * Helper for Alloc that never needs lazy initialisation.
    *
    * CapPtr-vs-MSVC triggering; xref CapPtr's constructor
    */
-  void* no_op_init(function_ref<void*(void*)>)
+  void no_op_register_clean_up()
   {
     SNMALLOC_CHECK(0 && "Should never be called!");
-    return nullptr;
   }
   /**
    * Sandbox class.  Allocates a memory region and an allocator that can
@@ -71,7 +60,7 @@ namespace
      * outside the sandbox proper: no memory allocation operations and
      * amplification confined to sandbox memory.
      */
-    using NoOpMemoryProvider = MemoryProviderStateMixin<NoOpPal, ArenaMap>;
+    using NoOpMemoryProvider = ChunkAllocator<NoOpPal, ArenaMap>;
 
     /**
      * Type for the allocator that lives outside of the sandbox and allocates
@@ -81,12 +70,12 @@ namespace
      * memory.  It (insecurely) routes messages to in-sandbox snmallocs,
      * though, so it can free any sandbox-backed snmalloc allocation.
      */
-    using ExternalAlloc = Allocator<
-      never_init,
-      no_op_init,
-      NoOpMemoryProvider,
-      SNMALLOC_DEFAULT_CHUNKMAP,
-      false>;
+    using ExternalCoreAlloc =
+      Allocator<NoOpMemoryProvider, SNMALLOC_DEFAULT_CHUNKMAP, false>;
+
+    using ExternalAlloc =
+      LocalAllocator<ExternalCoreAlloc, no_op_register_clean_up>;
+
     /**
      * Proxy class that forwards requests for large allocations to the real
      * memory provider.
@@ -158,8 +147,9 @@ namespace
      * Note that a real version of this would not have access to the shared
      * pagemap and would not be used outside of the sandbox.
      */
+    using InternalCoreAlloc = Allocator<MemoryProviderProxy>;
     using InternalAlloc =
-      Allocator<never_init, no_op_init, MemoryProviderProxy>;
+      LocalAllocator<InternalCoreAlloc, no_op_register_clean_up>;
 
     /**
      * The start of the sandbox memory region.
@@ -253,7 +243,7 @@ namespace
       // Use the outside-sandbox snmalloc to allocate memory, rather than using
       // the PAL directly, so that our out-of-sandbox can amplify sandbox
       // pointers
-      return ThreadAlloc::get_noncachable()->alloc(sb_size);
+      return ThreadAlloc::get().alloc(sb_size);
     }
   };
 }
@@ -269,7 +259,7 @@ int main()
   auto check = [](Sandbox& sb, auto& alloc, size_t sz) {
     void* ptr = alloc.alloc(sz);
     SNMALLOC_CHECK(sb.is_in_sandbox_heap(ptr, sz));
-    ThreadAlloc::get_noncachable()->dealloc(ptr);
+    ThreadAlloc::get().dealloc(ptr);
   };
   auto check_with_sb = [&](Sandbox& sb) {
     // Check with a range of sizes