Improve slow path performance for allocation (#143)

* Remote dealloc refactor.

* Improve remote dealloc

Change remote to count down to 0, so fast path does not need a constant.

Use signed value so that branch does not depend on addition.

* Inline remote_dealloc

The fast path of remote_dealloc is sufficiently compact that it can be
inlined.

* Improve fast path in Slab::alloc

Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.

* Refactor initialisation to help fast path.

Break lazy initialisation into two functions, so it is easier to codegen
fast paths.

* Minor tidy to statically sized dealloc.

* Refactor semi-slow path for alloc

Make the backup path a bit faster.  Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.

* Test initial operation of a thread

The first operation a new thread takes is special.  It results in
allocating an allocator, and swinging it into the TLS.  This makes
this a very special path, that is rarely tested.  This test generates
a lot of threads to cover the first alloc and dealloc operations.

* Correctly handle reusing get_noncachable

* Fix large alloc stats

Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).

* Fix TLS init on large alloc path

* Add Bump ptrs to allocator

Each allocator has a bump ptr for each size class.  This is no longer
slab local.

Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.

* Change to a cycle non-empty list

This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.

* Update differences

* Rename first allocation

Use needs initialisation as makes more sense for other scenarios.

* Use a ptrdiff to help with zero init.

* Make GlobalPlaceholder zero init

The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.

src/test/func/first_operation/first_operation.cc

@@ -0,0 +1,112 @@
+/**
+ * The first operation a thread performs takes a different path to every
+ * subsequent operation as it must lazily initialise the thread local allocator.
+ * This tests performs all sizes of allocation, and deallocation as the first
+ * operation.
+ */
+
+#include "test/setup.h"
+
+#include <snmalloc.h>
+#include <thread>
+
+void alloc1(size_t size)
+{
+  void* r = snmalloc::ThreadAlloc::get_noncachable()->alloc(size);
+  snmalloc::ThreadAlloc::get_noncachable()->dealloc(r);
+}
+
+void alloc2(size_t size)
+{
+  auto a = snmalloc::ThreadAlloc::get_noncachable();
+  void* r = a->alloc(size);
+  a->dealloc(r);
+}
+
+void alloc3(size_t size)
+{
+  auto a = snmalloc::ThreadAlloc::get_noncachable();
+  void* r = a->alloc(size);
+  a->dealloc(r, size);
+}
+
+void alloc4(size_t size)
+{
+  auto a = snmalloc::ThreadAlloc::get();
+  void* r = a->alloc(size);
+  a->dealloc(r);
+}
+
+void dealloc1(void* p, size_t)
+{
+  snmalloc::ThreadAlloc::get_noncachable()->dealloc(p);
+}
+
+void dealloc2(void* p, size_t size)
+{
+  snmalloc::ThreadAlloc::get_noncachable()->dealloc(p, size);
+}
+
+void dealloc3(void* p, size_t)
+{
+  snmalloc::ThreadAlloc::get()->dealloc(p);
+}
+
+void dealloc4(void* p, size_t size)
+{
+  snmalloc::ThreadAlloc::get()->dealloc(p, size);
+}
+
+void f(size_t size)
+{
+  auto t1 = std::thread(alloc1, size);
+  auto t2 = std::thread(alloc2, size);
+  auto t3 = std::thread(alloc3, size);
+  auto t4 = std::thread(alloc4, size);
+
+  auto a = snmalloc::ThreadAlloc::get();
+  auto p1 = a->alloc(size);
+  auto p2 = a->alloc(size);
+  auto p3 = a->alloc(size);
+  auto p4 = a->alloc(size);
+
+  auto t5 = std::thread(dealloc1, p1, size);
+  auto t6 = std::thread(dealloc2, p2, size);
+  auto t7 = std::thread(dealloc3, p3, size);
+  auto t8 = std::thread(dealloc4, p4, size);
+
+  t1.join();
+  t2.join();
+  t3.join();
+  t4.join();
+  t5.join();
+  t6.join();
+  t7.join();
+  t8.join();
+}
+
+int main(int, char**)
+{
+  setup();
+
+  f(0);
+  f(1);
+  f(3);
+  f(5);
+  f(7);
+  for (size_t exp = 1; exp < snmalloc::SUPERSLAB_BITS; exp++)
+  {
+    f(1ULL << exp);
+    f(3ULL << exp);
+    f(5ULL << exp);
+    f(7ULL << exp);
+    f((1ULL << exp) + 1);
+    f((3ULL << exp) + 1);
+    f((5ULL << exp) + 1);
+    f((7ULL << exp) + 1);
+    f((1ULL << exp) - 1);
+    f((3ULL << exp) - 1);
+    f((5ULL << exp) - 1);
+    f((7ULL << exp) - 1);
+  }
+}

src/test/perf/low_memory/low-memory.cc

@@ -97,7 +97,7 @@ void reduce_pressure(Queue& allocations)
  * Wrapper to handle Pals that don't have the method.
  * Template parameter required to handle `if constexpr` always evaluating both
  * sides.
- **/
+ */
 template<typename PAL>
 void register_for_pal_notifications()
 {