d900e29424
* Remote dealloc refactor. * Improve remote dealloc Change remote to count down to 0, so fast path does not need a constant. Use signed value so that branch does not depend on addition. * Inline remote_dealloc The fast path of remote_dealloc is sufficiently compact that it can be inlined. * Improve fast path in Slab::alloc Turn the internal structure into tail calls, to improve fast path. Should be no algorithmic changes. * Refactor initialisation to help fast path. Break lazy initialisation into two functions, so it is easier to codegen fast paths. * Minor tidy to statically sized dealloc. * Refactor semi-slow path for alloc Make the backup path a bit faster. Only algorithmic change is to delay checking for first allocation. Otherwise, should be unchanged. * Test initial operation of a thread The first operation a new thread takes is special. It results in allocating an allocator, and swinging it into the TLS. This makes this a very special path, that is rarely tested. This test generates a lot of threads to cover the first alloc and dealloc operations. * Correctly handle reusing get_noncachable * Fix large alloc stats Large alloc stats aren't necessarily balanced on a thread, this changes to tracking individual pushs and pops, rather than the net effect (with an unsigned value). * Fix TLS init on large alloc path * Add Bump ptrs to allocator Each allocator has a bump ptr for each size class. This is no longer slab local. Slabs that haven't been fully allocated no longer need to be in the DLL for this sizeclass. * Change to a cycle non-empty list This change reduces the branching in the case of finding a new free list. Using a non-empty cyclic list enables branch free add, and a single branch in remove to detect the empty case. * Update differences * Rename first allocation Use needs initialisation as makes more sense for other scenarios. * Use a ptrdiff to help with zero init. * Make GlobalPlaceholder zero init The GlobalPlaceholder allocator is now a zero init block of memory. This removes various issues for when things are initialised. It is made read-only to we detect write to it on some platforms.
1.9 KiB
Difference from published paper
This document outlines the changes that have diverged from
the published paper on snmalloc.
-
Link no longer terminates the bump-free list. The paper describes a complex invariant for how the final element of the bump-free list can also be the link node.
We now have a much simpler invariant. The link is either 1, signifying the block is completely full. Or not 1, signifying it has at least one free element at the offset contained in link, and that contains the DLL node for this sizeclass.
The bump-free list contains additional free elements, and the remaining bump allocated space.
The value 1, is never a valid bump allocation value, as we initially allocate the first entry as the link, so we can use 1 as the no more bump space value.
-
Separate Bump/Free list. We have separate bump ptr and free list. This is required to have a "fast free list" in each allocator for each sizeclass. We bump allocate a whole os page (4KiB) worth of allocations in one go, so that the CPU predicts the free list path for the fast path.
-
Per allocator per sizeclass fast free list. Each allocator has an array for each small size class that contains a free list of some elements for that sizeclass. This enables a very compressed path for the common allocation case.
-
We now store a direct pointer to the next element in each slabs free list rather than a relative offset into the slab. This enables list calculation on the fast path.
-
There is a single bump-ptr per size class that is part of the allocator structure. The per size class slab list now only contains slabs with free list, and not if it only has a bump ptr.
[2-4] Are changes that are directly inspired by (mimalloc)[http://github.com/microsoft/mimalloc].