Cheri-research/snmalloc
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Improve slow path performance for allocation (#143)

Browse Source 
* Remote dealloc refactor.

* Improve remote dealloc

Change remote to count down to 0, so fast path does not need a constant.

Use signed value so that branch does not depend on addition.

* Inline remote_dealloc

The fast path of remote_dealloc is sufficiently compact that it can be
inlined.

* Improve fast path in Slab::alloc

Turn the internal structure into tail calls, to improve fast path.
Should be no algorithmic changes.

* Refactor initialisation to help fast path.

Break lazy initialisation into two functions, so it is easier to codegen
fast paths.

* Minor tidy to statically sized dealloc.

* Refactor semi-slow path for alloc

Make the backup path a bit faster.  Only algorithmic change is to delay
checking for first allocation. Otherwise, should be unchanged.

* Test initial operation of a thread

The first operation a new thread takes is special.  It results in
allocating an allocator, and swinging it into the TLS.  This makes
this a very special path, that is rarely tested.  This test generates
a lot of threads to cover the first alloc and dealloc operations.

* Correctly handle reusing get_noncachable

* Fix large alloc stats

Large alloc stats aren't necessarily balanced on a thread, this changes
to tracking individual pushs and pops, rather than the net effect
(with an unsigned value).

* Fix TLS init on large alloc path

* Add Bump ptrs to allocator

Each allocator has a bump ptr for each size class.  This is no longer
slab local.

Slabs that haven't been fully allocated no longer need to be in the DLL
for this sizeclass.

* Change to a cycle non-empty list

This change reduces the branching in the case of finding a new free
list. Using a non-empty cyclic list enables branch free add, and a
single branch in remove to detect the empty case.

* Update differences

* Rename first allocation

Use needs initialisation as makes more sense for other scenarios.

* Use a ptrdiff to help with zero init.

* Make GlobalPlaceholder zero init

The GlobalPlaceholder allocator is now a zero init block of memory.
This removes various issues for when things are initialised. It is made read-only
to we detect write to it on some platforms.
This commit is contained in:
Matthew Parkinson
2020-03-31 09:17:53 +01:00
committed by GitHub
parent ecef894525
commit d900e29424
20 changed files with 690 additions and 239 deletions
Download Patch File Download Diff File Expand all files Collapse all files

281
src/mem/alloc.h
View File

@@ -49,14 +49,20 @@ namespace snmalloc
FastFreeLists() : small_fast_free_lists() {}
};
SNMALLOC_FAST_PATH void* no_replacement(void*)
{
return nullptr;
}
/**
* Allocator. This class is parameterised on three template parameters. The
* `MemoryProvider` defines the source of memory for this allocator.
* Allocator. This class is parameterised on five template parameters.
*
* The first two template parameter provides a hook to allow the allocator in
* use to be dynamically modified. This is used to implement a trick from
* mimalloc that avoids a conditional branch on the fast path. We
* initialise the thread-local allocator pointer with the address of a global
* allocator, which never owns any memory. The first returns true, if is
* passed the global allocator. The second initialises the thread-local
* allocator if it is has been been initialised already. Splitting into two
* functions allows for the code to be structured into tail calls to improve
* codegen.
*
* The `MemoryProvider` defines the source of memory for this allocator.
* Allocators try to reuse address space by allocating from existing slabs or
* reusing freed large allocations. When they need to allocate a new chunk
* of memory they request space from the `MemoryProvider`.
@@ -65,31 +71,35 @@ namespace snmalloc
* to associate metadata with large (16MiB, by default) regions, allowing an
* allocator to find the allocator responsible for that region.
*
* The next template parameter, `IsQueueInline`, defines whether the
* The final template parameter, `IsQueueInline`, defines whether the
* message queue for this allocator should be stored as a field of the
* allocator (`true`) or provided externally, allowing it to be anywhere else
* in the address space (`false`).
*
* The final template parameter provides a hook to allow the allocator in use
* to be dynamically modified. This is used to implement a trick from
* mimalloc that avoids a conditional branch on the fast path. We initialise
* the thread-local allocator pointer with the address of a global allocator,
* which never owns any memory. When we try to allocate memory, we call the
* replacement function.
*/
template<
bool (*NeedsInitialisation)(void*),
void* (*InitThreadAllocator)(),
class MemoryProvider = GlobalVirtual,
class ChunkMap = SNMALLOC_DEFAULT_CHUNKMAP,
bool IsQueueInline = true,
void* (*Replacement)(void*) = no_replacement>
class Allocator
: public FastFreeLists,
public Pooled<
Allocator<MemoryProvider, ChunkMap, IsQueueInline, Replacement>>
bool IsQueueInline = true>
class Allocator : public FastFreeLists,
public Pooled<Allocator<
NeedsInitialisation,
InitThreadAllocator,
MemoryProvider,
ChunkMap,
IsQueueInline>>
{
LargeAlloc<MemoryProvider> large_allocator;
ChunkMap chunk_map;
/**
* Per size class bumpptr for building new free lists
* If aligned to a SLAB start, then it is empty, and a new
* slab is required.
*/
void* bump_ptrs[NUM_SMALL_CLASSES] = {nullptr};
public:
Stats& stats()
{
@@ -155,8 +165,6 @@ namespace snmalloc
else
return calloc(1, size);
#else
stats().alloc_request(size);
// Perform the - 1 on size, so that zero wraps around and ends up on
// slow path.
if (likely((size - 1) <= (sizeclass_to_size(NUM_SMALL_CLASSES - 1) - 1)))
@@ -202,17 +210,14 @@ namespace snmalloc
UNUSED(size);
return free(p);
#else
constexpr sizeclass_t sizeclass = size_to_sizeclass_const(size);
handle_message_queue();
if (sizeclass < NUM_SMALL_CLASSES)
{
Superslab* super = Superslab::get(p);
RemoteAllocator* target = super->get_allocator();
if (target == public_state())
if (likely(target == public_state()))
small_dealloc(super, p, sizeclass);
else
remote_dealloc(target, p, sizeclass);
@@ -222,7 +227,7 @@ namespace snmalloc
Mediumslab* slab = Mediumslab::get(p);
RemoteAllocator* target = slab->get_allocator();
if (target == public_state())
if (likely(target == public_state()))
medium_dealloc(slab, p, sizeclass);
else
remote_dealloc(target, p, sizeclass);
@@ -262,7 +267,7 @@ namespace snmalloc
SNMALLOC_SLOW_PATH void dealloc_sized_slow(void* p, size_t size)
{
if (size == 0)
dealloc(p, 1);
return dealloc(p, 1);
if (likely(size <= sizeclass_to_size(NUM_SIZECLASSES - 1)))
{
@@ -505,8 +510,12 @@ namespace snmalloc
/// r is used for which round of sending this is.
inline size_t get_slot(size_t id, size_t r)
{
constexpr size_t allocator_size = sizeof(
Allocator<MemoryProvider, ChunkMap, IsQueueInline, Replacement>);
constexpr size_t allocator_size = sizeof(Allocator<
NeedsInitialisation,
InitThreadAllocator,
MemoryProvider,
ChunkMap,
IsQueueInline>);
constexpr size_t initial_shift =
bits::next_pow2_bits_const(allocator_size);
SNMALLOC_ASSERT((initial_shift + (r * REMOTE_SLOT_BITS)) < 64);
@@ -703,7 +712,7 @@ namespace snmalloc
*
* If result pointer is null, then this code raises a Pal::error on the
* particular check that fails, if any do fail.
**/
*/
void debug_is_empty(bool* result)
{
auto test = [&result](auto& queue) {
@@ -728,6 +737,26 @@ namespace snmalloc
}
}
// Dump bump allocators back into memory
for (size_t i = 0; i < NUM_SMALL_CLASSES; i++)
{
auto& bp = bump_ptrs[i];
auto rsize = sizeclass_to_size(i);
FreeListHead ffl;
while (pointer_align_up(bp, SLAB_SIZE) != bp)
{
Slab::alloc_new_list(bp, ffl, rsize);
void* prev = ffl.value;
while (prev != nullptr)
{
auto n = Metaslab::follow_next(prev);
Superslab* super = Superslab::get(prev);
small_dealloc_offseted_inner(super, prev, i);
prev = n;
}
}
}
for (size_t i = 0; i < NUM_SMALL_CLASSES; i++)
{
auto prev = small_fast_free_lists[i].value;
@@ -856,10 +885,19 @@ namespace snmalloc
remote.post(id());
}
/**
* Check if this allocator has messages to deallocate blocks from another
* thread
*/
SNMALLOC_FAST_PATH bool has_messages()
{
return !(message_queue().is_empty());
}
SNMALLOC_FAST_PATH void handle_message_queue()
{
// Inline the empty check, but not necessarily the full queue handling.
if (likely(message_queue().is_empty()))
if (likely(!has_messages()))
return;
handle_message_queue_inner();
@@ -921,7 +959,7 @@ namespace snmalloc
}
template<AllowReserve allow_reserve>
Slab* alloc_slab(sizeclass_t sizeclass)
SNMALLOC_SLOW_PATH Slab* alloc_slab(sizeclass_t sizeclass)
{
stats().sizeclass_alloc_slab(sizeclass);
if (Superslab::is_short_sizeclass(sizeclass))
@@ -970,17 +1008,19 @@ namespace snmalloc
SNMALLOC_ASSUME(size <= SLAB_SIZE);
sizeclass_t sizeclass = size_to_sizeclass(size);
return small_alloc_inner<zero_mem, allow_reserve>(sizeclass);
return small_alloc_inner<zero_mem, allow_reserve>(sizeclass, size);
}
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_FAST_PATH void* small_alloc_inner(sizeclass_t sizeclass)
SNMALLOC_FAST_PATH void*
small_alloc_inner(sizeclass_t sizeclass, size_t size)
{
SNMALLOC_ASSUME(sizeclass < NUM_SMALL_CLASSES);
auto& fl = small_fast_free_lists[sizeclass];
void* head = fl.value;
if (likely(head != nullptr))
{
stats().alloc_request(size);
stats().sizeclass_alloc(sizeclass);
// Read the next slot from the memory that's about to be allocated.
fl.value = Metaslab::follow_next(head);
@@ -993,46 +1033,140 @@ namespace snmalloc
return p;
}
return small_alloc_slow<zero_mem, allow_reserve>(sizeclass);
if (likely(!has_messages()))
return small_alloc_next_free_list<zero_mem, allow_reserve>(
sizeclass, size);
return small_alloc_mq_slow<zero_mem, allow_reserve>(sizeclass, size);
}
/**
* Slow path for handling message queue, before dealing with small
* allocation request.
*/
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_SLOW_PATH void* small_alloc_slow(sizeclass_t sizeclass)
SNMALLOC_SLOW_PATH void*
small_alloc_mq_slow(sizeclass_t sizeclass, size_t size)
{
if (void* replacement = Replacement(this))
{
return reinterpret_cast<Allocator*>(replacement)
->template small_alloc_inner<zero_mem, allow_reserve>(sizeclass);
}
handle_message_queue_inner();
stats().sizeclass_alloc(sizeclass);
return small_alloc_next_free_list<zero_mem, allow_reserve>(
sizeclass, size);
}
handle_message_queue();
/**
* Attempt to find a new free list to allocate from
*/
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_SLOW_PATH void*
small_alloc_next_free_list(sizeclass_t sizeclass, size_t size)
{
size_t rsize = sizeclass_to_size(sizeclass);
auto& sl = small_classes[sizeclass];
Slab* slab;
if (!sl.is_empty())
if (likely(!sl.is_empty()))
{
SlabLink* link = sl.get_head();
slab = link->get_slab();
stats().alloc_request(size);
stats().sizeclass_alloc(sizeclass);
SlabLink* link = sl.get_next();
slab = get_slab(link);
auto& ffl = small_fast_free_lists[sizeclass];
return slab->alloc<zero_mem>(
sl, ffl, rsize, large_allocator.memory_provider);
}
else
return small_alloc_rare<zero_mem, allow_reserve>(sizeclass, size);
}
/**
* Called when there are no available free list to service this request
* Could be due to using the dummy allocator, or needing to bump allocate a
* new free list.
*/
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_SLOW_PATH void*
small_alloc_rare(sizeclass_t sizeclass, size_t size)
{
if (likely(!NeedsInitialisation(this)))
{
slab = alloc_slab<allow_reserve>(sizeclass);
if ((allow_reserve == NoReserve) && (slab == nullptr))
return nullptr;
if (slab == nullptr)
return nullptr;
sl.insert_back(slab->get_link());
stats().alloc_request(size);
stats().sizeclass_alloc(sizeclass);
return small_alloc_new_free_list<zero_mem, allow_reserve>(sizeclass);
}
return small_alloc_first_alloc<zero_mem, allow_reserve>(sizeclass, size);
}
/**
* Called on first allocation to set up the thread local allocator,
* then directs the allocation request to the newly created allocator.
*/
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_SLOW_PATH void*
small_alloc_first_alloc(sizeclass_t sizeclass, size_t size)
{
auto replacement = InitThreadAllocator();
return reinterpret_cast<Allocator*>(replacement)
->template small_alloc_inner<zero_mem, allow_reserve>(sizeclass, size);
}
/**
* Called to create a new free list, and service the request from that new
* list.
*/
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_FAST_PATH void* small_alloc_new_free_list(sizeclass_t sizeclass)
{
auto& bp = bump_ptrs[sizeclass];
if (likely(pointer_align_up(bp, SLAB_SIZE) != bp))
{
return small_alloc_build_free_list<zero_mem, allow_reserve>(sizeclass);
}
// Fetch new slab
return small_alloc_new_slab<zero_mem, allow_reserve>(sizeclass);
}
/**
* Creates a new free list from the thread local bump allocator and service
* the request from that new list.
*/
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_FAST_PATH void* small_alloc_build_free_list(sizeclass_t sizeclass)
{
auto& bp = bump_ptrs[sizeclass];
auto rsize = sizeclass_to_size(sizeclass);
auto& ffl = small_fast_free_lists[sizeclass];
return slab->alloc<zero_mem>(
sl, ffl, rsize, large_allocator.memory_provider);
SNMALLOC_ASSERT(ffl.value == nullptr);
Slab::alloc_new_list(bp, ffl, rsize);
void* p = remove_cache_friendly_offset(ffl.value, sizeclass);
ffl.value = Metaslab::follow_next(p);
if constexpr (zero_mem == YesZero)
{
large_allocator.memory_provider.zero(p, sizeclass_to_size(sizeclass));
}
return p;
}
/**
* Allocates a new slab to allocate from, set it to be the bump allocator
* for this size class, and then builds a new free list from the thread
* local bump allocator and service the request from that new list.
*/
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_SLOW_PATH void* small_alloc_new_slab(sizeclass_t sizeclass)
{
auto& bp = bump_ptrs[sizeclass];
// Fetch new slab
Slab* slab = alloc_slab<allow_reserve>(sizeclass);
if (slab == nullptr)
return nullptr;
bp =
pointer_offset(slab, get_initial_offset(sizeclass, slab->is_short()));
return small_alloc_build_free_list<zero_mem, allow_reserve>(sizeclass);
}
SNMALLOC_FAST_PATH void
@@ -1149,8 +1283,9 @@ namespace snmalloc
}
else
{
if (void* replacement = Replacement(this))
if (NeedsInitialisation(this))
{
void* replacement = InitThreadAllocator();
return reinterpret_cast<Allocator*>(replacement)
->template medium_alloc<zero_mem, allow_reserve>(
sizeclass, rsize, size);
@@ -1170,6 +1305,7 @@ namespace snmalloc
sc->insert(slab);
}
stats().alloc_request(size);
stats().sizeclass_alloc(sizeclass);
return p;
}
@@ -1221,8 +1357,9 @@ namespace snmalloc
zero_mem == YesZero ? "zeromem" : "nozeromem",
allow_reserve == NoReserve ? "noreserve" : "reserve"));
if (void* replacement = Replacement(this))
if (NeedsInitialisation(this))
{
void* replacement = InitThreadAllocator();
return reinterpret_cast<Allocator*>(replacement)
->template large_alloc<zero_mem, allow_reserve>(size);
}
@@ -1237,6 +1374,7 @@ namespace snmalloc
{
chunkmap().set_large_size(p, size);
stats().alloc_request(size);
stats().large_alloc(large_class);
}
return p;
@@ -1246,6 +1384,13 @@ namespace snmalloc
{
MEASURE_TIME(large_dealloc, 4, 16);
if (NeedsInitialisation(this))
{
void* replacement = InitThreadAllocator();
return reinterpret_cast<Allocator*>(replacement)
->large_dealloc(p, size);
}
size_t size_bits = bits::next_pow2_bits(size);
SNMALLOC_ASSERT(bits::one_at_bit(size_bits) >= SUPERSLAB_SIZE);
size_t large_class = size_bits - SUPERSLAB_BITS;
@@ -1260,11 +1405,10 @@ namespace snmalloc
large_allocator.dealloc(slab, large_class);
}
// Note that this is on the slow path as it lead to better code.
// As it is tail, not inlining means that it is jumped to, so has no perf
// impact on the producer consumer scenarios, and doesn't require register
// spills in the fast path for local deallocation.
SNMALLOC_SLOW_PATH
// This is still considered the fast path as all the complex code is tail
// called in its slow path. This leads to one fewer unconditional jump in
// Clang.
SNMALLOC_FAST_PATH
void remote_dealloc(RemoteAllocator* target, void* p, sizeclass_t sizeclass)
{
MEASURE_TIME(remote_dealloc, 4, 16);
@@ -1293,8 +1437,9 @@ namespace snmalloc
// Now that we've established that we're in the slow path (if we're a
// real allocator, we will have to empty our cache now), check if we are
// a real allocator and construct one if we aren't.
if (void* replacement = Replacement(this))
if (NeedsInitialisation(this))
{
void* replacement = InitThreadAllocator();
// We have to do a dealloc, not a remote_dealloc here because this may
// have been allocated with the allocator that we've just had returned.
reinterpret_cast<Allocator*>(replacement)->dealloc(p);