Cheri-research/snmalloc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Major refactor of snmalloc (#343)

Browse Source 
# Pagemap
 
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.

This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.


# Backend

There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.

# Thread local state

The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.

# Difference

This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
This commit is contained in:
Matthew Parkinson
2021-07-12 15:53:36 +01:00
committed by GitHub
parent 18d7cc99b6
commit f0e2ab702a
83 changed files with 4404 additions and 5769 deletions
Download Patch File Download Diff File Expand all files Collapse all files

334
src/mem/threadalloc.h
View File

@@ -1,12 +1,32 @@
#pragma once
#include "../ds/helpers.h"
#include "globalalloc.h"
#if defined(SNMALLOC_USE_THREAD_DESTRUCTOR) && \
defined(SNMALLOC_USE_THREAD_CLEANUP)
#error At most one out of SNMALLOC_USE_THREAD_CLEANUP and SNMALLOC_USE_THREAD_DESTRUCTOR may be defined.
#include "globalconfig.h"
#include "localalloc.h"
#if defined(SNMALLOC_EXTERNAL_THREAD_ALLOC)
# define SNMALLOC_THREAD_TEARDOWN_DEFINED
#endif
#if defined(SNMALLOC_USE_THREAD_CLEANUP)
# if defined(SNMALLOC_THREAD_TEARDOWN_DEFINED)
# error At most one out of method of thread teardown can be specified.
# else
# define SNMALLOC_THREAD_TEARDOWN_DEFINED
# endif
#endif
#if defined(SNMALLOC_USE_PTHREAD_DESTRUCTORS)
# if defined(SNMALLOC_THREAD_TEARDOWN_DEFINED)
# error At most one out of method of thread teardown can be specified.
# else
# define SNMALLOC_THREAD_TEARDOWN_DEFINED
# endif
#endif
#if !defined(SNMALLOC_THREAD_TEARDOWN_DEFINED)
# define SNMALLOC_USE_CXX_THREAD_DESTRUCTORS
#endif
extern "C" void _malloc_thread_cleanup();
namespace snmalloc
@@ -14,42 +34,23 @@ namespace snmalloc
#ifdef SNMALLOC_EXTERNAL_THREAD_ALLOC
/**
* Version of the `ThreadAlloc` interface that does no management of thread
* local state, and just assumes that "ThreadAllocUntyped::get" has been
* declared before including snmalloc.h. As it is included before, it cannot
* know the allocator type, hence the casting.
* local state.
*
* This class is used only when snmalloc is compiled as part of a runtime,
* which has its own management of the thread local allocator pointer.
* It assumes that Alloc has been defined, and `ThreadAllocExternal` class
* has access to snmalloc_core.h.
*/
class ThreadAllocUntypedWrapper
class ThreadAlloc
{
protected:
static void register_cleanup() {}
public:
static SNMALLOC_FAST_PATH Alloc* get_noncachable()
static SNMALLOC_FAST_PATH Alloc& get()
{
return (Alloc*)ThreadAllocUntyped::get();
}
static SNMALLOC_FAST_PATH Alloc* get()
{
return (Alloc*)ThreadAllocUntyped::get();
return ThreadAllocExternal::get();
}
};
/**
* Function passed as a template parameter to `Allocator` to allow lazy
* replacement. This function returns true, if the allocator passed in
* requires initialisation. As the TLS state is managed externally,
* this will always return false.
*/
SNMALLOC_FAST_PATH bool needs_initialisation(void* existing)
{
UNUSED(existing);
return false;
}
/**
* Function passed as a template parameter to `Allocator` to allow lazy
* replacement. There is nothing to initialise in this case, so we expect
@@ -62,247 +63,100 @@ namespace snmalloc
# pragma warning(push)
# pragma warning(disable : 4702)
# endif
SNMALLOC_FAST_PATH void* init_thread_allocator(function_ref<void*(void*)> f)
inline void register_clean_up()
{
error("Critical Error: This should never be called.");
return f(nullptr);
}
# ifdef _MSC_VER
# pragma warning(pop)
# endif
using ThreadAlloc = ThreadAllocUntypedWrapper;
#else
/**
* A global fake allocator object. This never allocates memory and, as a
* result, never owns any slabs. On the slow paths, where it would fetch
* slabs to allocate from, it will discover that it is the placeholder and
* replace itself with the thread-local allocator, allocating one if
* required. This avoids a branch on the fast path.
*
* The fake allocator is a zero initialised area of memory of the correct
* size. All data structures used potentially before initialisation must be
* okay with zero init to move to the slow path, that is, zero must signify
* empty.
*/
inline const char GlobalPlaceHolder[sizeof(Alloc)] = {0};
inline Alloc* get_GlobalPlaceHolder()
{
// This cast is not legal. Effectively, we want a minimal constructor
// for the global allocator as zero, and then a second constructor for
// the rest. This is UB.
auto a = reinterpret_cast<const Alloc*>(&GlobalPlaceHolder);
return const_cast<Alloc*>(a);
}
/**
* Common aspects of thread local allocator. Subclasses handle how releasing
* the allocator is triggered.
* Holds the thread local state for the allocator. The state is constant
* initialised, and has no direct dectructor. Instead snmalloc will call
* `register_clean_up` on the slow path for bringing up thread local state.
* This is responsible for calling `teardown`, which effectively destructs the
* data structure, but in a way that allow it to still be used.
*/
class ThreadAllocCommon
class ThreadAlloc
{
friend void* init_thread_allocator(function_ref<void*(void*)>);
protected:
/**
* Thread local variable that is set to true, once `inner_release`
* has been run. If we try to reinitialise the allocator once
* `inner_release` has run, then we can stay on the slow path so we don't
* leak allocators.
*
* This is required to allow for the allocator to be called during
* destructors of other thread_local state.
*/
inline static thread_local bool destructor_has_run = false;
static inline void inner_release()
{
auto& per_thread = get_reference();
if (per_thread != get_GlobalPlaceHolder())
{
current_alloc_pool()->release(per_thread);
destructor_has_run = true;
per_thread = get_GlobalPlaceHolder();
}
}
/**
* Default clean up does nothing except print statistics if enabled.
*/
static bool register_cleanup()
{
# ifdef USE_SNMALLOC_STATS
Singleton<int, atexit_print_stats>::get();
# endif
return false;
}
# ifdef USE_SNMALLOC_STATS
static void print_stats()
{
Stats s;
current_alloc_pool()->aggregate_stats(s);
s.print<Alloc>(std::cout);
}
static int atexit_print_stats() noexcept
{
return atexit(print_stats);
}
# endif
public:
/**
* Returns a reference to the allocator for the current thread. This allows
* the caller to replace the current thread's allocator.
*/
static inline Alloc*& get_reference()
{
// Inline casting as codegen doesn't create a lazy init like this.
static thread_local Alloc* alloc =
const_cast<Alloc*>(reinterpret_cast<const Alloc*>(&GlobalPlaceHolder));
return alloc;
}
/**
* Public interface, returns the allocator for this thread, constructing
* one if necessary.
* Handle on thread local allocator
*
* If no operations have been performed on an allocator returned by either
* `get()` nor `get_noncachable()`, then the value contained in the return
* will be an Alloc* that will always use the slow path.
*
* Only use this API if you intend to use the returned allocator just once
* per call, or if you know other calls have already been made to the
* allocator.
* This structure will self initialise if it has not been called yet.
* It can be used during thread teardown, but its performance will be
* less good.
*/
static inline Alloc* get_noncachable()
static SNMALLOC_FAST_PATH Alloc& get()
{
return get_reference();
}
/**
* Public interface, returns the allocator for this thread, constructing
* one if necessary.
* This incurs a cost, so use `get_noncachable` if you can meet its
* criteria.
*/
static SNMALLOC_FAST_PATH Alloc* get()
{
# ifdef SNMALLOC_PASS_THROUGH
return get_reference();
# else
auto*& alloc = get_reference();
if (unlikely(needs_initialisation(alloc)) && !destructor_has_run)
{
// Call `init_thread_allocator` to perform down call in case
// register_clean_up does more.
// During teardown for the destructor based ThreadAlloc this will set
// alloc to GlobalPlaceHolder;
init_thread_allocator([](void*) { return nullptr; });
}
SNMALLOC_REQUIRE_CONSTINIT static thread_local Alloc alloc;
return alloc;
# endif
}
};
# ifdef SNMALLOC_USE_PTHREAD_DESTRUCTOR
/**
* Version of the `ThreadAlloc` interface that uses a hook provided by libc
* to destroy thread-local state. This is the ideal option, because it
* enforces ordering of destruction such that the malloc state is destroyed
* after anything that can allocate memory.
*
* This class is used only when snmalloc is compiled as part of a compatible
* libc (for example, FreeBSD libc).
* Used to give correct signature to teardown required by pthread_key.
*/
class ThreadAllocLibcCleanup : public ThreadAllocCommon
inline void pthread_cleanup(void*)
{
/**
* Libc will call `_malloc_thread_cleanup` just before a thread terminates.
* This function must be allowed to call back into this class to destroy
* the state.
*/
friend void ::_malloc_thread_cleanup();
};
ThreadAlloc::get().teardown();
}
/**
* Version of the `ThreadAlloc` interface that uses C++ `thread_local`
* destructors for cleanup. If a per-thread allocator is used during the
* destruction of other per-thread data, this class will create a new
* instance and register its destructor, so should eventually result in
* cleanup, but may result in allocators being returned to the global pool
* and then reacquired multiple times.
* Used to give correct signature to the pthread call for the Singleton class.
*/
inline pthread_key_t pthread_create() noexcept
{
pthread_key_t key;
pthread_key_create(&key, &pthread_cleanup);
return key;
}
/**
* Performs thread local teardown for the allocator using the pthread library.
*
* This removes the dependence on the C++ runtime.
*/
inline void register_clean_up()
{
Singleton<pthread_key_t, &pthread_create> p_key;
// We need to set a non-null value, so that the destructor is called,
// we never look at the value.
pthread_setspecific(p_key.get(), reinterpret_cast<void*>(1));
# ifdef SNMALLOC_TRACING
std::cout << "Using pthread clean up" << std::endl;
# endif
}
# elif defined(SNMALLOC_USE_CXX_THREAD_DESTRUCTORS)
/**
* This function is called by each thread once it starts using the
* thread local allocator.
*
* This implementation depends on nothing outside of a working C++
* environment and so should be the simplest for initial bringup on an
* unsupported platform. It is currently used in the FreeBSD kernel version.
* unsupported platform.
*/
class ThreadAllocThreadDestructor : public ThreadAllocCommon
inline void register_clean_up()
{
template<void f()>
friend class OnDestruct;
public:
static bool register_cleanup()
{
static thread_local OnDestruct<ThreadAllocCommon::inner_release> tidier;
ThreadAllocCommon::register_cleanup();
return destructor_has_run;
}
};
# ifdef SNMALLOC_USE_THREAD_CLEANUP
/**
* Entry point that allows libc to call into the allocator for per-thread
* cleanup.
*/
extern "C" void _malloc_thread_cleanup()
{
ThreadAllocLibcCleanup::inner_release();
static thread_local OnDestruct dummy(
[]() { ThreadAlloc::get().teardown(); });
UNUSED(dummy);
# ifdef SNMALLOC_TRACING
std::cout << "Using C++ destructor clean up" << std::endl;
# endif
}
using ThreadAlloc = ThreadAllocLibcCleanup;
# else
using ThreadAlloc = ThreadAllocThreadDestructor;
# endif
/**
* Slow path for the placeholder replacement.
* Function passed as a tempalte parameter to `Allocator` to allow lazy
* replacement. This function initialises the thread local state if requried.
* The simple check that this is the global placeholder is inlined, the rest
* of it is only hit in a very unusual case and so should go off the fast
* path.
* The second component of the return indicates if this TLS is being torndown.
*/
SNMALLOC_FAST_PATH void* init_thread_allocator(function_ref<void*(void*)> f)
{
auto*& local_alloc = ThreadAlloc::get_reference();
// If someone reuses a noncachable call, then we can end up here
// with an already initialised allocator. Could either error
// to say stop doing this, or just give them the initialised version.
if (local_alloc == get_GlobalPlaceHolder())
{
local_alloc = current_alloc_pool()->acquire();
}
auto result = f(local_alloc);
// Check if we have already run the destructor for the TLS. If so,
// we need to deallocate the allocator.
if (ThreadAlloc::register_cleanup())
ThreadAlloc::inner_release();
return result;
}
/**
* Function passed as a template parameter to `Allocator` to allow lazy
* replacement. This function returns true, if the allocated passed in,
* is the placeholder allocator. If it returns true, then
* `init_thread_allocator` should be called.
*/
SNMALLOC_FAST_PATH bool needs_initialisation(void* existing)
{
return existing == get_GlobalPlaceHolder();
}
#endif
} // namespace snmalloc
#ifdef SNMALLOC_USE_THREAD_CLEANUP
/**
* Entry point that allows libc to call into the allocator for per-thread
* cleanup.
*/
void _malloc_thread_cleanup()
{
ThreadAlloc::get().teardown();
}
#endif