Cheri-research/snmalloc
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Defensive code for alloc/dealloc during TLS teardown (#161)

Browse Source 
* Defensive code for alloc/dealloc during TLS teardown

If an allocation or deallocation occurs during TLS teardown, then it is
possible for a new allocator to be created and then this is leaked. On
the mimalloc-bench mstressN benchmark this was observed leading to a
large memory leak.

This fix, detects if we are in the TLS teardown phase, and if so,
the calls to alloc or dealloc must return the allocator once they have
perform the specific operation.

Uses a separate variable to represent if a thread_local's destructor has
run already.  This is used to detect thread teardown to put the
allocator into a special slow path to avoid leaks.

* Added some printing first operation to track progress

* Improve error messages on posix

Flush errors, print assert details, and present stack traces.

* Detect incorrect use of pool.

* Clang format.

* Replace broken LL/SC implementation

LL/SC implementation was broken, this replaces it with
a locking implementation. Changes the API to support LL/SC
for future implementation on ARM.

* Improve TLS teardown.

* Make std::function fully inlined.

* Factor out PALLinux stack trace.

* Add checks for leaking allocators.

* Add release build of Windows Clang
This commit is contained in:
Matthew Parkinson
2020-04-07 15:37:26 +01:00
committed by GitHub
parent d87888096e
commit 74657d9dbc
13 changed files with 359 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
src/mem/alloc.h
View File

@@ -17,6 +17,8 @@
#include "sizeclasstable.h"
#include "slab.h"
#include <functional>
namespace snmalloc
{
enum Boundary
@@ -60,7 +62,10 @@ namespace snmalloc
* passed the global allocator. The second initialises the thread-local
* allocator if it is has been been initialised already. Splitting into two
* functions allows for the code to be structured into tail calls to improve
* codegen.
* codegen. The second template takes a function that takes the allocator
* that is initialised, and the value returned, is returned by
* `InitThreadAllocator`. This is used incase we are running during teardown
* and the thread local allocator cannot be kept alive.
*
* The `MemoryProvider` defines the source of memory for this allocator.
* Allocators try to reuse address space by allocating from existing slabs or
@@ -78,7 +83,7 @@ namespace snmalloc
*/
template<
bool (*NeedsInitialisation)(void*),
void* (*InitThreadAllocator)(),
void* (*InitThreadAllocator)(std::function<void*(void*)>&),
class MemoryProvider = GlobalVirtual,
class ChunkMap = SNMALLOC_DEFAULT_CHUNKMAP,
bool IsQueueInline = true>
@@ -106,7 +111,7 @@ namespace snmalloc
return large_allocator.stats;
}
template<class MP>
template<class MP, class Alloc>
friend class AllocPool;
/**
@@ -1126,7 +1131,7 @@ namespace snmalloc
stats().sizeclass_alloc(sizeclass);
return small_alloc_new_free_list<zero_mem, allow_reserve>(sizeclass);
}
return small_alloc_first_alloc<zero_mem, allow_reserve>(sizeclass, size);
return small_alloc_first_alloc<zero_mem, allow_reserve>(size);
}
/**
@@ -1134,12 +1139,12 @@ namespace snmalloc
* then directs the allocation request to the newly created allocator.
*/
template<ZeroMem zero_mem, AllowReserve allow_reserve>
SNMALLOC_SLOW_PATH void*
small_alloc_first_alloc(sizeclass_t sizeclass, size_t size)
SNMALLOC_SLOW_PATH void* small_alloc_first_alloc(size_t size)
{
auto replacement = InitThreadAllocator();
return reinterpret_cast<Allocator*>(replacement)
->template small_alloc_inner<zero_mem, allow_reserve>(sizeclass, size);
std::function<void*(void*)> f = [size](void* alloc) {
return reinterpret_cast<Allocator*>(alloc)->alloc(size);
};
return InitThreadAllocator(f);
}
/**
@@ -1316,10 +1321,10 @@ namespace snmalloc
{
if (NeedsInitialisation(this))
{
void* replacement = InitThreadAllocator();
return reinterpret_cast<Allocator*>(replacement)
->template medium_alloc<zero_mem, allow_reserve>(
sizeclass, rsize, size);
std::function<void*(void*)> f = [size](void* alloc) {
return reinterpret_cast<Allocator*>(alloc)->alloc(size);
};
return InitThreadAllocator(f);
}
slab = reinterpret_cast<Mediumslab*>(
large_allocator.template alloc<NoZero, allow_reserve>(
@@ -1390,9 +1395,10 @@ namespace snmalloc
if (NeedsInitialisation(this))
{
void* replacement = InitThreadAllocator();
return reinterpret_cast<Allocator*>(replacement)
->template large_alloc<zero_mem, allow_reserve>(size);
std::function<void*(void*)> f = [size](void* alloc) {
return reinterpret_cast<Allocator*>(alloc)->alloc(size);
};
return InitThreadAllocator(f);
}
size_t size_bits = bits::next_pow2_bits(size);
@@ -1417,9 +1423,12 @@ namespace snmalloc
if (NeedsInitialisation(this))
{
void* replacement = InitThreadAllocator();
return reinterpret_cast<Allocator*>(replacement)
->large_dealloc(p, size);
std::function<void*(void*)> f = [p](void* alloc) {
reinterpret_cast<Allocator*>(alloc)->dealloc(p);
return nullptr;
};
InitThreadAllocator(f);
return;
}
size_t size_bits = bits::next_pow2_bits(size);
@@ -1470,10 +1479,11 @@ namespace snmalloc
// a real allocator and construct one if we aren't.
if (NeedsInitialisation(this))
{
void* replacement = InitThreadAllocator();
// We have to do a dealloc, not a remote_dealloc here because this may
// have been allocated with the allocator that we've just had returned.
reinterpret_cast<Allocator*>(replacement)->dealloc(p);
std::function<void*(void*)> f = [p](void* alloc) {
reinterpret_cast<Allocator*>(alloc)->dealloc(p);
return nullptr;
};
InitThreadAllocator(f);
return;
}

67
src/mem/globalalloc.h
View File

@@ -7,31 +7,11 @@
namespace snmalloc
{
inline bool needs_initialisation(void*);
void* init_thread_allocator();
void* init_thread_allocator(std::function<void*(void*)>&);
using Alloc = Allocator<
needs_initialisation,
init_thread_allocator,
GlobalVirtual,
SNMALLOC_DEFAULT_CHUNKMAP,
true>;
template<class MemoryProvider>
class AllocPool : Pool<
Allocator<
needs_initialisation,
init_thread_allocator,
MemoryProvider,
SNMALLOC_DEFAULT_CHUNKMAP,
true>,
MemoryProvider>
template<class MemoryProvider, class Alloc>
class AllocPool : Pool<Alloc, MemoryProvider>
{
using Alloc = Allocator<
needs_initialisation,
init_thread_allocator,
MemoryProvider,
SNMALLOC_DEFAULT_CHUNKMAP,
true>;
using Parent = Pool<Alloc, MemoryProvider>;
public:
@@ -173,19 +153,48 @@ namespace snmalloc
UNUSED(result);
#endif
}
void debug_in_use(size_t count)
{
auto alloc = Parent::iterate();
while (alloc != nullptr)
{
if (alloc->debug_is_in_use())
{
if (count == 0)
{
error("ERROR: allocator in use.");
}
count--;
}
alloc = Parent::iterate(alloc);
if (count != 0)
{
error("Error: two few allocators in use.");
}
}
}
};
inline AllocPool<GlobalVirtual>*& current_alloc_pool()
using Alloc = Allocator<
needs_initialisation,
init_thread_allocator,
GlobalVirtual,
SNMALLOC_DEFAULT_CHUNKMAP,
true>;
inline AllocPool<GlobalVirtual, Alloc>*& current_alloc_pool()
{
return Singleton<
AllocPool<GlobalVirtual>*,
AllocPool<GlobalVirtual>::make>::get();
AllocPool<GlobalVirtual, Alloc>*,
AllocPool<GlobalVirtual, Alloc>::make>::get();
}
template<class MemoryProvider>
inline AllocPool<MemoryProvider>* make_alloc_pool(MemoryProvider& mp)
template<class MemoryProvider, class Alloc>
inline AllocPool<MemoryProvider, Alloc>* make_alloc_pool(MemoryProvider& mp)
{
return AllocPool<MemoryProvider>::make(mp);
return AllocPool<MemoryProvider, Alloc>::make(mp);
}
} // namespace snmalloc

15
src/mem/pool.h
View File

@@ -50,7 +50,10 @@ namespace snmalloc
T* p = stack.pop();
if (p != nullptr)
{
p->set_in_use();
return p;
}
p = memory_provider
.template alloc_chunk<T, bits::next_pow2_const(sizeof(T))>(
@@ -60,14 +63,21 @@ namespace snmalloc
p->list_next = list;
list = p;
p->set_in_use();
return p;
}
/**
* Return to the pool an object previously retrieved by `acquire`
*
* Do not return objects from `extract`.
*/
void release(T* p)
{
// The object's destructor is not run. If the object is "reallocated", it
// is returned without the constructor being run, so the object is reused
// without re-initialisation.
p->reset_in_use();
stack.push(p);
}
@@ -80,6 +90,11 @@ namespace snmalloc
return p->next;
}
/**
* Return to the pool a list of object previously retrieved by `extract`
*
* Do not return objects from `acquire`.
*/
void restore(T* first, T* last)
{
// Pushes a linked list of objects onto the stack. Use to put a linked

21
src/mem/pooled.h
View File

@@ -17,5 +17,26 @@ namespace snmalloc
std::atomic<T*> next = nullptr;
/// Used by the pool to keep the list of all entries ever created.
T* list_next;
std::atomic_flag in_use = ATOMIC_FLAG_INIT;
public:
void set_in_use()
{
if (in_use.test_and_set())
error("Critical error: double use of Pooled Type!");
}
void reset_in_use()
{
in_use.clear();
}
bool debug_is_in_use()
{
bool result = in_use.test_and_set();
if (!result)
in_use.clear();
return result;
}
};
} // namespace snmalloc

81
src/mem/threadalloc.h
View File

@@ -51,14 +51,25 @@ namespace snmalloc
}
/**
* Function passed as a tempalte parameter to `Allocator` to allow lazy
* Function passed as a template parameter to `Allocator` to allow lazy
* replacement. There is nothing to initialise in this case, so we expect
* this to never be called.
*/
SNMALLOC_FAST_PATH void* init_thread_allocator()
# ifdef _MSC_VER
// 32Bit Windows release MSVC is determining this as having unreachable code for
// f(nullptr), which is true. But other platforms don't. Disabling the warning
// seems simplist.
# pragma warning(push)
# pragma warning(disable : 4702)
# endif
SNMALLOC_FAST_PATH void* init_thread_allocator(std::function<void*(void*)>& f)
{
return nullptr;
error("Critical Error: This should never be called.");
return f(nullptr);
}
# ifdef _MSC_VER
# pragma warning(pop)
# endif
using ThreadAlloc = ThreadAllocUntypedWrapper;
#else
@@ -90,15 +101,27 @@ namespace snmalloc
*/
class ThreadAllocCommon
{
friend void* init_thread_allocator();
friend void* init_thread_allocator(std::function<void*(void*)>&);
protected:
/**
* Thread local variable that is set to true, once `inner_release`
* has been run. If we try to reinitialise the allocator once
* `inner_release` has run, then we can stay on the slow path so we don't
* leak allocators.
*
* This is required to allow for the allocator to be called during
* destructors of other thread_local state.
*/
inline static thread_local bool destructor_has_run = false;
static inline void inner_release()
{
auto& per_thread = get_reference();
if (per_thread != get_GlobalPlaceHolder())
{
current_alloc_pool()->release(per_thread);
destructor_has_run = true;
per_thread = get_GlobalPlaceHolder();
}
}
@@ -106,11 +129,12 @@ namespace snmalloc
/**
* Default clean up does nothing except print statistics if enabled.
*/
static void register_cleanup()
static bool register_cleanup()
{
# ifdef USE_SNMALLOC_STATS
Singleton<int, atexit_print_stats>::get();
# endif
return false;
}
# ifdef USE_SNMALLOC_STATS
@@ -134,7 +158,9 @@ namespace snmalloc
*/
static inline Alloc*& get_reference()
{
static thread_local Alloc* alloc = get_GlobalPlaceHolder();
// Inline casting as codegen doesn't create a lazy init like this.
static thread_local Alloc* alloc =
const_cast<Alloc*>(reinterpret_cast<const Alloc*>(&GlobalPlaceHolder));
return alloc;
}
@@ -158,19 +184,23 @@ namespace snmalloc
/**
* Public interface, returns the allocator for this thread, constructing
* one if necessary.
*
* The returned Alloc* is guaranteed to be initialised. This incurs a cost,
* so use `get_noncachable` if you can meet its criteria.
* This incurs a cost, so use `get_noncachable` if you can meet its
* criteria.
*/
static SNMALLOC_FAST_PATH Alloc* get()
{
# ifdef USE_MALLOC
return get_reference();
# else
auto alloc = get_reference();
if (unlikely(needs_initialisation(alloc)))
auto*& alloc = get_reference();
if (unlikely(needs_initialisation(alloc)) && !destructor_has_run)
{
alloc = reinterpret_cast<Alloc*>(init_thread_allocator());
std::function<void*(void*)> f = [](void*) { return nullptr; };
// Call `init_thread_allocator` to perform down call in case
// register_clean_up does more.
// During teardown for the destructor based ThreadAlloc this will set
// alloc to GlobalPlaceHolder;
init_thread_allocator(f);
}
return alloc;
# endif
@@ -214,11 +244,13 @@ namespace snmalloc
friend class OnDestruct;
public:
static void register_cleanup()
static bool register_cleanup()
{
static thread_local OnDestruct<ThreadAllocCommon::inner_release> tidier;
ThreadAllocCommon::register_cleanup();
return destructor_has_run;
}
};
@@ -243,21 +275,24 @@ namespace snmalloc
* The simple check that this is the global placeholder is inlined, the rest
* of it is only hit in a very unusual case and so should go off the fast
* path.
* The second component of the return indicates if this TLS is being torndown.
*/
SNMALLOC_SLOW_PATH inline void* init_thread_allocator()
SNMALLOC_FAST_PATH void* init_thread_allocator(std::function<void*(void*)>& f)
{
auto*& local_alloc = ThreadAlloc::get_reference();
if (local_alloc != get_GlobalPlaceHolder())
// If someone reuses a noncachable call, then we can end up here
// with an already initialised allocator. Could either error
// to say stop doing this, or just give them the initialised version.
if (local_alloc == get_GlobalPlaceHolder())
{
// If someone reuses a noncachable call, then we can end up here.
// The allocator has already been initialised. Could either error
// to say stop doing this, or just give them the initialised version.
return local_alloc;
local_alloc = current_alloc_pool()->acquire();
}
local_alloc = current_alloc_pool()->acquire();
SNMALLOC_ASSERT(local_alloc != get_GlobalPlaceHolder());
ThreadAlloc::register_cleanup();
return local_alloc;
auto result = f(local_alloc);
// Check if we have already run the destructor for the TLS. If so,
// we need to deallocate the allocator.
if (ThreadAlloc::register_cleanup())
ThreadAlloc::inner_release();
return result;
}
/**