Cheri-research/snmalloc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Major refactor of snmalloc (#343)

Browse Source 
# Pagemap
 
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.

This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.


# Backend

There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.

# Thread local state

The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.

# Difference

This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
This commit is contained in:
Matthew Parkinson
2021-07-12 15:53:36 +01:00
committed by GitHub
parent 18d7cc99b6
commit f0e2ab702a
83 changed files with 4404 additions and 5769 deletions
Download Patch File Download Diff File Expand all files Collapse all files

207
src/backend/address_space.h Normal file
View File

@@ -0,0 +1,207 @@
#pragma once
#include "../ds/address.h"
#include "../ds/flaglock.h"
#include "../pal/pal.h"
#include "address_space_core.h"
#include <array>
#ifdef SNMALLOC_TRACING
# include <iostream>
#endif
namespace snmalloc
{
/**
* Implements a power of two allocator, where all blocks are aligned to the
* same power of two as their size. This is what snmalloc uses to get
* alignment of very large sizeclasses.
*
* It cannot unreserve memory, so this does not require the
* usual complexity of a buddy allocator.
*/
template<SNMALLOC_CONCEPT(ConceptPAL) PAL>
class AddressSpaceManager
{
AddressSpaceManagerCore core;
/**
* This is infrequently used code, a spin lock simplifies the code
* considerably, and should never be on the fast path.
*/
std::atomic_flag spin_lock = ATOMIC_FLAG_INIT;
public:
/**
* Returns a pointer to a block of memory of the supplied size.
* The block will be committed, if specified by the template parameter.
* The returned block is guaranteed to be aligened to the size.
*
* Only request 2^n sizes, and not less than a pointer.
*
* On StrictProvenance architectures, any underlying allocations made as
* part of satisfying the request will be registered with the provided
* arena_map for use in subsequent amplification.
*/
template<bool committed, bool align = true>
CapPtr<void, CBChunk> reserve(size_t size)
{
#ifdef SNMALLOC_TRACING
std::cout << "ASM reserve request:" << size << std::endl;
#endif
SNMALLOC_ASSERT(bits::is_pow2(size));
SNMALLOC_ASSERT(size >= sizeof(void*));
if constexpr ((align == false) && !pal_supports<NoAllocation, PAL>)
{
if constexpr (pal_supports<AlignedAllocation, PAL>)
{
// TODO wasting size here.
size = bits::max(size, PAL::minimum_alloc_size);
return CapPtr<void, CBChunk>(
PAL::template reserve_aligned<committed>(size));
}
else
{
auto [block, size2] = PAL::reserve_at_least(size);
// TODO wasting size here.
UNUSED(size2);
#ifdef SNMALLOC_TRACING
std::cout << "Unaligned alloc here:" << block << " (" << size2 << ")"
<< std::endl;
#endif
return CapPtr<void, CBChunk>(block);
}
}
else
{
/*
* For sufficiently large allocations with platforms that support
* aligned allocations and architectures that don't require
* StrictProvenance, try asking the platform first.
*/
if constexpr (
pal_supports<AlignedAllocation, PAL> &&
!aal_supports<StrictProvenance>)
{
if (size >= PAL::minimum_alloc_size)
return CapPtr<void, CBChunk>(
PAL::template reserve_aligned<committed>(size));
}
CapPtr<void, CBChunk> res;
{
FlagLock lock(spin_lock);
res = core.template reserve<PAL>(size);
if (res == nullptr)
{
// Allocation failed ask OS for more memory
CapPtr<void, CBChunk> block = nullptr;
size_t block_size = 0;
if constexpr (pal_supports<AlignedAllocation, PAL>)
{
/*
* We will have handled the case where size >=
* minimum_alloc_size above, so we are left to handle only small
* things here.
*/
block_size = PAL::minimum_alloc_size;
void* block_raw =
PAL::template reserve_aligned<false>(block_size);
// It's a bit of a lie to convert without applying bounds, but the
// platform will have bounded block for us and it's better that
// the rest of our internals expect CBChunk bounds.
block = CapPtr<void, CBChunk>(block_raw);
}
else if constexpr (!pal_supports<NoAllocation, PAL>)
{
// Need at least 2 times the space to guarantee alignment.
// Hold lock here as a race could cause additional requests to
// the PAL, and this could lead to suprious OOM. This is
// particularly bad if the PAL gives all the memory on first call.
auto block_and_size = PAL::reserve_at_least(size * 2);
block = CapPtr<void, CBChunk>(block_and_size.first);
block_size = block_and_size.second;
// Ensure block is pointer aligned.
if (
pointer_align_up(block, sizeof(void*)) != block ||
bits::align_up(block_size, sizeof(void*)) > block_size)
{
auto diff =
pointer_diff(block, pointer_align_up(block, sizeof(void*)));
block_size = block_size - diff;
block_size = bits::align_down(block_size, sizeof(void*));
}
}
if (block == nullptr)
{
return nullptr;
}
core.template add_range<PAL>(block, block_size);
// still holding lock so guaranteed to succeed.
res = core.template reserve<PAL>(size);
}
}
// Don't need lock while committing pages.
if constexpr (committed)
core.template commit_block<PAL>(res, size);
return res;
}
}
/**
* Aligns block to next power of 2 above size, and unused space at the end
* of the block is retained by the address space manager.
*
* This is useful for allowing the space required for alignment to be
* used, by smaller objects.
*/
template<bool committed>
CapPtr<void, CBChunk> reserve_with_left_over(size_t size)
{
SNMALLOC_ASSERT(size >= sizeof(void*));
size = bits::align_up(size, sizeof(void*));
size_t rsize = bits::next_pow2(size);
auto res = reserve<false>(rsize);
if (res != nullptr)
{
if (rsize > size)
{
FlagLock lock(spin_lock);
core.template add_range<PAL>(pointer_offset(res, size), rsize - size);
}
if constexpr (committed)
core.commit_block<PAL>(res, size);
}
return res;
}
/**
* Default constructor. An address-space manager constructed in this way
* does not own any memory at the start and will request any that it needs
* from the PAL.
*/
AddressSpaceManager() = default;
/**
* Add a range of memory to the address space.
* Divides blocks into power of two sizes with natural alignment
*/
void add_range(CapPtr<void, CBChunk> base, size_t length)
{
FlagLock lock(spin_lock);
core.add_range<PAL>(base, length);
}
};
} // namespace snmalloc

239
src/backend/address_space_core.h Normal file
View File

@@ -0,0 +1,239 @@
#pragma once
#include "../ds/address.h"
#include "../ds/flaglock.h"
#include "../pal/pal.h"
#include <array>
#ifdef SNMALLOC_TRACING
# include <iostream>
#endif
namespace snmalloc
{
/**
* TODO all comment in this file need revisiting. Core versus locking global
* version.
*
* Implements a power of two allocator, where all blocks are aligned to the
* same power of two as their size. This is what snmalloc uses to get
* alignment of very large sizeclasses.
*
* It cannot unreserve memory, so this does not require the
* usual complexity of a buddy allocator.
*/
class AddressSpaceManagerCore
{
/**
* Stores the blocks of address space
*
* The first level of array indexes based on power of two size.
*
* The first entry ranges[n][0] is just a pointer to an address range
* of size 2^n.
*
* The second entry ranges[n][1] is a pointer to a linked list of blocks
* of this size. The final block in the list is not committed, so we commit
* on pop for this corner case.
*
* Invariants
* ranges[n][1] != nullptr => ranges[n][0] != nullptr
*
* bits::BITS is used for simplicity, we do not use below the pointer size,
* and large entries will be unlikely to be supported by the platform.
*/
std::array<std::array<CapPtr<void, CBChunk>, 2>, bits::BITS> ranges = {};
/**
* Checks a block satisfies its invariant.
*/
inline void check_block(CapPtr<void, CBChunk> base, size_t align_bits)
{
SNMALLOC_ASSERT(
base == pointer_align_up(base, bits::one_at_bit(align_bits)));
// All blocks need to be bigger than a pointer.
SNMALLOC_ASSERT(bits::one_at_bit(align_bits) >= sizeof(void*));
UNUSED(base);
UNUSED(align_bits);
}
/**
* Adds a block to `ranges`.
*/
template<SNMALLOC_CONCEPT(ConceptPAL) PAL>
void add_block(size_t align_bits, CapPtr<void, CBChunk> base)
{
check_block(base, align_bits);
SNMALLOC_ASSERT(align_bits < 64);
if (ranges[align_bits][0] == nullptr)
{
// Prefer first slot if available.
ranges[align_bits][0] = base;
return;
}
if (ranges[align_bits][1] != nullptr)
{
#ifdef SNMALLOC_TRACING
std::cout << "Add range linking." << std::endl;
#endif
// Add to linked list.
commit_block<PAL>(base, sizeof(void*));
*(base.template as_static<CapPtr<void, CBChunk>>().unsafe_ptr()) =
ranges[align_bits][1];
check_block(ranges[align_bits][1], align_bits);
}
// Update head of list
ranges[align_bits][1] = base;
check_block(ranges[align_bits][1], align_bits);
}
/**
* Find a block of the correct size. May split larger blocks
* to satisfy this request.
*/
template<SNMALLOC_CONCEPT(ConceptPAL) PAL>
CapPtr<void, CBChunk> remove_block(size_t align_bits)
{
CapPtr<void, CBChunk> first = ranges[align_bits][0];
if (first == nullptr)
{
if (align_bits == (bits::BITS - 1))
{
// Out of memory
return nullptr;
}
// Look for larger block and split up recursively
CapPtr<void, CBChunk> bigger = remove_block<PAL>(align_bits + 1);
if (bigger != nullptr)
{
size_t left_over_size = bits::one_at_bit(align_bits);
auto left_over = pointer_offset(bigger, left_over_size);
ranges[align_bits][0] =
Aal::capptr_bound<void, CBChunk>(left_over, left_over_size);
check_block(left_over, align_bits);
}
check_block(bigger, align_bits + 1);
return bigger;
}
CapPtr<void, CBChunk> second = ranges[align_bits][1];
if (second != nullptr)
{
commit_block<PAL>(second, sizeof(void*));
auto psecond =
second.template as_static<CapPtr<void, CBChunk>>().unsafe_ptr();
auto next = *psecond;
ranges[align_bits][1] = next;
// Zero memory. Client assumes memory contains only zeros.
*psecond = nullptr;
check_block(second, align_bits);
check_block(next, align_bits);
return second;
}
check_block(first, align_bits);
ranges[align_bits][0] = nullptr;
return first;
}
public:
/**
* Add a range of memory to the address space.
* Divides blocks into power of two sizes with natural alignment
*/
template<SNMALLOC_CONCEPT(ConceptPAL) PAL>
void add_range(CapPtr<void, CBChunk> base, size_t length)
{
// Find the minimum set of maximally aligned blocks in this range.
// Each block's alignment and size are equal.
while (length >= sizeof(void*))
{
size_t base_align_bits = bits::ctz(address_cast(base));
size_t length_align_bits = (bits::BITS - 1) - bits::clz(length);
size_t align_bits = bits::min(base_align_bits, length_align_bits);
size_t align = bits::one_at_bit(align_bits);
check_block(base, align_bits);
add_block<PAL>(align_bits, base);
base = pointer_offset(base, align);
length -= align;
}
}
/**
* Commit a block of memory
*/
template<SNMALLOC_CONCEPT(ConceptPAL) PAL>
void commit_block(CapPtr<void, CBChunk> base, size_t size)
{
// Rounding required for sub-page allocations.
auto page_start = pointer_align_down<OS_PAGE_SIZE, char>(base);
auto page_end =
pointer_align_up<OS_PAGE_SIZE, char>(pointer_offset(base, size));
size_t using_size = pointer_diff(page_start, page_end);
PAL::template notify_using<NoZero>(page_start.unsafe_ptr(), using_size);
}
/**
* Returns a pointer to a block of memory of the supplied size.
* The block will be committed, if specified by the template parameter.
* The returned block is guaranteed to be aligened to the size.
*
* Only request 2^n sizes, and not less than a pointer.
*
* On StrictProvenance architectures, any underlying allocations made as
* part of satisfying the request will be registered with the provided
* arena_map for use in subsequent amplification.
*/
template<SNMALLOC_CONCEPT(ConceptPAL) PAL>
CapPtr<void, CBChunk> reserve(size_t size)
{
#ifdef SNMALLOC_TRACING
std::cout << "ASM Core reserve request:" << size << std::endl;
#endif
SNMALLOC_ASSERT(bits::is_pow2(size));
SNMALLOC_ASSERT(size >= sizeof(void*));
return remove_block<PAL>(bits::next_pow2_bits(size));
}
/**
* Aligns block to next power of 2 above size, and unused space at the end
* of the block is retained by the address space manager.
*
* This is useful for allowing the space required for alignment to be
* used, by smaller objects.
*/
template<SNMALLOC_CONCEPT(ConceptPAL) PAL>
CapPtr<void, CBChunk> reserve_with_left_over(size_t size)
{
SNMALLOC_ASSERT(size >= sizeof(void*));
size = bits::align_up(size, sizeof(void*));
size_t rsize = bits::next_pow2(size);
auto res = reserve<PAL>(rsize);
if (res != nullptr)
{
if (rsize > size)
{
add_range<PAL>(pointer_offset(res, size), rsize - size);
}
}
return res;
}
/**
* Default constructor. An address-space manager constructed in this way
* does not own any memory at the start and will request any that it needs
* from the PAL.
*/
AddressSpaceManagerCore() = default;
};
} // namespace snmalloc

206
src/backend/backend.h Normal file
View File

@@ -0,0 +1,206 @@
#pragma once
#include "../mem/allocconfig.h"
#include "../mem/metaslab.h"
#include "../pal/pal.h"
#include "address_space.h"
#include "pagemap.h"
namespace snmalloc
{
/**
* This class implements the standard backend for handling allocations.
* It abstracts page table management and address space management.
*/
template<typename PAL, bool fixed_range>
class BackendAllocator
{
public:
using Pal = PAL;
/**
* Local state for the backend allocator.
*
* This class contains thread local structures to make the implementation
* of the backend allocator more efficient.
*/
class LocalState
{
friend BackendAllocator;
// TODO Separate meta data and object
AddressSpaceManagerCore local_address_space;
};
/**
* Global state for the backend allocator
*
* This contains the various global datastructures required to store
* meta-data for each chunk of memory, and to provide well aligned chunks
* of memory.
*
* This type is required by snmalloc to exist as part of the Backend.
*/
class GlobalState
{
friend BackendAllocator;
// TODO Separate meta data and object
AddressSpaceManager<PAL> address_space;
FlatPagemap<MIN_CHUNK_BITS, MetaEntry, PAL, fixed_range> pagemap;
public:
void init()
{
pagemap.init(&address_space);
if constexpr (fixed_range)
{
abort();
}
}
void init(CapPtr<void, CBChunk> base, size_t length)
{
address_space.add_range(base, length);
pagemap.init(&address_space, address_cast(base), length);
if constexpr (!fixed_range)
{
abort();
}
}
};
private:
/**
* Internal method for acquiring state from the local and global address
* space managers.
*/
template<bool is_meta>
static CapPtr<void, CBChunk>
reserve(GlobalState& h, LocalState* local_state, size_t size)
{
// TODO have two address spaces.
UNUSED(is_meta);
CapPtr<void, CBChunk> p;
if (local_state != nullptr)
{
p =
local_state->local_address_space.template reserve_with_left_over<PAL>(
size);
if (p != nullptr)
local_state->local_address_space.template commit_block<PAL>(p, size);
else
{
auto& a = h.address_space;
// TODO Improve heuristics and params
auto refill_size = bits::max(size, bits::one_at_bit(21));
auto refill = a.template reserve<false>(refill_size);
if (refill == nullptr)
return nullptr;
local_state->local_address_space.template add_range<PAL>(
refill, refill_size);
// This should succeed
p = local_state->local_address_space
.template reserve_with_left_over<PAL>(size);
if (p != nullptr)
local_state->local_address_space.template commit_block<PAL>(
p, size);
}
}
else
{
auto& a = h.address_space;
p = a.template reserve_with_left_over<true>(size);
}
return p;
}
public:
/**
* Provide a block of meta-data with size and align.
*
* Backend allocator may use guard pages and separate area of
* address space to protect this from corruption.
*/
static CapPtr<void, CBChunk>
alloc_meta_data(GlobalState& h, LocalState* local_state, size_t size)
{
return reserve<true>(h, local_state, size);
}
/**
* Returns a chunk of memory with alignment and size of `size`, and a
* metaslab block.
*
* It additionally set the meta-data for this chunk of memory to
* be
* (remote, sizeclass, metaslab)
* where metaslab, is the second element of the pair return.
*/
static std::pair<CapPtr<void, CBChunk>, Metaslab*> alloc_chunk(
GlobalState& h,
LocalState* local_state,
size_t size,
RemoteAllocator* remote,
sizeclass_t sizeclass)
{
SNMALLOC_ASSERT(bits::is_pow2(size));
SNMALLOC_ASSERT(size >= MIN_CHUNK_SIZE);
CapPtr<void, CBChunk> p = reserve<false>(h, local_state, size);
#ifdef SNMALLOC_TRACING
std::cout << "Alloc chunk: " << p.unsafe_ptr() << " (" << size << ")"
<< std::endl;
#endif
if (p == nullptr)
{
#ifdef SNMALLOC_TRACING
std::cout << "Out of memory" << std::endl;
#endif
return {p, nullptr};
}
auto meta = reinterpret_cast<Metaslab*>(
reserve<true>(h, local_state, sizeof(Metaslab)).unsafe_ptr());
MetaEntry t(meta, remote, sizeclass);
for (address_t a = address_cast(p);
a < address_cast(pointer_offset(p, size));
a += MIN_CHUNK_SIZE)
{
h.pagemap.add(a, t);
}
return {p, meta};
}
/**
* Get the metadata associated with a chunk.
*
* Set template parameter to true if it not an error
* to access a location that is not backed by a chunk.
*/
template<bool potentially_out_of_range = false>
static const MetaEntry& get_meta_data(GlobalState& h, address_t p)
{
return h.pagemap.template get<potentially_out_of_range>(p);
}
/**
* Set the metadata associated with a chunk.
*/
static void
set_meta_data(GlobalState& h, address_t p, size_t size, MetaEntry t)
{
for (address_t a = p; a < p + size; a += MIN_CHUNK_SIZE)
{
h.pagemap.set(a, t);
}
}
};
} // namespace snmalloc

179
src/backend/pagemap.h Normal file
View File

@@ -0,0 +1,179 @@
#pragma once
#include "../ds/bits.h"
#include "../ds/helpers.h"
#include "../pal/pal.h"
#include <atomic>
#include <utility>
namespace snmalloc
{
/**
* Simple pagemap that for each GRANULARITY_BITS of the address range
* stores a T.
*/
template<size_t GRANULARITY_BITS, typename T, typename PAL, bool has_bounds>
class FlatPagemap
{
private:
static constexpr size_t SHIFT = GRANULARITY_BITS;
// Before init is called will contain a single entry
// that is the default value. This is needed so that
// various calls do not have to check for nullptr.
// free(nullptr)
// and
// malloc_usable_size(nullptr)
// do not require an allocation to have ocurred before
// they are called.
inline static const T default_value{};
T* body{const_cast<T*>(&default_value)};
address_t base{0};
size_t size{0};
/**
* Commit entry
*/
void commit_entry(void* p)
{
auto entry_size = sizeof(T);
static_assert(sizeof(T) < OS_PAGE_SIZE);
// Rounding required for sub-page allocations.
auto page_start = pointer_align_down<OS_PAGE_SIZE, char>(p);
auto page_end =
pointer_align_up<OS_PAGE_SIZE, char>(pointer_offset(p, entry_size));
size_t using_size = pointer_diff(page_start, page_end);
PAL::template notify_using<NoZero>(page_start, using_size);
}
public:
constexpr FlatPagemap() = default;
template<typename ASM>
void init(ASM* a, address_t b = 0, size_t s = 0)
{
if constexpr (has_bounds)
{
#ifdef SNMALLOC_TRACING
std::cout << "Pagemap.init " << (void*)b << " (" << s << ")"
<< std::endl;
#endif
SNMALLOC_ASSERT(s != 0);
// Align the start and end. We won't store for the very ends as they
// are not aligned to a chunk boundary.
base = bits::align_up(b, bits::one_at_bit(GRANULARITY_BITS));
auto end = bits::align_down(b + s, bits::one_at_bit(GRANULARITY_BITS));
size = end - base;
body = a->template reserve<false, false>(
bits::next_pow2((size >> SHIFT) * sizeof(T)))
.template as_static<T>()
.unsafe_ptr();
;
}
else
{
// The parameters should not be set without has_bounds.
UNUSED(s);
UNUSED(b);
SNMALLOC_ASSERT(s == 0);
SNMALLOC_ASSERT(b == 0);
static constexpr size_t COVERED_BITS =
bits::ADDRESS_BITS - GRANULARITY_BITS;
static constexpr size_t ENTRIES = bits::one_at_bit(COVERED_BITS);
auto new_body = (a->template reserve<false, false>(ENTRIES * sizeof(T)))
.template as_static<T>()
.unsafe_ptr();
// Ensure bottom page is committed
commit_entry(&new_body[0]);
// Set up zero page
new_body[0] = body[0];
body = new_body;
// TODO this is pretty sparse, should we ignore huge pages for it?
// madvise(body, size, MADV_NOHUGEPAGE);
}
}
/**
* If the location has not been used before, then
* `potentially_out_of_range` should be set to true.
* This will ensure there is a location for the
* read/write.
*/
template<bool potentially_out_of_range>
const T& get(address_t p)
{
if constexpr (has_bounds)
{
if (p - base > size)
{
if constexpr (potentially_out_of_range)
{
return default_value;
}
else
{
// Out of range null should
// still return the default value.
if (p == 0)
return default_value;
PAL::error("Internal error: Pagemap read access out of range.");
}
}
p = p - base;
}
// This means external pointer on Windows will be slow.
if constexpr (potentially_out_of_range)
{
commit_entry(&body[p >> SHIFT]);
}
return body[p >> SHIFT];
}
void set(address_t p, T t)
{
#ifdef SNMALLOC_TRACING
std::cout << "Pagemap.Set " << (void*)p << std::endl;
#endif
if constexpr (has_bounds)
{
if (p - base > size)
{
PAL::error("Internal error: Pagemap write access out of range.");
}
p = p - base;
}
body[p >> SHIFT] = t;
}
void add(address_t p, T t)
{
#ifdef SNMALLOC_TRACING
std::cout << "Pagemap.Add " << (void*)p << std::endl;
#endif
if constexpr (has_bounds)
{
if (p - base > size)
{
PAL::error("Internal error: Pagemap new write access out of range.");
}
p = p - base;
}
// This could be the first time this page is used
// This will potentially be expensive on Windows,
// and we should revisit the performance here.
commit_entry(&body[p >> SHIFT]);
body[p >> SHIFT] = t;
}
};
} // namespace snmalloc