Cheri-research/snmalloc
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Major refactor of snmalloc (#343)

Browse Source 
# Pagemap
 
The Pagemap now stores all the meta-data for the object allocation. The meta-data in the pagemap is effectively a triple of the sizeclass, the remote allocator, and a pointer to a 64 byte block of meta-data for this chunk of memory. By storing the pointer to a block, it allows the pagemap to handle multiple slab sizes without branching on the fast path. There is one entry in the pagemap per 16KiB of address space, but by using the same entry in the pagemap for 4 adjacent entries, then we can treat a 64KiB range can be treated as a single slab of allocations.

This change also means there is almost no capability amplification required by the implementation on CHERI for finding meta-data. The only amplification is required, when we change the way a chunk is used to a size of object allocation.


# Backend

There is a second major aspect of the refactor that there is now a narrow API that abstracts the Pagemap, PAL and address space management. This should better enable the compartmentalisation and makes it easier to produce alternative backends for various research directions. This is a template parameter that can be used to specialised by the front-end in different ways.

# Thread local state

The thread local state has been refactored into two components, one (called 'localalloc') that is stored directly in the TLS and is constant initialised, and one that is allocated in the address space (called 'coreallloc') which is lazily created and pooled.

# Difference

This removes Superslabs/Medium slabs as there meta-data is now part of the pagemap.
This commit is contained in:
Matthew Parkinson
2021-07-12 15:53:36 +01:00
committed by GitHub
parent 18d7cc99b6
commit f0e2ab702a
83 changed files with 4404 additions and 5769 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
src/ds/aba.h
View File

@@ -13,7 +13,7 @@
*/
namespace snmalloc
{
#ifndef NDEBUG
#if !defined(NDEBUG) && !defined(SNMALLOC_DISABLE_ABA_VERIFY)
// LL/SC typically can only perform one operation at a time
// check this on other platforms using a thread_local.
inline thread_local bool operation_in_flight = false;
@@ -24,24 +24,20 @@ namespace snmalloc
// fall back to locked implementation.
#if defined(PLATFORM_IS_X86) && \
!(defined(GCC_NOT_CLANG) && defined(OPEN_ENCLAVE))
template<
typename T,
Construction c = RequiresInit,
template<typename> typename Ptr = Pointer,
template<typename> typename AtomicPtr = AtomicPointer>
template<typename T, Construction c = RequiresInit>
class ABA
{
public:
struct alignas(2 * sizeof(std::size_t)) Linked
{
Ptr<T> ptr;
uintptr_t aba;
T* ptr{nullptr};
uintptr_t aba{0};
};
struct Independent
{
AtomicPtr<T> ptr;
std::atomic<uintptr_t> aba;
std::atomic<T*> ptr{nullptr};
std::atomic<uintptr_t> aba{0};
};
static_assert(
@@ -59,13 +55,9 @@ namespace snmalloc
};
public:
ABA()
{
if constexpr (c == RequiresInit)
init(nullptr);
}
constexpr ABA() : independent() {}
void init(Ptr<T> x)
void init(T* x)
{
independent.ptr.store(x, std::memory_order_relaxed);
independent.aba.store(0, std::memory_order_relaxed);
@@ -75,7 +67,7 @@ namespace snmalloc
Cmp read()
{
# ifndef NDEBUG
# if !defined(NDEBUG) && !defined(SNMALLOC_DISABLE_ABA_VERIFY)
if (operation_in_flight)
error("Only one inflight ABA operation at a time is allowed.");
operation_in_flight = true;
@@ -97,12 +89,12 @@ namespace snmalloc
*/
Cmp(Linked old, ABA* parent) : old(old), parent(parent) {}
Ptr<T> ptr()
T* ptr()
{
return old.ptr;
}
bool store_conditional(Ptr<T> value)
bool store_conditional(T* value)
{
# if defined(_MSC_VER) && defined(SNMALLOC_VA_BITS_64)
auto result = _InterlockedCompareExchange128(
@@ -127,7 +119,7 @@ namespace snmalloc
~Cmp()
{
# ifndef NDEBUG
# if !defined(NDEBUG) && !defined(SNMALLOC_DISABLE_ABA_VERIFY)
operation_in_flight = false;
# endif
}
@@ -137,7 +129,7 @@ namespace snmalloc
};
// This method is used in Verona
Ptr<T> peek()
T* peek()
{
return independent.ptr.load(std::memory_order_relaxed);
}
@@ -146,19 +138,15 @@ namespace snmalloc
/**
* Naive implementation of ABA protection using a spin lock.
*/
template<
typename T,
Construction c = RequiresInit,
template<typename> typename Ptr = Pointer,
template<typename> typename AtomicPtr = AtomicPointer>
template<typename T, Construction c = RequiresInit>
class ABA
{
AtomicPtr<T> ptr = nullptr;
std::atomic<T*> ptr = nullptr;
std::atomic_flag lock = ATOMIC_FLAG_INIT;
public:
// This method is used in Verona
void init(Ptr<T> x)
void init(T* x)
{
ptr.store(x, std::memory_order_relaxed);
}
@@ -170,7 +158,7 @@ namespace snmalloc
while (lock.test_and_set(std::memory_order_acquire))
Aal::pause();
# ifndef NDEBUG
# if !defined(NDEBUG) && !defined(SNMALLOC_DISABLE_ABA_VERIFY)
if (operation_in_flight)
error("Only one inflight ABA operation at a time is allowed.");
operation_in_flight = true;
@@ -184,12 +172,12 @@ namespace snmalloc
ABA* parent;
public:
Ptr<T> ptr()
T* ptr()
{
return parent->ptr;
}
bool store_conditional(Ptr<T> t)
bool store_conditional(T* t)
{
parent->ptr = t;
return true;
@@ -198,14 +186,14 @@ namespace snmalloc
~Cmp()
{
parent->lock.clear(std::memory_order_release);
# ifndef NDEBUG
# if !defined(NDEBUG) && !defined(SNMALLOC_DISABLE_ABA_VERIFY)
operation_in_flight = false;
# endif
}
};
// This method is used in Verona
Ptr<T> peek()
T* peek()
{
return ptr.load(std::memory_order_relaxed);
}

21
src/ds/address.h
View File

@@ -29,7 +29,7 @@ namespace snmalloc
inline CapPtr<void, bounds>
pointer_offset(CapPtr<T, bounds> base, size_t diff)
{
return CapPtr<void, bounds>(pointer_offset(base.unsafe_capptr, diff));
return CapPtr<void, bounds>(pointer_offset(base.unsafe_ptr(), diff));
}
/**
@@ -45,8 +45,7 @@ namespace snmalloc
inline CapPtr<void, bounds>
pointer_offset_signed(CapPtr<T, bounds> base, ptrdiff_t diff)
{
return CapPtr<void, bounds>(
pointer_offset_signed(base.unsafe_capptr, diff));
return CapPtr<void, bounds>(pointer_offset_signed(base.unsafe_ptr(), diff));
}
/**
@@ -69,7 +68,7 @@ namespace snmalloc
template<typename T, enum capptr_bounds bounds>
inline address_t address_cast(CapPtr<T, bounds> a)
{
return address_cast(a.unsafe_capptr);
return address_cast(a.unsafe_ptr());
}
/**
@@ -95,7 +94,7 @@ namespace snmalloc
* power of two.
*/
template<size_t alignment, typename T = void>
SNMALLOC_FAST_PATH T* pointer_align_down(void* p)
inline T* pointer_align_down(void* p)
{
static_assert(alignment > 0);
static_assert(bits::is_pow2(alignment));
@@ -115,7 +114,7 @@ namespace snmalloc
template<size_t alignment, typename T, capptr_bounds bounds>
inline CapPtr<T, bounds> pointer_align_down(CapPtr<void, bounds> p)
{
return CapPtr<T, bounds>(pointer_align_down<alignment, T>(p.unsafe_capptr));
return CapPtr<T, bounds>(pointer_align_down<alignment, T>(p.unsafe_ptr()));
}
template<size_t alignment>
@@ -149,7 +148,7 @@ namespace snmalloc
template<size_t alignment, typename T = void, enum capptr_bounds bounds>
inline CapPtr<T, bounds> pointer_align_up(CapPtr<void, bounds> p)
{
return CapPtr<T, bounds>(pointer_align_up<alignment, T>(p.unsafe_capptr));
return CapPtr<T, bounds>(pointer_align_up<alignment, T>(p.unsafe_ptr()));
}
template<size_t alignment>
@@ -163,7 +162,7 @@ namespace snmalloc
* a power of two.
*/
template<typename T = void>
SNMALLOC_FAST_PATH T* pointer_align_down(void* p, size_t alignment)
inline T* pointer_align_down(void* p, size_t alignment)
{
SNMALLOC_ASSERT(alignment > 0);
SNMALLOC_ASSERT(bits::is_pow2(alignment));
@@ -196,7 +195,7 @@ namespace snmalloc
inline CapPtr<T, bounds>
pointer_align_up(CapPtr<void, bounds> p, size_t alignment)
{
return CapPtr<T, bounds>(pointer_align_up<T>(p.unsafe_capptr, alignment));
return CapPtr<T, bounds>(pointer_align_up<T>(p.unsafe_ptr(), alignment));
}
/**
@@ -218,7 +217,7 @@ namespace snmalloc
enum capptr_bounds Ubounds>
inline size_t pointer_diff(CapPtr<T, Tbounds> base, CapPtr<U, Ubounds> cursor)
{
return pointer_diff(base.unsafe_capptr, cursor.unsafe_capptr);
return pointer_diff(base.unsafe_ptr(), cursor.unsafe_ptr());
}
/**
@@ -239,7 +238,7 @@ namespace snmalloc
inline ptrdiff_t
pointer_diff_signed(CapPtr<T, Tbounds> base, CapPtr<U, Ubounds> cursor)
{
return pointer_diff_signed(base.unsafe_capptr, cursor.unsafe_capptr);
return pointer_diff_signed(base.unsafe_ptr(), cursor.unsafe_ptr());
}
} // namespace snmalloc

19
src/ds/bits.h
View File

@@ -53,7 +53,7 @@ namespace snmalloc
static constexpr size_t ADDRESS_BITS = is64() ? 48 : 32;
SNMALLOC_FAST_PATH size_t clz(size_t x)
inline SNMALLOC_FAST_PATH size_t clz(size_t x)
{
SNMALLOC_ASSERT(x != 0); // Calling with 0 is UB on some implementations
#if defined(_MSC_VER)
@@ -219,7 +219,7 @@ namespace snmalloc
return (x & (x - 1)) == 0;
}
SNMALLOC_FAST_PATH size_t next_pow2(size_t x)
inline SNMALLOC_FAST_PATH size_t next_pow2(size_t x)
{
// Correct for numbers [0..MAX_SIZE >> 1).
// Returns 1 for x > (MAX_SIZE >> 1).
@@ -246,12 +246,20 @@ namespace snmalloc
return one_at_bit(BITS - clz_const(x - 1));
}
constexpr size_t next_pow2_bits_const(size_t x)
constexpr size_t prev_pow2_const(size_t x)
{
if (x <= 2)
return x;
return one_at_bit(BITS - (clz_const(x + 1) + 1));
}
inline constexpr size_t next_pow2_bits_const(size_t x)
{
return BITS - clz_const(x - 1);
}
constexpr SNMALLOC_FAST_PATH size_t
inline constexpr SNMALLOC_FAST_PATH size_t
align_down(size_t value, size_t alignment)
{
SNMALLOC_ASSERT(is_pow2(alignment));
@@ -261,7 +269,8 @@ namespace snmalloc
return value;
}
constexpr SNMALLOC_FAST_PATH size_t align_up(size_t value, size_t alignment)
inline constexpr SNMALLOC_FAST_PATH size_t
align_up(size_t value, size_t alignment)
{
SNMALLOC_ASSERT(is_pow2(alignment));

160
src/ds/cdllist.h
View File

@@ -1,5 +1,6 @@
#pragma once
#include "address.h"
#include "defines.h"
#include "ptrwrap.h"
@@ -8,146 +9,67 @@
namespace snmalloc
{
template<typename T, template<typename> typename Ptr = Pointer>
class CDLLNodeBase
{
/**
* to_next is used to handle a zero initialised data structure.
* This means that `is_empty` works even when the constructor hasn't
* been run.
*/
ptrdiff_t to_next = 0;
protected:
void set_next(Ptr<T> c)
{
to_next = pointer_diff_signed(Ptr<CDLLNodeBase<T, Ptr>>(this), c);
}
public:
SNMALLOC_FAST_PATH bool is_empty()
{
return to_next == 0;
}
SNMALLOC_FAST_PATH Ptr<T> get_next()
{
return static_cast<Ptr<T>>(pointer_offset_signed<T>(this, to_next));
}
};
template<typename T, template<typename> typename Ptr = Pointer>
class CDLLNodeBaseNext
{
/**
* Like to_next in the pointer-less case, this version still works with
* zero-initialized data structure. To make `is_empty` work in this case,
* next is set to `nullptr` rather than `this` when the list is empty.
*
*/
Ptr<T> next = nullptr;
protected:
void set_next(Ptr<T> c)
{
next = address_cast(c) == address_cast(this) ? nullptr : c;
}
public:
SNMALLOC_FAST_PATH bool is_empty()
{
return next == nullptr;
}
SNMALLOC_FAST_PATH Ptr<T> get_next()
{
return next == nullptr ? Ptr<T>(static_cast<T*>(this)) : next;
}
};
template<typename T, template<typename> typename Ptr = Pointer>
using CDLLNodeParent = std::conditional_t<
aal_supports<StrictProvenance>,
CDLLNodeBaseNext<T, Ptr>,
CDLLNodeBase<T, Ptr>>;
/**
* TODO Rewrite for actual use, no longer Cyclic or doubly linked.
*
* Special class for cyclic doubly linked non-empty linked list
*
* This code assumes there is always one element in the list. The client
* must ensure there is a sentinal element.
*/
template<template<typename> typename Ptr = Pointer>
class CDLLNode : public CDLLNodeParent<CDLLNode<Ptr>, Ptr>
class CDLLNode
{
Ptr<CDLLNode> prev = nullptr;
Ptr<CDLLNode> next{nullptr};
constexpr void set_next(Ptr<CDLLNode> c)
{
next = c;
}
public:
/**
* Single element cyclic list. This is the empty case.
*/
CDLLNode()
constexpr CDLLNode()
{
this->set_next(Ptr<CDLLNode>(this));
prev = Ptr<CDLLNode>(this);
this->set_next(nullptr);
}
SNMALLOC_FAST_PATH bool is_empty()
{
return next == nullptr;
}
SNMALLOC_FAST_PATH Ptr<CDLLNode> get_next()
{
return next;
}
/**
* Removes this element from the cyclic list is it part of.
* Single element cyclic list. This is the uninitialised case.
*
* This entry should never be accessed and is only used to make
* a fake metaslab.
*/
SNMALLOC_FAST_PATH void remove()
constexpr CDLLNode(bool) {}
SNMALLOC_FAST_PATH Ptr<CDLLNode> pop()
{
SNMALLOC_ASSERT(!this->is_empty());
debug_check();
this->get_next()->prev = prev;
prev->set_next(this->get_next());
// As this is no longer in the list, check invariant for
// neighbouring element.
this->get_next()->debug_check();
#ifndef NDEBUG
this->set_next(nullptr);
prev = nullptr;
#endif
auto result = get_next();
set_next(result->get_next());
return result;
}
/**
* Nulls the previous pointer
*
* The Meta-slab uses nullptr in prev to mean that it is not part of a
* size class list.
**/
void null_prev()
{
prev = nullptr;
}
SNMALLOC_FAST_PATH Ptr<CDLLNode> get_prev()
{
return prev;
}
SNMALLOC_FAST_PATH void insert_next(Ptr<CDLLNode> item)
SNMALLOC_FAST_PATH void insert(Ptr<CDLLNode> item)
{
debug_check();
item->set_next(this->get_next());
this->get_next()->prev = item;
item->prev = this;
set_next(item);
debug_check();
}
SNMALLOC_FAST_PATH void insert_prev(Ptr<CDLLNode> item)
{
debug_check();
item->prev = prev;
prev->set_next(item);
item->set_next(Ptr<CDLLNode>(this));
prev = item;
debug_check();
}
/**
* Checks the lists invariants
* x->next->prev = x
@@ -156,15 +78,15 @@ namespace snmalloc
void debug_check()
{
#ifndef NDEBUG
Ptr<CDLLNode> item = this->get_next();
auto p = Ptr<CDLLNode>(this);
// Ptr<CDLLNode> item = this->get_next();
// auto p = Ptr<CDLLNode>(this);
do
{
SNMALLOC_ASSERT(item->prev == p);
p = item;
item = item->get_next();
} while (item != Ptr<CDLLNode>(this));
// do
// {
// SNMALLOC_ASSERT(item->prev == p);
// p = item;
// item = item->get_next();
// } while (item != Ptr<CDLLNode>(this));
#endif
}
};

39
src/ds/defines.h
View File

@@ -7,17 +7,30 @@
# define unlikely(x) !!(x)
# define SNMALLOC_SLOW_PATH NOINLINE
# define SNMALLOC_FAST_PATH ALWAYSINLINE
# if _MSC_VER >= 1927
# define SNMALLOC_FAST_PATH_LAMBDA [[msvc::forceinline]]
# else
# define SNMALLOC_FAST_PATH_LAMBDA
# endif
# define SNMALLOC_PURE
# define SNMALLOC_COLD
# define SNMALLOC_REQUIRE_CONSTINIT
#else
# define likely(x) __builtin_expect(!!(x), 1)
# define unlikely(x) __builtin_expect(!!(x), 0)
# define ALWAYSINLINE __attribute__((always_inline))
# define NOINLINE __attribute__((noinline))
# define SNMALLOC_SLOW_PATH NOINLINE
# define SNMALLOC_FAST_PATH inline ALWAYSINLINE
# define SNMALLOC_FAST_PATH ALWAYSINLINE
# define SNMALLOC_FAST_PATH_LAMBDA SNMALLOC_FAST_PATH
# define SNMALLOC_PURE __attribute__((const))
# define SNMALLOC_COLD __attribute__((cold))
# ifdef __clang__
# define SNMALLOC_REQUIRE_CONSTINIT \
[[clang::require_constant_initialization]]
# else
# define SNMALLOC_REQUIRE_CONSTINIT
# endif
#endif
#if defined(__cpp_constinit) && __cpp_constinit >= 201907
@@ -100,3 +113,27 @@ namespace snmalloc
} while (0)
# endif
#endif
// // The CHECK_CLIENT macro is used to turn on minimal checking of the client
// // calling the API correctly.
// #if !defined(NDEBUG) && !defined(CHECK_CLIENT)
// # define CHECK_CLIENT
// #endif
inline SNMALLOC_FAST_PATH void check_client_error(const char* const str)
{
//[[clang::musttail]]
return snmalloc::error(str);
}
inline SNMALLOC_FAST_PATH void
check_client_impl(bool test, const char* const str)
{
if (unlikely(!test))
check_client_error(str);
}
#ifdef CHECK_CLIENT
# define check_client(test, str) check_client_impl(test, str)
#else
# define check_client(test, str)
#endif

1
src/ds/dllist.h
View File

@@ -1,5 +1,6 @@
#pragma once
#include "address.h"
#include "helpers.h"
#include "invalidptr.h"
#include "ptrwrap.h"

14
src/ds/helpers.h
View File

@@ -3,6 +3,7 @@
#include "bits.h"
#include "flaglock.h"
#include <array>
#include <type_traits>
namespace snmalloc
@@ -72,6 +73,7 @@ namespace snmalloc
}
};
#ifdef CHECK_CLIENT
template<size_t length, typename T>
class ModArray
{
@@ -84,7 +86,7 @@ namespace snmalloc
};
static constexpr size_t rlength = bits::next_pow2_const(length);
TWrap array[rlength];
std::array<TWrap, rlength> array;
public:
constexpr const T& operator[](const size_t i) const
@@ -97,14 +99,22 @@ namespace snmalloc
return array[i & (rlength - 1)].v;
}
};
#else
template<size_t length, typename T>
using ModArray = std::array<T, length>;
#endif
/**
* Helper class to execute a specified function on destruction.
*/
template<void f()>
template<typename F>
class OnDestruct
{
F f;
public:
OnDestruct(F f) : f(f) {}
~OnDestruct()
{
f();

2
src/ds/invalidptr.h
View File

@@ -1,5 +1,7 @@
#pragma once
#include "address.h"
namespace snmalloc
{
/**

35
src/ds/mpmcstack.h
View File

@@ -1,50 +1,49 @@
#pragma once
#include "aba.h"
#include "ptrwrap.h"
namespace snmalloc
{
template<
class T,
Construction c = RequiresInit,
template<typename> typename Ptr = Pointer,
template<typename> typename AtomicPtr = AtomicPointer>
template<class T, Construction c = RequiresInit>
class MPMCStack
{
using ABAT = ABA<T, c, Ptr, AtomicPtr>;
using ABAT = ABA<T, c>;
private:
static_assert(
std::is_same<decltype(T::next), AtomicPtr<T>>::value,
"T->next must be an AtomicPtr<T>");
ABAT stack;
public:
void push(Ptr<T> item)
constexpr MPMCStack() = default;
void push(T* item)
{
static_assert(
std::is_same<decltype(T::next), std::atomic<T*>>::value,
"T->next must be an std::atomic<T*>");
return push(item, item);
}
void push(Ptr<T> first, Ptr<T> last)
void push(T* first, T* last)
{
// Pushes an item on the stack.
auto cmp = stack.read();
do
{
Ptr<T> top = cmp.ptr();
auto top = cmp.ptr();
last->next.store(top, std::memory_order_release);
} while (!cmp.store_conditional(first));
}
Ptr<T> pop()
T* pop()
{
// Returns the next item. If the returned value is decommitted, it is
// possible for the read of top->next to segfault.
auto cmp = stack.read();
Ptr<T> top;
Ptr<T> next;
T* top;
T* next;
do
{
@@ -59,11 +58,11 @@ namespace snmalloc
return top;
}
Ptr<T> pop_all()
T* pop_all()
{
// Returns all items as a linked list, leaving an empty stack.
auto cmp = stack.read();
Ptr<T> top;
T* top;
do
{

11
src/ds/mpscq.h
View File

@@ -18,9 +18,11 @@ namespace snmalloc
"T->next must be an AtomicPtr<T>");
AtomicPtr<T> back{nullptr};
Ptr<T> front = nullptr;
Ptr<T> front{nullptr};
public:
constexpr MPSCQ() = default;
void invariant()
{
SNMALLOC_ASSERT(back != nullptr);
@@ -61,6 +63,11 @@ namespace snmalloc
prev->next.store(first, std::memory_order_relaxed);
}
Ptr<T> peek()
{
return front;
}
std::pair<Ptr<T>, bool> dequeue()
{
// Returns the front message, or null if not possible to return a message.
@@ -68,10 +75,10 @@ namespace snmalloc
Ptr<T> first = front;
Ptr<T> next = first->next.load(std::memory_order_relaxed);
Aal::prefetch(&(next->next));
if (next != nullptr)
{
front = next;
Aal::prefetch(&(next->next));
SNMALLOC_ASSERT(front != nullptr);
std::atomic_thread_fence(std::memory_order_acquire);
invariant();

45
src/ds/ptrwrap.h
View File

@@ -1,5 +1,7 @@
#pragma once
#include "defines.h"
#include <atomic>
namespace snmalloc
@@ -99,16 +101,17 @@ namespace snmalloc
* summary of its StrictProvenance metadata.
*/
template<typename T, capptr_bounds bounds>
struct CapPtr
class CapPtr
{
T* unsafe_capptr;
uintptr_t unsafe_capptr;
public:
/**
* nullptr is implicitly constructable at any bounds type
*/
CapPtr(const std::nullptr_t n) : unsafe_capptr(n) {}
constexpr CapPtr(const std::nullptr_t) : unsafe_capptr(0) {}
CapPtr() : CapPtr(nullptr) {}
constexpr CapPtr() : CapPtr(nullptr){};
/**
* all other constructions must be explicit
@@ -124,18 +127,20 @@ namespace snmalloc
# pragma warning(push)
# pragma warning(disable : 4702)
#endif
explicit CapPtr(T* p) : unsafe_capptr(p) {}
constexpr explicit CapPtr(uintptr_t p) : unsafe_capptr(p) {}
#ifdef _MSC_VER
# pragma warning(pop)
#endif
explicit CapPtr(T* p) : unsafe_capptr(reinterpret_cast<uintptr_t>(p)) {}
/**
* Allow static_cast<>-s that preserve bounds but vary the target type.
*/
template<typename U>
SNMALLOC_FAST_PATH CapPtr<U, bounds> as_static()
{
return CapPtr<U, bounds>(static_cast<U*>(this->unsafe_capptr));
return CapPtr<U, bounds>(this->unsafe_capptr);
}
SNMALLOC_FAST_PATH CapPtr<void, bounds> as_void()
@@ -149,7 +154,7 @@ namespace snmalloc
template<typename U>
SNMALLOC_FAST_PATH CapPtr<U, bounds> as_reinterpret()
{
return CapPtr<U, bounds>(reinterpret_cast<U*>(this->unsafe_capptr));
return CapPtr<U, bounds>(this->unsafe_capptr);
}
SNMALLOC_FAST_PATH bool operator==(const CapPtr& rhs) const
@@ -167,6 +172,16 @@ namespace snmalloc
return this->unsafe_capptr < rhs.unsafe_capptr;
}
[[nodiscard]] SNMALLOC_FAST_PATH T* unsafe_ptr() const
{
return reinterpret_cast<T*>(this->unsafe_capptr);
}
[[nodiscard]] SNMALLOC_FAST_PATH uintptr_t unsafe_uintptr() const
{
return this->unsafe_capptr;
}
SNMALLOC_FAST_PATH T* operator->() const
{
/*
@@ -174,7 +189,7 @@ namespace snmalloc
* client; we should be doing nothing with them.
*/
static_assert(bounds != CBAllocE);
return this->unsafe_capptr;
return unsafe_ptr();
}
};
@@ -198,7 +213,7 @@ namespace snmalloc
* several chunks) to be the allocation.
*/
template<typename T>
SNMALLOC_FAST_PATH CapPtr<T, CBAllocE>
inline SNMALLOC_FAST_PATH CapPtr<T, CBAllocE>
capptr_chunk_is_alloc(CapPtr<T, CBChunkE> p)
{
return CapPtr<T, CBAlloc>(p.unsafe_capptr);
@@ -208,9 +223,9 @@ namespace snmalloc
* With all the bounds and constraints in place, it's safe to extract a void
* pointer (to reveal to the client).
*/
SNMALLOC_FAST_PATH void* capptr_reveal(CapPtr<void, CBAllocE> p)
inline SNMALLOC_FAST_PATH void* capptr_reveal(CapPtr<void, CBAllocE> p)
{
return p.unsafe_capptr;
return p.unsafe_ptr();
}
/**
@@ -230,12 +245,12 @@ namespace snmalloc
/**
* nullptr is constructable at any bounds type
*/
AtomicCapPtr(const std::nullptr_t n) : unsafe_capptr(n) {}
constexpr AtomicCapPtr(const std::nullptr_t n) : unsafe_capptr(n) {}
/**
* Interconversion with CapPtr
*/
AtomicCapPtr(CapPtr<T, bounds> p) : unsafe_capptr(p.unsafe_capptr) {}
AtomicCapPtr(CapPtr<T, bounds> p) : unsafe_capptr(p.unsafe_ptr()) {}
operator CapPtr<T, bounds>() const noexcept
{
@@ -261,7 +276,7 @@ namespace snmalloc
CapPtr<T, bounds> desired,
std::memory_order order = std::memory_order_seq_cst) noexcept
{
this->unsafe_capptr.store(desired.unsafe_capptr, order);
this->unsafe_capptr.store(desired.unsafe_ptr(), order);
}
SNMALLOC_FAST_PATH CapPtr<T, bounds> exchange(
@@ -269,7 +284,7 @@ namespace snmalloc
std::memory_order order = std::memory_order_seq_cst) noexcept
{
return CapPtr<T, bounds>(
this->unsafe_capptr.exchange(desired.unsafe_capptr, order));
this->unsafe_capptr.exchange(desired.unsafe_ptr(), order));
}
SNMALLOC_FAST_PATH bool operator==(const AtomicCapPtr& rhs) const