Cheri-research/snmalloc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor: Remove unused features and functions, and move most allocator operations to a global namespace. (#750)

Browse Source 
* Factor out explicit Config type

Instead of using snmalloc::Alloc::Config, expose snmalloc::Config, which is then used to derive the allocator type.

* Move globalalloc to front end.

* Remove unneed template parameter from global snmalloc functions.

* Remove SNMALLOC_PASS_THROUGH

VeronaRT now has an abstraction layer which can easily replace the allocator.
Having such a complex integration still in snmalloc does not make sense.

* Take some global functions off of local alloc.

* Drop comparison overloads on atomic Capptr.

Performing a comparison on two atomic ptr is a complex operation, and should not be implicit.  The memory model order and such things needs to be considered by the caller.

* Remove function_ref and use templates

The implementation prefers to use templates over the function_ref.  This now only exists in the Pal for a currently unused feature.

* Removing function_ref reduces stl needs.

* Remove use of __is_convertible to support older g++

* Inline function that is only used once.

* Remove unused function

* Restrict ThreadAlloc usage to globalalloc

This commit introduces various inline functions on snmalloc:: that perform allocation/deallocation using the thread local allocator.

They remove all usage from a particular test.

* Move cheri checks to own file.

* Refactor is_owned checks.

* Move alloc_size and check_size to globalalloc.

* Minor simplification of dealloc path

* Fix up is_owned to take a config

* Improve usage of scoped allocator.

* Handle Config_ in globalalloc.
This commit is contained in:
Matthew Parkinson
2025-02-22 19:53:27 +00:00
committed by GitHub
parent 6d50141e35
commit 5f7baef755
50 changed files with 818 additions and 1403 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/snmalloc/README.md
View File

@@ -20,7 +20,7 @@ These are arranged in a hierarchy such that each of the directories may include
This includes data structures such as pagemap implementations (efficient maps from a chunk address to associated metadata) and buddy allocators for managing address-space ranges.
- `backend/` provides some example implementations for snmalloc embeddings that provide a global memory allocator for an address space.
Users may ignore this entirely and use the types in `mem/` with a custom back end to expose an snmalloc instance with specific behaviour.
Layers above this can be used with a custom configuration by defining `SNMALLOC_PROVIDE_OWN_CONFIG` and exporting a type as `snmalloc::Alloc` that defines the type of an `snmalloc::LocalAllocator` template specialisation.
Layers above this can be used with a custom configuration by defining `SNMALLOC_PROVIDE_OWN_CONFIG` and exporting a type as `snmalloc::Config` that defines the configuration.
- `global/` provides some front-end components that assume that snmalloc is available in a global configuration.
- `override/` builds on top of `global/` to provide specific implementations with compatibility with external specifications (for example C `malloc`, C++ `operator new`, jemalloc's `*allocx`, or Rust's `std::alloc`).

139
src/snmalloc/ds_core/cheri.h Normal file
View File

@@ -0,0 +1,139 @@
#include "mitigations.h"
namespace snmalloc
{
/*
* Many of these tests come with an "or is null" branch that they'd need to
* add if we did them up front. Instead, defer them until we're past the
* point where we know, from the pagemap, or by explicitly testing, that the
* pointer under test is not nullptr.
*/
SNMALLOC_FAST_PATH_INLINE void dealloc_cheri_checks(void* p)
{
#if defined(__CHERI_PURE_CAPABILITY__)
/*
* Enforce the use of an unsealed capability.
*
* TODO In CHERI+MTE, this, is part of the CAmoCDecVersion instruction;
* elide this test in that world.
*/
snmalloc_check_client(
mitigations(cheri_checks),
!__builtin_cheri_sealed_get(p),
"Sealed capability in deallocation");
/*
* Enforce permissions on the returned pointer. These pointers end up in
* free queues and will be cycled out to clients again, so try to catch
* erroneous behavior now, rather than later.
*
* TODO In the CHERI+MTE case, we must reconstruct the pointer for the
* free queues as part of the discovery of the start of the object (so
* that it has the correct version), and the CAmoCDecVersion call imposes
* its own requirements on the permissions (to ensure that it's at least
* not zero). They are somewhat more lax than we might wish, so this test
* may remain, guarded by SNMALLOC_CHECK_CLIENT, but no explicit
* permissions checks are required in the non-SNMALLOC_CHECK_CLIENT case
* to defend ourselves or other clients against a misbehaving client.
*/
static const size_t reqperm = CHERI_PERM_LOAD | CHERI_PERM_STORE |
CHERI_PERM_LOAD_CAP | CHERI_PERM_STORE_CAP;
snmalloc_check_client(
mitigations(cheri_checks),
(__builtin_cheri_perms_get(p) & reqperm) == reqperm,
"Insufficient permissions on capability in deallocation");
/*
* We check for a valid tag here, rather than in domestication, because
* domestication might be answering a slightly different question, about
* the plausibility of addresses rather than of exact pointers.
*
* TODO Further, in the CHERI+MTE case, the tag check will be implicit in
* a future CAmoCDecVersion instruction, and there should be no harm in
* the lookups we perform along the way to get there. In that world,
* elide this test.
*/
snmalloc_check_client(
mitigations(cheri_checks),
__builtin_cheri_tag_get(p),
"Untagged capability in deallocation");
/*
* Verify that the capability is not zero-length, ruling out the other
* edge case around monotonicity.
*/
snmalloc_check_client(
mitigations(cheri_checks),
__builtin_cheri_length_get(p) > 0,
"Zero-length capability in deallocation");
/*
* At present we check for the pointer also being the start of an
* allocation closer to dealloc; for small objects, that happens in
* dealloc_local_object_fast, either below or *on the far end of message
* receipt*. For large objects, it happens below by directly rounding to
* power of two rather than using the is_start_of_object helper.
* (XXX This does mean that we might end up threading our remote queue
* state somewhere slightly unexpected rather than at the head of an
* object. That is perhaps fine for now?)
*/
/*
* TODO
*
* We could enforce other policies here, including that the length exactly
* match the sizeclass. At present, we bound caps we give for allocations
* to the underlying sizeclass, so even malloc(0) will have a non-zero
* length. Monotonicity would then imply that the pointer must be the
* head of an object (modulo, perhaps, temporal aliasing if we somehow
* introduced phase shifts in heap layout like some allocators do).
*
* If we switched to bounding with upwards-rounded representable bounds
* (c.f., CRRL) rather than underlying object size, then we should,
* instead, in general require plausibility of p_raw by checking that its
* length is nonzero and the snmalloc size class associated with its
* length is the one for the slab in question... except for the added
* challenge of malloc(0). Since 0 rounds up to 0, we might end up
* constructing zero-length caps to hand out, which we would then reject
* upon receipt. Instead, as part of introducing CRRL bounds, we should
* introduce a sizeclass for slabs holding zero-size objects. All told,
* we would want to check that
*
* size_to_sizeclass(length) == entry.get_sizeclass()
*
* I believe a relaxed CRRL test of
*
* length > 0 || (length == sizeclass_to_size(entry.get_sizeclass()))
*
* would also suffice and may be slightly less expensive than the test
* above, at the cost of not catching as many misbehaving clients.
*
* In either case, having bounded by CRRL bounds, we would need to be
* *reconstructing* the capabilities headed to our free lists to be given
* out to clients again; there are many more CRRL classes than snmalloc
* sizeclasses (this is the same reason that we can always get away with
* CSetBoundsExact in capptr_bound). Switching to CRRL bounds, if that's
* ever a thing we want to do, will be easier after we've done the
* plumbing for CHERI+MTE.
*/
/*
* TODO: Unsurprisingly, the CHERI+MTE case once again has something to
* say here. In that world, again, we are certain to be reconstructing
* the capability for the free queue anyway, and so exactly what we wish
* to enforce, length-wise, of the provided capability, is somewhat more
* flexible. Using the provided capability bounds when recoloring memory
* could be a natural way to enforce that it covers the entire object, at
* the cost of a more elaborate recovery story (as we risk aborting with a
* partially recolored object). On non-SNMALLOC_CHECK_CLIENT builds, it
* likely makes sense to just enforce that length > 0 (*not* enforced by
* the CAmoCDecVersion instruction) and say that any authority-bearing
* interior pointer suffices to free the object. I believe that to be an
* acceptable security posture for the allocator and between clients;
* misbehavior is confined to the misbehaving client.
*/
#else
UNUSED(p);
#endif
}
} // namespace snmalloc

1
src/snmalloc/ds_core/ds_core.h
View File

@@ -8,6 +8,7 @@
*/
#include "bits.h"
#include "cheri.h"
#include "concept.h"
#include "defines.h"
#include "helpers.h"

49
src/snmalloc/ds_core/helpers.h
View File

@@ -85,55 +85,6 @@ namespace snmalloc
}
};
/**
* Non-owning version of std::function. Wraps a reference to a callable object
* (eg. a lambda) and allows calling it through dynamic dispatch, with no
* allocation. This is useful in the allocator code paths, where we can't
* safely use std::function.
*
* Inspired by the C++ proposal:
* http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0792r2.html
*/
template<typename Fn>
struct function_ref;
template<typename R, typename... Args>
struct function_ref<R(Args...)>
{
// The enable_if is used to stop this constructor from shadowing the default
// copy / move constructors.
template<
typename Fn,
typename =
stl::enable_if_t<!stl::is_same_v<stl::decay_t<Fn>, function_ref>>>
function_ref(Fn&& fn)
{
data_ = static_cast<void*>(&fn);
fn_ = execute<Fn>;
}
R operator()(Args... args) const
{
return fn_(data_, args...);
}
private:
void* data_;
R (*fn_)(void*, Args...);
template<typename Fn>
static R execute(void* p, Args... args)
{
return (*static_cast<stl::add_pointer_t<Fn>>(p))(args...);
};
};
template<class T, template<typename> typename Ptr>
void ignore(Ptr<T> t)
{
UNUSED(t);
}
/**
* Helper class for building fatal errors. Used by `report_fatal_error` to
* build an on-stack buffer containing the formatted string.

15
src/snmalloc/ds_core/ptrwrap.h
View File

@@ -517,21 +517,6 @@ namespace snmalloc
return CapPtr<T, bounds>::unsafe_from(
this->unsafe_capptr.exchange(desired.unsafe_ptr(), order));
}
SNMALLOC_FAST_PATH bool operator==(const AtomicCapPtr& rhs) const
{
return this->unsafe_capptr == rhs.unsafe_capptr;
}
SNMALLOC_FAST_PATH bool operator!=(const AtomicCapPtr& rhs) const
{
return this->unsafe_capptr != rhs.unsafe_capptr;
}
SNMALLOC_FAST_PATH bool operator<(const AtomicCapPtr& rhs) const
{
return this->unsafe_capptr < rhs.unsafe_capptr;
}
};
namespace capptr

15
src/snmalloc/global/bounds_checks.h
View File

@@ -1,4 +1,5 @@
#pragma once
#include "globalalloc.h"
#include "threadalloc.h"
namespace snmalloc
@@ -61,18 +62,17 @@ namespace snmalloc
}
else
{
auto& alloc = ThreadAlloc::get();
void* p = const_cast<void*>(ptr);
auto range_end = pointer_offset(p, len);
auto object_end = alloc.template external_pointer<OnePastEnd>(p);
auto object_end = external_pointer<OnePastEnd>(p);
report_fatal_error(
"Fatal Error!\n{}: \n\trange [{}, {})\n\tallocation [{}, "
"{})\nrange goes beyond allocation by {} bytes \n",
msg,
p,
range_end,
alloc.template external_pointer<Start>(p),
external_pointer<Start>(p),
object_end,
pointer_diff(object_end, range_end));
}
@@ -86,13 +86,16 @@ namespace snmalloc
* The template parameter indicates whether the check should be performed. It
* defaults to true. If it is false, the check will always succeed.
*/
template<bool PerformCheck = true>
template<bool PerformCheck = true, SNMALLOC_CONCEPT(IsConfig) Config = Config>
SNMALLOC_FAST_PATH_INLINE bool check_bounds(const void* ptr, size_t len)
{
if constexpr (PerformCheck)
{
auto& alloc = ThreadAlloc::get();
return alloc.check_bounds(ptr, len);
if (SNMALLOC_LIKELY(Config::is_initialised()))
{
return remaining_bytes(address_cast(ptr)) >= len;
}
return true;
}
else
{

1
src/snmalloc/global/global.h
View File

@@ -1,4 +1,5 @@
#include "bounds_checks.h"
#include "globalalloc.h"
#include "libc.h"
#include "memcpy.h"
#include "scopedalloc.h"

378
src/snmalloc/global/globalalloc.h Normal file
View File

@@ -0,0 +1,378 @@
#pragma once
#include "../mem/mem.h"
#include "threadalloc.h"
namespace snmalloc
{
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
inline static void cleanup_unused()
{
static_assert(
Config_::Options.CoreAllocIsPoolAllocated,
"Global cleanup is available only for pool-allocated configurations");
// Call this periodically to free and coalesce memory allocated by
// allocators that are not currently in use by any thread.
// One atomic operation to extract the stack, another to restore it.
// Handling the message queue for each stack is non-atomic.
auto* first = AllocPool<Config_>::extract();
auto* alloc = first;
decltype(alloc) last;
if (alloc != nullptr)
{
while (alloc != nullptr)
{
alloc->flush();
last = alloc;
alloc = AllocPool<Config_>::extract(alloc);
}
AllocPool<Config_>::restore(first, last);
}
}
/**
If you pass a pointer to a bool, then it returns whether all the
allocators are empty. If you don't pass a pointer to a bool, then will
raise an error all the allocators are not empty.
*/
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
inline static void debug_check_empty(bool* result = nullptr)
{
static_assert(
Config_::Options.CoreAllocIsPoolAllocated,
"Global status is available only for pool-allocated configurations");
// This is a debugging function. It checks that all memory from all
// allocators has been freed.
auto* alloc = AllocPool<Config_>::iterate();
#ifdef SNMALLOC_TRACING
message<1024>("debug check empty: first {}", alloc);
#endif
bool done = false;
bool okay = true;
while (!done)
{
#ifdef SNMALLOC_TRACING
message<1024>("debug_check_empty: Check all allocators!");
#endif
done = true;
alloc = AllocPool<Config_>::iterate();
okay = true;
while (alloc != nullptr)
{
#ifdef SNMALLOC_TRACING
message<1024>("debug check empty: {}", alloc);
#endif
// Check that the allocator has freed all memory.
// repeat the loop if empty caused message sends.
if (alloc->debug_is_empty(&okay))
{
done = false;
#ifdef SNMALLOC_TRACING
message<1024>("debug check empty: sent messages {}", alloc);
#endif
}
#ifdef SNMALLOC_TRACING
message<1024>("debug check empty: okay = {}", okay);
#endif
alloc = AllocPool<Config_>::iterate(alloc);
}
}
if (result != nullptr)
{
*result = okay;
return;
}
// Redo check so abort is on allocator with allocation left.
if (!okay)
{
alloc = AllocPool<Config_>::iterate();
while (alloc != nullptr)
{
alloc->debug_is_empty(nullptr);
alloc = AllocPool<Config_>::iterate(alloc);
}
}
}
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
inline static void debug_in_use(size_t count)
{
static_assert(
Config_::Options.CoreAllocIsPoolAllocated,
"Global status is available only for pool-allocated configurations");
auto alloc = AllocPool<Config_>::iterate();
while (alloc != nullptr)
{
if (alloc->debug_is_in_use())
{
if (count == 0)
{
error("ERROR: allocator in use.");
}
count--;
}
alloc = AllocPool<Config_>::iterate(alloc);
if (count != 0)
{
error("Error: two few allocators in use.");
}
}
}
/**
* Returns the number of remaining bytes in an object.
*
* auto p = (char*)malloc(size)
* remaining_bytes(p + n) == size - n provided n < size
*/
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
size_t remaining_bytes(address_t p)
{
const auto& entry = Config_::Backend::template get_metaentry<true>(p);
auto sizeclass = entry.get_sizeclass();
return snmalloc::remaining_bytes(sizeclass, p);
}
/**
* Returns the byte offset into an object.
*
* auto p = (char*)malloc(size)
* index_in_object(p + n) == n provided n < size
*/
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
static inline size_t index_in_object(address_t p)
{
const auto& entry = Config_::Backend::template get_metaentry<true>(p);
auto sizeclass = entry.get_sizeclass();
return snmalloc::index_in_object(sizeclass, p);
}
enum Boundary
{
/**
* The location of the first byte of this allocation.
*/
Start,
/**
* The location of the last byte of the allocation.
*/
End,
/**
* The location one past the end of the allocation. This is mostly useful
* for bounds checking, where anything less than this value is safe.
*/
OnePastEnd
};
/**
* Returns the Start/End of an object allocated by this allocator
*
* It is valid to pass any pointer, if the object was not allocated
* by this allocator, then it give the start and end as the whole of
* the potential pointer space.
*/
template<
Boundary location = Start,
SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
inline static void* external_pointer(void* p)
{
/*
* Note that:
* * each case uses `pointer_offset`, so that on CHERI, our behaviour is
* monotone with respect to the capability `p`.
*
* * the returned pointer could be outside the CHERI bounds of `p`, and
* thus not something that can be followed.
*
* * we don't use capptr_from_client()/capptr_reveal(), to avoid the
* syntactic clutter. By inspection, `p` flows only to address_cast
* and pointer_offset, and so there's no risk that we follow or act
* to amplify the rights carried by `p`.
*/
if constexpr (location == Start)
{
size_t index = index_in_object<Config_>(address_cast(p));
return pointer_offset(p, 0 - index);
}
else if constexpr (location == End)
{
return pointer_offset(p, remaining_bytes(address_cast(p)) - 1);
}
else
{
return pointer_offset(p, remaining_bytes(address_cast(p)));
}
}
/**
* @brief Get the client meta data for the snmalloc allocation covering this
* pointer.
*/
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
typename Config_::ClientMeta::DataRef get_client_meta_data(void* p)
{
const auto& entry = Config_::Backend::get_metaentry(address_cast(p));
size_t index = slab_index(entry.get_sizeclass(), address_cast(p));
auto* meta_slab = entry.get_slab_metadata();
if (SNMALLOC_UNLIKELY(entry.is_backend_owned()))
{
error("Cannot access meta-data for write for freed memory!");
}
if (SNMALLOC_UNLIKELY(meta_slab == nullptr))
{
error(
"Cannot access meta-data for non-snmalloc object in writable form!");
}
return meta_slab->get_meta_for_object(index);
}
/**
* @brief Get the client meta data for the snmalloc allocation covering this
* pointer.
*/
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
stl::add_const_t<typename Config_::ClientMeta::DataRef>
get_client_meta_data_const(void* p)
{
const auto& entry =
Config_::Backend::template get_metaentry<true>(address_cast(p));
size_t index = slab_index(entry.get_sizeclass(), address_cast(p));
auto* meta_slab = entry.get_slab_metadata();
if (SNMALLOC_UNLIKELY((meta_slab == nullptr) || (entry.is_backend_owned())))
{
static typename Config_::ClientMeta::StorageType null_meta_store{};
return Config_::ClientMeta::get(&null_meta_store, 0);
}
return meta_slab->get_meta_for_object(index);
}
/**
* @brief Checks that the supplied size of the allocation matches the size
* snmalloc believes the allocation is. Only performs the check if
* mitigations(sanity_checks)
* is enabled.
*/
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
SNMALLOC_FAST_PATH_INLINE void check_size(void* p, size_t size)
{
if constexpr (mitigations(sanity_checks))
{
const auto& entry = Config_::Backend::get_metaentry(address_cast(p));
if (!entry.is_owned())
return;
size = size == 0 ? 1 : size;
auto sc = size_to_sizeclass_full(size);
auto pm_sc = entry.get_sizeclass();
auto rsize = sizeclass_full_to_size(sc);
auto pm_size = sizeclass_full_to_size(pm_sc);
snmalloc_check_client(
mitigations(sanity_checks),
(sc == pm_sc) || (p == nullptr),
"Dealloc rounded size mismatch: {} != {}",
rsize,
pm_size);
}
else
UNUSED(p, size);
}
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
SNMALLOC_FAST_PATH_INLINE size_t alloc_size(const void* p_raw)
{
const auto& entry = Config_::Backend::get_metaentry(address_cast(p_raw));
if (SNMALLOC_UNLIKELY(
!SecondaryAllocator::pass_through && !entry.is_owned() &&
p_raw != nullptr))
return SecondaryAllocator::alloc_size(p_raw);
// TODO What's the domestication policy here? At the moment we just
// probe the pagemap with the raw address, without checks. There could
// be implicit domestication through the `Config::Pagemap` or
// we could just leave well enough alone.
// Note that alloc_size should return 0 for nullptr.
// Other than nullptr, we know the system will be initialised as it must
// be called with something we have already allocated.
//
// To handle this case we require the uninitialised pagemap contain an
// entry for the first chunk of memory, that states it represents a
// large object, so we can pull the check for null off the fast path.
return sizeclass_full_to_size(entry.get_sizeclass());
}
template<size_t size, ZeroMem zero_mem = NoZero, size_t align = 1>
SNMALLOC_FAST_PATH_INLINE void* alloc()
{
return ThreadAlloc::get().alloc<zero_mem>(aligned_size(align, size));
}
template<ZeroMem zero_mem = NoZero, size_t align = 1>
SNMALLOC_FAST_PATH_INLINE void* alloc(size_t size)
{
return ThreadAlloc::get().alloc<zero_mem>(aligned_size(align, size));
}
template<ZeroMem zero_mem = NoZero>
SNMALLOC_FAST_PATH_INLINE void* alloc_aligned(size_t align, size_t size)
{
return ThreadAlloc::get().alloc<zero_mem>(aligned_size(align, size));
}
SNMALLOC_FAST_PATH_INLINE void dealloc(void* p)
{
ThreadAlloc::get().dealloc(p);
}
SNMALLOC_FAST_PATH_INLINE void dealloc(void* p, size_t size)
{
check_size(p, size);
ThreadAlloc::get().dealloc(p);
}
template<size_t size>
SNMALLOC_FAST_PATH_INLINE void dealloc(void* p)
{
check_size(p, size);
ThreadAlloc::get().dealloc(p);
}
SNMALLOC_FAST_PATH_INLINE void dealloc(void* p, size_t size, size_t align)
{
auto rsize = aligned_size(align, size);
check_size(p, rsize);
ThreadAlloc::get().dealloc(p);
}
SNMALLOC_FAST_PATH_INLINE void debug_teardown()
{
return ThreadAlloc::get().teardown();
}
template<SNMALLOC_CONCEPT(IsConfig) Config_ = Config>
SNMALLOC_FAST_PATH_INLINE bool is_owned(void* p)
{
const auto& entry = Config_::Backend::get_metaentry(address_cast(p));
return entry.is_owned();
}
} // namespace snmalloc

52
src/snmalloc/global/libc.h
View File

@@ -1,6 +1,6 @@
#pragma once
#include "threadalloc.h"
#include "globalalloc.h"
#include <errno.h>
#include <string.h>
@@ -21,22 +21,22 @@ namespace snmalloc::libc
inline void* __malloc_end_pointer(void* ptr)
{
return ThreadAlloc::get().external_pointer<OnePastEnd>(ptr);
return snmalloc::external_pointer<OnePastEnd>(ptr);
}
SNMALLOC_FAST_PATH_INLINE void* malloc(size_t size)
{
return ThreadAlloc::get().alloc(size);
return snmalloc::alloc(size);
}
SNMALLOC_FAST_PATH_INLINE void free(void* ptr)
{
ThreadAlloc::get().dealloc(ptr);
dealloc(ptr);
}
SNMALLOC_FAST_PATH_INLINE void free_sized(void* ptr, size_t size)
{
ThreadAlloc::get().dealloc(ptr, size);
dealloc(ptr, size);
}
SNMALLOC_FAST_PATH_INLINE void* calloc(size_t nmemb, size_t size)
@@ -47,27 +47,19 @@ namespace snmalloc::libc
{
return set_error();
}
return ThreadAlloc::get().alloc<ZeroMem::YesZero>(sz);
return alloc<ZeroMem::YesZero>(sz);
}
SNMALLOC_FAST_PATH_INLINE void* realloc(void* ptr, size_t size)
{
auto& a = ThreadAlloc::get();
size_t sz = a.alloc_size(ptr);
size_t sz = alloc_size(ptr);
// Keep the current allocation if the given size is in the same sizeclass.
if (sz == round_size(size))
{
#ifdef SNMALLOC_PASS_THROUGH
// snmallocs alignment guarantees can be broken by realloc in pass-through
// this is not exercised, by existing clients, but is tested.
if (pointer_align_up(ptr, natural_alignment(size)) == ptr)
return ptr;
#else
return ptr;
#endif
}
void* p = a.alloc(size);
void* p = alloc(size);
if (SNMALLOC_LIKELY(p != nullptr))
{
sz = bits::min(size, sz);
@@ -78,11 +70,11 @@ namespace snmalloc::libc
SNMALLOC_ASSUME(ptr != nullptr);
::memcpy(p, ptr, sz);
}
a.dealloc(ptr);
dealloc(ptr);
}
else if (SNMALLOC_LIKELY(size == 0))
{
a.dealloc(ptr);
dealloc(ptr);
}
else
{
@@ -93,7 +85,7 @@ namespace snmalloc::libc
inline size_t malloc_usable_size(const void* ptr)
{
return ThreadAlloc::get().alloc_size(ptr);
return alloc_size(ptr);
}
inline void* reallocarray(void* ptr, size_t nmemb, size_t size)
@@ -110,7 +102,6 @@ namespace snmalloc::libc
inline int reallocarr(void* ptr_, size_t nmemb, size_t size)
{
int err = errno;
auto& a = ThreadAlloc::get();
bool overflow = false;
size_t sz = bits::umul(size, nmemb, overflow);
if (SNMALLOC_UNLIKELY(sz == 0))
@@ -124,13 +115,13 @@ namespace snmalloc::libc
}
void** ptr = reinterpret_cast<void**>(ptr_);
void* p = a.alloc(sz);
void* p = alloc(sz);
if (SNMALLOC_UNLIKELY(p == nullptr))
{
return set_error_and_return(ENOMEM);
}
sz = bits::min(sz, a.alloc_size(*ptr));
sz = bits::min(sz, alloc_size(*ptr));
SNMALLOC_ASSUME(*ptr != nullptr || sz == 0);
// Guard memcpy as GCC is assuming not nullptr for ptr after the memcpy
@@ -138,7 +129,7 @@ namespace snmalloc::libc
if (SNMALLOC_UNLIKELY(sz != 0))
::memcpy(p, *ptr, sz);
errno = err;
a.dealloc(*ptr);
dealloc(*ptr);
*ptr = p;
return 0;
}
@@ -150,7 +141,7 @@ namespace snmalloc::libc
return set_error(EINVAL);
}
return malloc(aligned_size(alignment, size));
return alloc_aligned(alignment, size);
}
inline void* aligned_alloc(size_t alignment, size_t size)
@@ -175,17 +166,4 @@ namespace snmalloc::libc
*memptr = p;
return 0;
}
inline typename snmalloc::Alloc::Config::ClientMeta::DataRef
get_client_meta_data(void* p)
{
return ThreadAlloc::get().get_client_meta_data(p);
}
inline stl::add_const_t<typename snmalloc::Alloc::Config::ClientMeta::DataRef>
get_client_meta_data_const(void* p)
{
return ThreadAlloc::get().get_client_meta_data_const(p);
}
} // namespace snmalloc::libc

8
src/snmalloc/global/scopedalloc.h
View File

@@ -15,12 +15,13 @@ namespace snmalloc
* This does not depend on thread-local storage working, so can be used for
* bootstrapping.
*/
template<typename SAlloc = Alloc>
struct ScopedAllocator
{
/**
* The allocator that this wrapper will use.
*/
Alloc alloc;
SAlloc alloc;
/**
* Constructor. Claims an allocator from the global pool
@@ -66,7 +67,7 @@ namespace snmalloc
* Arrow operator, allows methods exposed by `Alloc` to be called on the
* wrapper.
*/
Alloc* operator->()
SAlloc* operator->()
{
return &alloc;
}
@@ -76,7 +77,8 @@ namespace snmalloc
* Returns a new scoped allocator. When the `ScopedAllocator` goes out of
* scope, the underlying `Alloc` will be returned to the pool.
*/
inline ScopedAllocator get_scoped_allocator()
template<typename SAlloc = Alloc>
inline ScopedAllocator<SAlloc> get_scoped_allocator()
{
return {};
}

89
src/snmalloc/mem/external_alloc.h
View File

@@ -1,89 +0,0 @@
#pragma once
#ifdef SNMALLOC_PASS_THROUGH
# if defined(__HAIKU__)
# define _GNU_SOURCE
# endif
# include <stdlib.h>
# if defined(_WIN32) //|| defined(__APPLE__)
# error "Pass through not supported on this platform"
// The Windows aligned allocation API is not capable of supporting the
// snmalloc API Apple was not providing aligned memory in some tests.
# else
// Defines malloc_size for the platform.
# if defined(_WIN32)
namespace snmalloc::external_alloc
{
inline size_t malloc_usable_size(void* ptr)
{
return _msize(ptr);
}
}
# elif defined(__APPLE__)
# include <malloc/malloc.h>
namespace snmalloc::external_alloc
{
inline size_t malloc_usable_size(void* ptr)
{
return malloc_size(ptr);
}
}
# elif defined(__linux__) || defined(__HAIKU__)
# include <malloc.h>
namespace snmalloc::external_alloc
{
using ::malloc_usable_size;
}
# elif defined(__sun) || defined(__NetBSD__) || defined(__OpenBSD__)
namespace snmalloc::external_alloc
{
using ::malloc_usable_size;
}
# elif defined(__FreeBSD__)
# include <malloc_np.h>
namespace snmalloc::external_alloc
{
using ::malloc_usable_size;
}
# elif defined(__DragonFly__)
namespace snmalloc::external_alloc
{
using ::malloc_usable_size;
}
# else
# error Define malloc size macro for this platform.
# endif
namespace snmalloc::external_alloc
{
inline void* aligned_alloc(size_t alignment, size_t size)
{
// TSAN complains if allocation is large than this.
if constexpr (bits::BITS == 64)
{
if (size >= 0x10000000000)
{
errno = ENOMEM;
return nullptr;
}
}
if (alignment < sizeof(void*))
alignment = sizeof(void*);
void* result;
int err = posix_memalign(&result, alignment, size);
if (err != 0)
{
errno = err;
result = nullptr;
}
return result;
}
using ::free;
}
# endif
#endif

2
src/snmalloc/mem/freelist_queue.h
View File

@@ -52,7 +52,7 @@ namespace snmalloc
{
SNMALLOC_ASSERT(
(address_cast(front.load()) == address_cast(&stub)) ||
(back != nullptr));
(back.load() != nullptr));
}
void init()

137
src/snmalloc/mem/globalalloc.h
View File

@@ -1,137 +0,0 @@
#pragma once
#include "../ds_core/ds_core.h"
#include "localalloc.h"
namespace snmalloc
{
template<SNMALLOC_CONCEPT(IsConfig) Config>
inline static void cleanup_unused()
{
#ifndef SNMALLOC_PASS_THROUGH
static_assert(
Config::Options.CoreAllocIsPoolAllocated,
"Global cleanup is available only for pool-allocated configurations");
// Call this periodically to free and coalesce memory allocated by
// allocators that are not currently in use by any thread.
// One atomic operation to extract the stack, another to restore it.
// Handling the message queue for each stack is non-atomic.
auto* first = AllocPool<Config>::extract();
auto* alloc = first;
decltype(alloc) last;
if (alloc != nullptr)
{
while (alloc != nullptr)
{
alloc->flush();
last = alloc;
alloc = AllocPool<Config>::extract(alloc);
}
AllocPool<Config>::restore(first, last);
}
#endif
}
/**
If you pass a pointer to a bool, then it returns whether all the
allocators are empty. If you don't pass a pointer to a bool, then will
raise an error all the allocators are not empty.
*/
template<SNMALLOC_CONCEPT(IsConfig) Config>
inline static void debug_check_empty(bool* result = nullptr)
{
#ifndef SNMALLOC_PASS_THROUGH
static_assert(
Config::Options.CoreAllocIsPoolAllocated,
"Global status is available only for pool-allocated configurations");
// This is a debugging function. It checks that all memory from all
// allocators has been freed.
auto* alloc = AllocPool<Config>::iterate();
# ifdef SNMALLOC_TRACING
message<1024>("debug check empty: first {}", alloc);
# endif
bool done = false;
bool okay = true;
while (!done)
{
# ifdef SNMALLOC_TRACING
message<1024>("debug_check_empty: Check all allocators!");
# endif
done = true;
alloc = AllocPool<Config>::iterate();
okay = true;
while (alloc != nullptr)
{
# ifdef SNMALLOC_TRACING
message<1024>("debug check empty: {}", alloc);
# endif
// Check that the allocator has freed all memory.
// repeat the loop if empty caused message sends.
if (alloc->debug_is_empty(&okay))
{
done = false;
# ifdef SNMALLOC_TRACING
message<1024>("debug check empty: sent messages {}", alloc);
# endif
}
# ifdef SNMALLOC_TRACING
message<1024>("debug check empty: okay = {}", okay);
# endif
alloc = AllocPool<Config>::iterate(alloc);
}
}
if (result != nullptr)
{
*result = okay;
return;
}
// Redo check so abort is on allocator with allocation left.
if (!okay)
{
alloc = AllocPool<Config>::iterate();
while (alloc != nullptr)
{
alloc->debug_is_empty(nullptr);
alloc = AllocPool<Config>::iterate(alloc);
}
}
#else
UNUSED(result);
#endif
}
template<SNMALLOC_CONCEPT(IsConfig) Config>
inline static void debug_in_use(size_t count)
{
static_assert(
Config::Options.CoreAllocIsPoolAllocated,
"Global status is available only for pool-allocated configurations");
auto alloc = AllocPool<Config>::iterate();
while (alloc != nullptr)
{
if (alloc->debug_is_in_use())
{
if (count == 0)
{
error("ERROR: allocator in use.");
}
count--;
}
alloc = AllocPool<Config>::iterate(alloc);
if (count != 0)
{
error("Error: two few allocators in use.");
}
}
}
} // namespace snmalloc

453
src/snmalloc/mem/localalloc.h
View File

@@ -18,34 +18,12 @@
#include "pool.h"
#include "remotecache.h"
#include "sizeclasstable.h"
#ifdef SNMALLOC_PASS_THROUGH
# include "external_alloc.h"
#endif
#include "snmalloc/stl/utility.h"
#include <string.h>
namespace snmalloc
{
enum Boundary
{
/**
* The location of the first byte of this allocation.
*/
Start,
/**
* The location of the last byte of the allocation.
*/
End,
/**
* The location one past the end of the allocation. This is mostly useful
* for bounds checking, where anything less than this value is safe.
*/
OnePastEnd
};
/**
* A local allocator contains the fast-path allocation routines and
* encapsulates all of the behaviour of an allocator that is local to some
@@ -270,14 +248,6 @@ namespace snmalloc
return local_cache.template alloc<zero_mem>(domesticate, size, slowpath);
}
/**
* Send all remote deallocation to other threads.
*/
void post_remote_cache()
{
core_alloc->post();
}
/**
* Slow path for deallocation we do not have space for this remote
* deallocation. This could be because,
@@ -296,12 +266,13 @@ namespace snmalloc
message<1024>(
"Remote dealloc post {} ({}, {})",
p.unsafe_ptr(),
alloc_size(p.unsafe_ptr()),
sizeclass_full_to_size(entry.get_sizeclass()),
address_cast(entry.get_slab_metadata()));
#endif
local_cache.remote_dealloc_cache.template dealloc<sizeof(CoreAlloc)>(
entry.get_slab_metadata(), p, &local_cache.entropy);
post_remote_cache();
core_alloc->post();
return;
}
@@ -317,15 +288,6 @@ namespace snmalloc
p);
}
/**
* Abstracts access to the message queue to handle different
* layout configurations of the allocator.
*/
auto& message_queue()
{
return local_cache.remote_allocator;
}
/**
* Call `Config::is_initialised()` if it is implemented,
* unconditionally returns true otherwise.
@@ -449,16 +411,6 @@ namespace snmalloc
template<ZeroMem zero_mem = NoZero>
SNMALLOC_FAST_PATH ALLOCATOR void* alloc(size_t size)
{
#ifdef SNMALLOC_PASS_THROUGH
// snmalloc guarantees a lot of alignment, so we can depend on this
// make pass through call aligned_alloc with the alignment snmalloc
// would guarantee.
void* result = external_alloc::aligned_alloc(
natural_alignment(size), round_size(size));
if (zero_mem == YesZero && result != nullptr)
memset(result, 0, size);
return result;
#else
// Perform the - 1 on size, so that zero wraps around and ends up on
// slow path.
if (SNMALLOC_LIKELY(
@@ -470,158 +422,9 @@ namespace snmalloc
}
return capptr_reveal(alloc_not_small<zero_mem>(size));
#endif
}
/**
* Allocate memory of a statically known size.
*/
template<size_t size, ZeroMem zero_mem = NoZero>
SNMALLOC_FAST_PATH ALLOCATOR void* alloc()
{
return alloc<zero_mem>(size);
}
/*
* Many of these tests come with an "or is null" branch that they'd need to
* add if we did them up front. Instead, defer them until we're past the
* point where we know, from the pagemap, or by explicitly testing, that the
* pointer under test is not nullptr.
*/
SNMALLOC_FAST_PATH void dealloc_cheri_checks(void* p)
{
#if defined(__CHERI_PURE_CAPABILITY__)
/*
* Enforce the use of an unsealed capability.
*
* TODO In CHERI+MTE, this, is part of the CAmoCDecVersion instruction;
* elide this test in that world.
*/
snmalloc_check_client(
mitigations(cheri_checks),
!__builtin_cheri_sealed_get(p),
"Sealed capability in deallocation");
/*
* Enforce permissions on the returned pointer. These pointers end up in
* free queues and will be cycled out to clients again, so try to catch
* erroneous behavior now, rather than later.
*
* TODO In the CHERI+MTE case, we must reconstruct the pointer for the
* free queues as part of the discovery of the start of the object (so
* that it has the correct version), and the CAmoCDecVersion call imposes
* its own requirements on the permissions (to ensure that it's at least
* not zero). They are somewhat more lax than we might wish, so this test
* may remain, guarded by SNMALLOC_CHECK_CLIENT, but no explicit
* permissions checks are required in the non-SNMALLOC_CHECK_CLIENT case
* to defend ourselves or other clients against a misbehaving client.
*/
static const size_t reqperm = CHERI_PERM_LOAD | CHERI_PERM_STORE |
CHERI_PERM_LOAD_CAP | CHERI_PERM_STORE_CAP;
snmalloc_check_client(
mitigations(cheri_checks),
(__builtin_cheri_perms_get(p) & reqperm) == reqperm,
"Insufficient permissions on capability in deallocation");
/*
* We check for a valid tag here, rather than in domestication, because
* domestication might be answering a slightly different question, about
* the plausibility of addresses rather than of exact pointers.
*
* TODO Further, in the CHERI+MTE case, the tag check will be implicit in
* a future CAmoCDecVersion instruction, and there should be no harm in
* the lookups we perform along the way to get there. In that world,
* elide this test.
*/
snmalloc_check_client(
mitigations(cheri_checks),
__builtin_cheri_tag_get(p),
"Untagged capability in deallocation");
/*
* Verify that the capability is not zero-length, ruling out the other
* edge case around monotonicity.
*/
snmalloc_check_client(
mitigations(cheri_checks),
__builtin_cheri_length_get(p) > 0,
"Zero-length capability in deallocation");
/*
* At present we check for the pointer also being the start of an
* allocation closer to dealloc; for small objects, that happens in
* dealloc_local_object_fast, either below or *on the far end of message
* receipt*. For large objects, it happens below by directly rounding to
* power of two rather than using the is_start_of_object helper.
* (XXX This does mean that we might end up threading our remote queue
* state somewhere slightly unexpected rather than at the head of an
* object. That is perhaps fine for now?)
*/
/*
* TODO
*
* We could enforce other policies here, including that the length exactly
* match the sizeclass. At present, we bound caps we give for allocations
* to the underlying sizeclass, so even malloc(0) will have a non-zero
* length. Monotonicity would then imply that the pointer must be the
* head of an object (modulo, perhaps, temporal aliasing if we somehow
* introduced phase shifts in heap layout like some allocators do).
*
* If we switched to bounding with upwards-rounded representable bounds
* (c.f., CRRL) rather than underlying object size, then we should,
* instead, in general require plausibility of p_raw by checking that its
* length is nonzero and the snmalloc size class associated with its
* length is the one for the slab in question... except for the added
* challenge of malloc(0). Since 0 rounds up to 0, we might end up
* constructing zero-length caps to hand out, which we would then reject
* upon receipt. Instead, as part of introducing CRRL bounds, we should
* introduce a sizeclass for slabs holding zero-size objects. All told,
* we would want to check that
*
* size_to_sizeclass(length) == entry.get_sizeclass()
*
* I believe a relaxed CRRL test of
*
* length > 0 || (length == sizeclass_to_size(entry.get_sizeclass()))
*
* would also suffice and may be slightly less expensive than the test
* above, at the cost of not catching as many misbehaving clients.
*
* In either case, having bounded by CRRL bounds, we would need to be
* *reconstructing* the capabilities headed to our free lists to be given
* out to clients again; there are many more CRRL classes than snmalloc
* sizeclasses (this is the same reason that we can always get away with
* CSetBoundsExact in capptr_bound). Switching to CRRL bounds, if that's
* ever a thing we want to do, will be easier after we've done the
* plumbing for CHERI+MTE.
*/
/*
* TODO: Unsurprisingly, the CHERI+MTE case once again has something to
* say here. In that world, again, we are certain to be reconstructing
* the capability for the free queue anyway, and so exactly what we wish
* to enforce, length-wise, of the provided capability, is somewhat more
* flexible. Using the provided capability bounds when recoloring memory
* could be a natural way to enforce that it covers the entire object, at
* the cost of a more elaborate recovery story (as we risk aborting with a
* partially recolored object). On non-SNMALLOC_CHECK_CLIENT builds, it
* likely makes sense to just enforce that length > 0 (*not* enforced by
* the CAmoCDecVersion instruction) and say that any authority-bearing
* interior pointer suffices to free the object. I believe that to be an
* acceptable security posture for the allocator and between clients;
* misbehavior is confined to the misbehaving client.
*/
#else
UNUSED(p);
#endif
}
// The domestic pointer with its origin allocator
using DomesticInfo = stl::Pair<capptr::Alloc<void>, const PagemapEntry&>;
// Check whether the raw pointer is owned by snmalloc
SNMALLOC_FAST_PATH_INLINE DomesticInfo get_domestic_info(const void* p_raw)
SNMALLOC_FAST_PATH void dealloc(void* p_raw)
{
#ifdef __CHERI_PURE_CAPABILITY__
/*
@@ -646,22 +449,7 @@ namespace snmalloc
capptr_domesticate<Config>(core_alloc->backend_state_ptr(), p_wild);
const PagemapEntry& entry =
Config::Backend::get_metaentry(address_cast(p_tame));
return {p_tame, entry};
}
// Check if a pointer is domestic to SnMalloc
SNMALLOC_FAST_PATH bool is_snmalloc_owned(const void* p_raw)
{
auto [_, entry] = get_domestic_info(p_raw);
RemoteAllocator* remote = entry.get_remote();
return remote != nullptr;
}
SNMALLOC_FAST_PATH void dealloc(void* p_raw)
{
#ifdef SNMALLOC_PASS_THROUGH
external_alloc::free(p_raw);
#else
/*
* p_tame may be nullptr, even if p_raw/p_wild are not, in the case
* where domestication fails. We exclusively use p_tame below so that
@@ -674,8 +462,6 @@ namespace snmalloc
* well-formedness) of this pointer. The remainder of the logic will
* deal with the object's extent.
*/
auto [p_tame, entry] = get_domestic_info(p_raw);
if (SNMALLOC_LIKELY(local_cache.remote_allocator == entry.get_remote()))
{
dealloc_cheri_checks(p_tame.unsafe_ptr());
@@ -689,8 +475,7 @@ namespace snmalloc
SNMALLOC_SLOW_PATH void
dealloc_remote(const PagemapEntry& entry, capptr::Alloc<void> p_tame)
{
RemoteAllocator* remote = entry.get_remote();
if (SNMALLOC_LIKELY(remote != nullptr))
if (SNMALLOC_LIKELY(entry.is_owned()))
{
dealloc_cheri_checks(p_tame.unsafe_ptr());
@@ -705,13 +490,13 @@ namespace snmalloc
{
local_cache.remote_dealloc_cache.template dealloc<sizeof(CoreAlloc)>(
entry.get_slab_metadata(), p_tame, &local_cache.entropy);
# ifdef SNMALLOC_TRACING
#ifdef SNMALLOC_TRACING
message<1024>(
"Remote dealloc fast {} ({}, {})",
address_cast(p_tame),
alloc_size(p_tame.unsafe_ptr()),
sizeclass_full_to_size(entry.get_sizeclass()),
address_cast(entry.get_slab_metadata()));
# endif
#endif
return;
}
@@ -721,54 +506,14 @@ namespace snmalloc
if (SNMALLOC_LIKELY(p_tame == nullptr))
{
# ifdef SNMALLOC_TRACING
#ifdef SNMALLOC_TRACING
message<1024>("nullptr deallocation");
# endif
#endif
return;
}
dealloc_cheri_checks(p_tame.unsafe_ptr());
SecondaryAllocator::deallocate(p_tame.unsafe_ptr());
#endif
}
void check_size(void* p, size_t size)
{
#ifdef SNMALLOC_PASS_THROUGH
UNUSED(p, size);
#else
if constexpr (mitigations(sanity_checks))
{
if (!is_snmalloc_owned(p))
return;
size = size == 0 ? 1 : size;
auto sc = size_to_sizeclass_full(size);
auto pm_sc =
Config::Backend::get_metaentry(address_cast(p)).get_sizeclass();
auto rsize = sizeclass_full_to_size(sc);
auto pm_size = sizeclass_full_to_size(pm_sc);
snmalloc_check_client(
mitigations(sanity_checks),
(sc == pm_sc) || (p == nullptr),
"Dealloc rounded size mismatch: {} != {}",
rsize,
pm_size);
}
else
UNUSED(p, size);
#endif
}
SNMALLOC_FAST_PATH void dealloc(void* p, size_t s)
{
check_size(p, s);
dealloc(p);
}
template<size_t size>
SNMALLOC_FAST_PATH void dealloc(void* p)
{
check_size(p, size);
dealloc(p);
}
void teardown()
@@ -783,182 +528,6 @@ namespace snmalloc
}
}
SNMALLOC_FAST_PATH size_t alloc_size(const void* p_raw)
{
#ifdef SNMALLOC_PASS_THROUGH
return external_alloc::malloc_usable_size(const_cast<void*>(p_raw));
#else
if (
!SecondaryAllocator::pass_through && !is_snmalloc_owned(p_raw) &&
p_raw != nullptr)
return SecondaryAllocator::alloc_size(p_raw);
// TODO What's the domestication policy here? At the moment we just
// probe the pagemap with the raw address, without checks. There could
// be implicit domestication through the `Config::Pagemap` or
// we could just leave well enough alone.
// Note that alloc_size should return 0 for nullptr.
// Other than nullptr, we know the system will be initialised as it must
// be called with something we have already allocated.
//
// To handle this case we require the uninitialised pagemap contain an
// entry for the first chunk of memory, that states it represents a
// large object, so we can pull the check for null off the fast path.
const PagemapEntry& entry =
Config::Backend::get_metaentry(address_cast(p_raw));
return sizeclass_full_to_size(entry.get_sizeclass());
#endif
}
/**
* Returns the Start/End of an object allocated by this allocator
*
* It is valid to pass any pointer, if the object was not allocated
* by this allocator, then it give the start and end as the whole of
* the potential pointer space.
*/
template<Boundary location = Start>
void* external_pointer(void* p)
{
/*
* Note that:
* * each case uses `pointer_offset`, so that on CHERI, our behaviour is
* monotone with respect to the capability `p`.
*
* * the returned pointer could be outside the CHERI bounds of `p`, and
* thus not something that can be followed.
*
* * we don't use capptr_from_client()/capptr_reveal(), to avoid the
* syntactic clutter. By inspection, `p` flows only to address_cast
* and pointer_offset, and so there's no risk that we follow or act
* to amplify the rights carried by `p`.
*/
if constexpr (location == Start)
{
size_t index = index_in_object(address_cast(p));
return pointer_offset(p, 0 - index);
}
else if constexpr (location == End)
{
return pointer_offset(p, remaining_bytes(address_cast(p)) - 1);
}
else
{
return pointer_offset(p, remaining_bytes(address_cast(p)));
}
}
/**
* @brief Get the client meta data for the snmalloc allocation covering this
* pointer.
*/
typename Config::ClientMeta::DataRef get_client_meta_data(void* p)
{
const PagemapEntry& entry =
Config::Backend::get_metaentry(address_cast(p));
size_t index = slab_index(entry.get_sizeclass(), address_cast(p));
auto* meta_slab = entry.get_slab_metadata();
if (SNMALLOC_UNLIKELY(entry.is_backend_owned()))
{
error("Cannot access meta-data for write for freed memory!");
}
if (SNMALLOC_UNLIKELY(meta_slab == nullptr))
{
error(
"Cannot access meta-data for non-snmalloc object in writable form!");
}
return meta_slab->get_meta_for_object(index);
}
/**
* @brief Get the client meta data for the snmalloc allocation covering this
* pointer.
*/
stl::add_const_t<typename Config::ClientMeta::DataRef>
get_client_meta_data_const(void* p)
{
const PagemapEntry& entry =
Config::Backend::template get_metaentry<true>(address_cast(p));
size_t index = slab_index(entry.get_sizeclass(), address_cast(p));
auto* meta_slab = entry.get_slab_metadata();
if (SNMALLOC_UNLIKELY(
(meta_slab == nullptr) || (entry.is_backend_owned())))
{
static typename Config::ClientMeta::StorageType null_meta_store{};
return Config::ClientMeta::get(&null_meta_store, 0);
}
return meta_slab->get_meta_for_object(index);
}
/**
* Returns the number of remaining bytes in an object.
*
* auto p = (char*)malloc(size)
* remaining_bytes(p + n) == size - n provided n < size
*/
size_t remaining_bytes(address_t p)
{
#ifndef SNMALLOC_PASS_THROUGH
const PagemapEntry& entry =
Config::Backend::template get_metaentry<true>(p);
auto sizeclass = entry.get_sizeclass();
return snmalloc::remaining_bytes(sizeclass, p);
#else
constexpr address_t mask = static_cast<address_t>(-1);
constexpr bool is_signed = mask < 0;
constexpr address_t sign_bit =
bits::one_at_bit<address_t>(CHAR_BIT * sizeof(address_t) - 1);
if constexpr (is_signed)
{
return (mask ^ sign_bit) - p;
}
else
{
return mask - p;
}
#endif
}
bool check_bounds(const void* p, size_t s)
{
if (SNMALLOC_LIKELY(Config::is_initialised()))
{
return remaining_bytes(address_cast(p)) >= s;
}
return true;
}
/**
* Returns the byte offset into an object.
*
* auto p = (char*)malloc(size)
* index_in_object(p + n) == n provided n < size
*/
size_t index_in_object(address_t p)
{
#ifndef SNMALLOC_PASS_THROUGH
const PagemapEntry& entry =
Config::Backend::template get_metaentry<true>(p);
auto sizeclass = entry.get_sizeclass();
return snmalloc::index_in_object(sizeclass, p);
#else
return reinterpret_cast<size_t>(p);
#endif
}
/**
* Accessor, returns the local cache. If embedding code is allocating the
* core allocator for use by this local allocator then it needs to access

2
src/snmalloc/mem/mem.h
View File

@@ -2,9 +2,7 @@
#include "backend_wrappers.h"
#include "corealloc.h"
#include "entropy.h"
#include "external_alloc.h"
#include "freelist.h"
#include "globalalloc.h"
#include "localalloc.h"
#include "localcache.h"
#include "metadata.h"

14
src/snmalloc/mem/metadata.h
View File

@@ -207,6 +207,15 @@ namespace snmalloc
get_remote_and_sizeclass()));
}
/**
* Returns true if this memory is owned by snmalloc. Some backend memory
* may return false, but all frontend memory will return true.
*/
[[nodiscard]] SNMALLOC_FAST_PATH bool is_owned() const
{
return get_remote() != nullptr;
}
/**
* Return the sizeclass.
*
@@ -708,9 +717,8 @@ namespace snmalloc
* Ensure that the template parameter is valid.
*/
static_assert(
stl::is_convertible_v<SlabMetadataType, FrontendSlabMetadata_Trait>,
"The front end requires that the back end provides slab metadata that is "
"compatible with the front-end's structure");
stl::is_base_of_v<FrontendSlabMetadata_Trait, SlabMetadataType>,
"Template should be a subclass of FrontendSlabMetadata");
public:
using SlabMetadata = SlabMetadataType;

47
src/snmalloc/override/jemalloc_compat.cc
View File

@@ -143,11 +143,11 @@ extern "C"
}
if (f.should_zero())
{
*ptr = ThreadAlloc::get().alloc<ZeroMem::YesZero>(size);
*ptr = alloc<ZeroMem::YesZero>(size);
}
else
{
*ptr = ThreadAlloc::get().alloc(size);
*ptr = alloc(size);
}
return (*ptr != nullptr) ? allocm_success : allocm_err_oom;
}
@@ -164,12 +164,11 @@ extern "C"
void** ptr, size_t* rsize, size_t size, size_t extra, int flags)
{
auto f = JEMallocFlags(flags);
auto alloc_size = f.aligned_size(size);
auto asize = f.aligned_size(size);
auto& a = ThreadAlloc::get();
size_t sz = a.alloc_size(*ptr);
size_t sz = alloc_size(*ptr);
// Keep the current allocation if the given size is in the same sizeclass.
if (sz == round_size(alloc_size))
if (sz == round_size(asize))
{
if (rsize != nullptr)
{
@@ -185,25 +184,24 @@ extern "C"
if (SIZE_MAX - size > extra)
{
alloc_size = f.aligned_size(size + extra);
asize = f.aligned_size(size + extra);
}
void* p =
f.should_zero() ? a.alloc<YesZero>(alloc_size) : a.alloc(alloc_size);
void* p = f.should_zero() ? alloc<YesZero>(asize) : alloc(asize);
if (SNMALLOC_LIKELY(p != nullptr))
{
sz = bits::min(alloc_size, sz);
sz = bits::min(asize, sz);
// Guard memcpy as GCC is assuming not nullptr for ptr after the memcpy
// otherwise.
if (sz != 0)
{
memcpy(p, *ptr, sz);
}
a.dealloc(*ptr);
dealloc(*ptr);
*ptr = p;
if (rsize != nullptr)
{
*rsize = alloc_size;
*rsize = asize;
}
return allocm_success;
}
@@ -217,7 +215,7 @@ extern "C"
*/
int SNMALLOC_NAME_MANGLE(sallocm)(const void* ptr, size_t* rsize, int)
{
*rsize = ThreadAlloc::get().alloc_size(ptr);
*rsize = alloc_size(ptr);
return allocm_success;
}
@@ -228,7 +226,7 @@ extern "C"
*/
int SNMALLOC_NAME_MANGLE(dallocm)(void* ptr, int)
{
ThreadAlloc::get().dealloc(ptr);
dealloc(ptr);
return allocm_success;
}
@@ -257,9 +255,9 @@ extern "C"
size = f.aligned_size(size);
if (f.should_zero())
{
return ThreadAlloc::get().alloc<ZeroMem::YesZero>(size);
return alloc<ZeroMem::YesZero>(size);
}
return ThreadAlloc::get().alloc(size);
return alloc(size);
}
/**
@@ -274,8 +272,7 @@ extern "C"
auto f = JEMallocFlags(flags);
size = f.aligned_size(size);
auto& a = ThreadAlloc::get();
size_t sz = round_size(a.alloc_size(ptr));
size_t sz = round_size(alloc_size(ptr));
// Keep the current allocation if the given size is in the same sizeclass.
if (sz == size)
{
@@ -292,7 +289,7 @@ extern "C"
// allocations, because we get zeroed memory from the PAL and don't zero it
// twice. This is not profiled and so should be considered for refactoring
// if anyone cares about the performance of these APIs.
void* p = f.should_zero() ? a.alloc<YesZero>(size) : a.alloc(size);
void* p = f.should_zero() ? alloc<YesZero>(size) : alloc(size);
if (SNMALLOC_LIKELY(p != nullptr))
{
sz = bits::min(size, sz);
@@ -300,7 +297,7 @@ extern "C"
// otherwise.
if (sz != 0)
memcpy(p, ptr, sz);
a.dealloc(ptr);
dealloc(ptr);
}
return p;
}
@@ -313,8 +310,7 @@ extern "C"
*/
size_t SNMALLOC_NAME_MANGLE(xallocx)(void* ptr, size_t, size_t, int)
{
auto& a = ThreadAlloc::get();
return a.alloc_size(ptr);
return alloc_size(ptr);
}
/**
@@ -323,8 +319,7 @@ extern "C"
*/
size_t SNMALLOC_NAME_MANGLE(sallocx)(const void* ptr, int)
{
auto& a = ThreadAlloc::get();
return a.alloc_size(ptr);
return alloc_size(ptr);
}
/**
@@ -334,7 +329,7 @@ extern "C"
*/
void SNMALLOC_NAME_MANGLE(dallocx)(void* ptr, int)
{
ThreadAlloc::get().dealloc(ptr);
dealloc(ptr);
}
/**
@@ -347,7 +342,7 @@ extern "C"
*/
void SNMALLOC_NAME_MANGLE(sdallocx)(void* ptr, size_t, int)
{
ThreadAlloc::get().dealloc(ptr);
dealloc(ptr);
}
/**

2
src/snmalloc/override/malloc.cc
View File

@@ -109,7 +109,7 @@ extern "C"
#if __has_include(<features.h>)
# include <features.h>
#endif
#if defined(__GLIBC__) && !defined(SNMALLOC_PASS_THROUGH)
#if defined(__GLIBC__)
// glibc uses these hooks to replace malloc.
// This is required when RTL_DEEPBIND is used and the library is
// LD_PRELOADed.

12
src/snmalloc/override/rust.cc
View File

@@ -12,19 +12,19 @@ using namespace snmalloc;
extern "C" SNMALLOC_EXPORT void*
SNMALLOC_NAME_MANGLE(rust_alloc)(size_t alignment, size_t size)
{
return ThreadAlloc::get().alloc(aligned_size(alignment, size));
return alloc(aligned_size(alignment, size));
}
extern "C" SNMALLOC_EXPORT void*
SNMALLOC_NAME_MANGLE(rust_alloc_zeroed)(size_t alignment, size_t size)
{
return ThreadAlloc::get().alloc<YesZero>(aligned_size(alignment, size));
return alloc<YesZero>(aligned_size(alignment, size));
}
extern "C" SNMALLOC_EXPORT void
SNMALLOC_NAME_MANGLE(rust_dealloc)(void* ptr, size_t alignment, size_t size)
{
ThreadAlloc::get().dealloc(ptr, aligned_size(alignment, size));
dealloc(ptr, aligned_size(alignment, size));
}
extern "C" SNMALLOC_EXPORT void* SNMALLOC_NAME_MANGLE(rust_realloc)(
@@ -36,11 +36,11 @@ extern "C" SNMALLOC_EXPORT void* SNMALLOC_NAME_MANGLE(rust_realloc)(
size_to_sizeclass_full(aligned_old_size).raw() ==
size_to_sizeclass_full(aligned_new_size).raw())
return ptr;
void* p = ThreadAlloc::get().alloc(aligned_new_size);
void* p = alloc(aligned_new_size);
if (p)
{
memcpy(p, ptr, old_size < new_size ? old_size : new_size);
ThreadAlloc::get().dealloc(ptr, aligned_old_size);
dealloc(ptr, aligned_old_size);
}
return p;
}
@@ -55,5 +55,5 @@ extern "C" SNMALLOC_EXPORT void SNMALLOC_NAME_MANGLE(rust_statistics)(
extern "C" SNMALLOC_EXPORT size_t
SNMALLOC_NAME_MANGLE(rust_usable_size)(const void* ptr)
{
return ThreadAlloc::get().alloc_size(ptr);
return alloc_size(ptr);
}

3
src/snmalloc/pal/pal_ds.h
View File

@@ -38,7 +38,8 @@ namespace snmalloc
/**
* Applies function to all the elements of the list
*/
void apply_all(function_ref<void(T*)> func)
template<typename F>
void apply_all(F func)
{
T* curr = elements;
while (curr != nullptr)

10
src/snmalloc/snmalloc.h
View File

@@ -6,19 +6,19 @@
// Provides the global configuration for the snmalloc implementation.
#include "backend/globalconfig.h"
namespace snmalloc
{
// If you define SNMALLOC_PROVIDE_OWN_CONFIG then you must provide your own
// definition of `snmalloc::Alloc` before including any files that include
// `snmalloc.h` or consume the global allocation APIs.
#ifndef SNMALLOC_PROVIDE_OWN_CONFIG
namespace snmalloc
{
using Config = snmalloc::StandardConfigClientMeta<NoClientMetaDataProvider>;
#endif
/**
* Create allocator type for this configuration.
*/
using Alloc = snmalloc::LocalAllocator<
snmalloc::StandardConfigClientMeta<NoClientMetaDataProvider>>;
using Alloc = snmalloc::LocalAllocator<Config>;
} // namespace snmalloc
#endif
// User facing API surface, needs to know what `Alloc` is.
#include "snmalloc_front.h"

6
src/snmalloc/stl/cxx/type_traits.h
View File

@@ -13,19 +13,14 @@ namespace snmalloc
using std::bool_constant;
using std::conditional;
using std::conditional_t;
using std::decay;
using std::decay_t;
using std::enable_if;
using std::enable_if_t;
using std::false_type;
using std::has_unique_object_representations_v;
using std::integral_constant;
using std::is_array_v;
using std::is_base_of_v;
using std::is_convertible_v;
using std::is_copy_assignable_v;
using std::is_copy_constructible_v;
using std::is_function_v;
using std::is_integral;
using std::is_integral_v;
using std::is_move_assignable_v;
@@ -37,7 +32,6 @@ namespace snmalloc
using std::remove_const_t;
using std::remove_cv;
using std::remove_cv_t;
using std::remove_extent_t;
using std::remove_reference;
using std::remove_reference_t;
using std::true_type;

90
src/snmalloc/stl/gnu/type_traits.h
View File

@@ -231,90 +231,6 @@ namespace snmalloc
template<class T>
using remove_reference_t = typename remove_reference<T>::type;
/**
* add_pointer
*/
#if __has_builtin(__add_pointer)
template<class T>
using add_pointer_t = __add_pointer(T);
#else
template<class T>
auto __add_pointer_impl(int) -> type_identity<remove_reference_t<T>*>;
template<class T>
auto __add_pointer_impl(...) -> type_identity<T>;
template<class T>
struct add_pointer : decltype(__add_pointer_impl<T>(0))
{};
template<class T>
using add_pointer_t = typename add_pointer<T>::type;
#endif
/**
* is_array
*/
template<class T>
inline constexpr bool is_array_v = __is_array(T);
/**
* is_function
*/
template<typename T>
inline constexpr bool is_function_v = __is_function(T);
/**
* remove_extent
*/
#if __has_builtin(__remove_extent)
template<class T>
using remove_extent_t = __remove_extent(T);
#else
template<class T>
struct remove_extent
{
using type = T;
};
template<class T>
struct remove_extent<T[]>
{
using type = T;
};
template<class T, size_t N>
struct remove_extent<T[N]>
{
using type = T;
};
template<class T>
using remove_extent_t = typename remove_extent<T>::type;
#endif
/**
* decay
*/
#if __has_builtin(__decay)
template<class T>
using decay_t = __decay(T);
#else
template<class T>
class decay
{
using U = remove_reference_t<T>;
public:
using type = conditional_t<
is_array_v<U>,
add_pointer_t<remove_extent_t<U>>,
conditional_t<is_function_v<U>, add_pointer_t<U>, remove_cv_t<U>>>;
};
template<class T>
using decay_t = typename decay<T>::type;
#endif
/**
* is_copy_assignable
*/
@@ -343,12 +259,6 @@ namespace snmalloc
inline constexpr bool is_move_constructible_v =
__is_constructible(T, add_rvalue_reference_t<T>);
/**
* is_convertible
*/
template<class From, class To>
inline constexpr bool is_convertible_v = __is_convertible(From, To);
/**
* is_base_of
*/

2
src/test/func/cheri/cheri.cc
View File

@@ -1,6 +1,6 @@
#include <iostream>
#if defined(SNMALLOC_PASS_THROUGH) || !defined(__CHERI_PURE_CAPABILITY__)
#if !defined(__CHERI_PURE_CAPABILITY__)
// This test does not make sense in pass-through or w/o CHERI
int main()
{

15
src/test/func/client_meta/client_meta.cc
View File

@@ -14,8 +14,8 @@
namespace snmalloc
{
// Create an allocator that stores an std::atomic<size_t>> per allocation.
using Alloc = snmalloc::LocalAllocator<snmalloc::StandardConfigClientMeta<
ArrayClientMetaDataProvider<std::atomic<size_t>>>>;
using Config = snmalloc::StandardConfigClientMeta<
ArrayClientMetaDataProvider<std::atomic<size_t>>>;
}
#define SNMALLOC_PROVIDE_OWN_CONFIG
@@ -23,9 +23,8 @@ namespace snmalloc
int main()
{
#if defined(SNMALLOC_PASS_THROUGH) || \
defined(SNMALLOC_ENABLE_GWP_ASAN_INTEGRATION)
// This test does not make sense in pass-through
#if defined(SNMALLOC_ENABLE_GWP_ASAN_INTEGRATION)
// This test does not make sense in GWP-ASan mode.
return 0;
#else
// Allocate a bunch of objects, and store the index into the meta-data.
@@ -33,7 +32,7 @@ int main()
for (size_t i = 0; i < 10000; i++)
{
auto p = snmalloc::libc::malloc(1024);
auto& meta = snmalloc::libc::get_client_meta_data(p);
auto& meta = snmalloc::get_client_meta_data(p);
meta = i;
ptrs.push_back(p);
memset(p, (uint8_t)i, 1024);
@@ -44,7 +43,7 @@ int main()
for (size_t i = 0; i < 10000; i++)
{
auto p = ptrs[i];
auto& meta = snmalloc::libc::get_client_meta_data(p);
auto& meta = snmalloc::get_client_meta_data(p);
if (meta != i)
{
std::cout << "Failed at index " << i << std::endl;
@@ -63,7 +62,7 @@ int main()
// Access in a read-only way meta-data associated with the stack.
// This would fail if it was accessed for write.
auto& meta = snmalloc::libc::get_client_meta_data_const(&ptrs);
auto& meta = snmalloc::get_client_meta_data_const(&ptrs);
std::cout << "meta for stack" << meta << std::endl;
return 0;

41
src/test/func/domestication/domestication.cc
View File

@@ -1,22 +1,14 @@
#include <iostream>
#ifdef SNMALLOC_PASS_THROUGH
// This test does not make sense in pass-through
int main()
{
return 0;
}
#else
// # define SNMALLOC_TRACING
# include <snmalloc/backend/backend.h>
# include <snmalloc/backend/standard_range.h>
# include <snmalloc/backend_helpers/backend_helpers.h>
# include <snmalloc/snmalloc_core.h>
#include <snmalloc/backend/backend.h>
#include <snmalloc/backend/standard_range.h>
#include <snmalloc/backend_helpers/backend_helpers.h>
#include <snmalloc/snmalloc_core.h>
// Specify type of allocator
# define SNMALLOC_PROVIDE_OWN_CONFIG
#define SNMALLOC_PROVIDE_OWN_CONFIG
namespace snmalloc
{
@@ -98,9 +90,9 @@ namespace snmalloc
if (domesticate_trace)
{
std::cout << "Domesticating " << p.unsafe_ptr()
# if __has_builtin(__builtin_return_address)
#if __has_builtin(__builtin_return_address)
<< " from " << __builtin_return_address(0)
# endif
#endif
<< std::endl;
}
@@ -121,11 +113,11 @@ namespace snmalloc
}
};
using Alloc = LocalAllocator<CustomConfig>;
using Config = CustomConfig;
}
# define SNMALLOC_NAME_MANGLE(a) test_##a
# include <snmalloc/override/malloc.cc>
#define SNMALLOC_NAME_MANGLE(a) test_##a
#include <snmalloc/override/malloc.cc>
int main()
{
@@ -141,7 +133,7 @@ int main()
entropy.make_free_list_key(RemoteAllocator::key_global);
entropy.make_free_list_key(freelist::Object::key_root);
auto alloc1 = new Alloc();
ScopedAllocator alloc1;
// Allocate from alloc1; the size doesn't matter a whole lot, it just needs to
// be a small object and so definitely owned by this allocator rather.
@@ -149,7 +141,7 @@ int main()
std::cout << "Allocated p " << p << std::endl;
// Put that free object on alloc1's remote queue
auto alloc2 = new Alloc();
ScopedAllocator alloc2;
alloc2->dealloc(p);
alloc2->flush();
@@ -184,12 +176,5 @@ int main()
static constexpr size_t expected_count =
snmalloc::CustomConfig::Options.QueueHeadsAreTame ? 2 : 3;
SNMALLOC_CHECK(snmalloc::CustomConfig::domesticate_count == expected_count);
// Prevent the allocators from going out of scope during the above test
alloc1->flush();
alloc2->flush();
return 0;
}
#endif
}

3
src/test/func/external_pagemap/external_pagemap.cc
View File

@@ -1,5 +1,4 @@
#if defined(SNMALLOC_PASS_THROUGH) || defined(_WIN32) || \
!defined(TODO_REINSTATE_POSSIBLY)
#if defined(_WIN32) || !defined(TODO_REINSTATE_POSSIBLY)
// This test does not make sense with malloc pass-through, skip it.
// The malloc definitions are also currently incompatible with Windows headers
// so skip this test on Windows as well.

87
src/test/func/first_operation/first_operation.cc
View File

@@ -13,29 +13,14 @@
void alloc1(size_t size)
{
void* r = snmalloc::ThreadAlloc::get().alloc(size);
snmalloc::ThreadAlloc::get().dealloc(r);
void* r = snmalloc::alloc(size);
snmalloc::dealloc(r);
}
void alloc2(size_t size)
{
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc(size);
a.dealloc(r);
}
void alloc3(size_t size)
{
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc(size);
a.dealloc(r, size);
}
void alloc4(size_t size)
{
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc(size);
a.dealloc(r);
void* r = snmalloc::alloc(size);
snmalloc::dealloc(r, size);
}
void check_calloc(void* p, size_t size)
@@ -62,79 +47,43 @@ void check_calloc(void* p, size_t size)
void calloc1(size_t size)
{
void* r =
snmalloc::ThreadAlloc::get().alloc<snmalloc::ZeroMem::YesZero>(size);
void* r = snmalloc::alloc<snmalloc::ZeroMem::YesZero>(size);
check_calloc(r, size);
snmalloc::ThreadAlloc::get().dealloc(r);
snmalloc::dealloc(r);
}
void calloc2(size_t size)
{
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc<snmalloc::ZeroMem::YesZero>(size);
void* r = snmalloc::alloc<snmalloc::ZeroMem::YesZero>(size);
check_calloc(r, size);
a.dealloc(r);
}
void calloc3(size_t size)
{
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc<snmalloc::ZeroMem::YesZero>(size);
check_calloc(r, size);
a.dealloc(r, size);
}
void calloc4(size_t size)
{
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc<snmalloc::ZeroMem::YesZero>(size);
check_calloc(r, size);
a.dealloc(r);
snmalloc::dealloc(r, size);
}
void dealloc1(void* p, size_t)
{
snmalloc::ThreadAlloc::get().dealloc(p);
snmalloc::dealloc(p);
}
void dealloc2(void* p, size_t size)
{
snmalloc::ThreadAlloc::get().dealloc(p, size);
}
void dealloc3(void* p, size_t)
{
snmalloc::ThreadAlloc::get().dealloc(p);
}
void dealloc4(void* p, size_t size)
{
snmalloc::ThreadAlloc::get().dealloc(p, size);
snmalloc::dealloc(p, size);
}
void f(size_t size)
{
auto t1 = std::thread(alloc1, size);
auto t2 = std::thread(alloc2, size);
auto t3 = std::thread(alloc3, size);
auto t4 = std::thread(alloc4, size);
auto t5 = std::thread(calloc1, size);
auto t6 = std::thread(calloc2, size);
auto t7 = std::thread(calloc3, size);
auto t8 = std::thread(calloc4, size);
auto t3 = std::thread(calloc1, size);
auto t4 = std::thread(calloc2, size);
{
auto a = snmalloc::get_scoped_allocator();
auto p1 = a->alloc(size);
auto p2 = a->alloc(size);
auto p3 = a->alloc(size);
auto p4 = a->alloc(size);
auto t9 = std::thread(dealloc1, p1, size);
auto t10 = std::thread(dealloc2, p2, size);
auto t11 = std::thread(dealloc3, p3, size);
auto t12 = std::thread(dealloc4, p4, size);
auto t5 = std::thread(dealloc1, p1, size);
auto t6 = std::thread(dealloc2, p2, size);
t1.join();
t2.join();
@@ -142,14 +91,8 @@ void f(size_t size)
t4.join();
t5.join();
t6.join();
t7.join();
t8.join();
t9.join();
t10.join();
t11.join();
t12.join();
} // Drops a.
// snmalloc::current_alloc_pool()->debug_in_use(0);
snmalloc::debug_in_use(0);
printf(".");
fflush(stdout);
}

12
src/test/func/fixed_region/fixed_region.cc
View File

@@ -17,7 +17,6 @@ using FixedAlloc = LocalAllocator<CustomGlobals>;
int main()
{
#ifndef SNMALLOC_PASS_THROUGH // Depends on snmalloc specific features
setup();
// 28 is large enough to produce a nested allocator.
@@ -31,14 +30,14 @@ int main()
<< pointer_offset(oe_base, size) << std::endl;
CustomGlobals::init(nullptr, oe_base, size);
FixedAlloc a;
auto a = get_scoped_allocator<FixedAlloc>();
size_t object_size = 128;
size_t count = 0;
size_t i = 0;
while (true)
{
auto r1 = a.alloc(object_size);
auto r1 = a->alloc(object_size);
count += object_size;
i++;
@@ -48,9 +47,9 @@ int main()
if (r1 == nullptr)
break;
if (!a.is_snmalloc_owned(r1))
if (!snmalloc::is_owned<CustomGlobals>(r1))
{
a.dealloc(r1);
a->dealloc(r1);
continue;
}
@@ -75,7 +74,4 @@ int main()
std::cout << "Total allocated: " << count << " out of " << size << std::endl;
std::cout << "Overhead: 1/" << (double)size / (double)(size - count)
<< std::endl;
a.teardown();
#endif
}

3
src/test/func/jemalloc/jemalloc.cc
View File

@@ -318,9 +318,6 @@ extern "C"
int main()
{
#ifdef SNMALLOC_PASS_THROUGH
return 0;
#endif
check_lg_align_macro<63>();
static_assert(
OUR_MALLOCX_ZERO == MALLOCX_ZERO, "Our MALLOCX_ZERO macro is wrong");

16
src/test/func/malloc/malloc.cc
View File

@@ -33,21 +33,7 @@ void check_result(size_t size, size_t align, void* p, int err, bool null)
}
const auto alloc_size = our_malloc_usable_size(p);
auto expected_size = our_malloc_good_size(size);
#ifdef SNMALLOC_PASS_THROUGH
// Calling system allocator may allocate a larger block than
// snmalloc. Note, we have called the system allocator with
// the size snmalloc would allocate, so it won't be smaller.
const auto exact_size = false;
// We allocate MIN_ALLOC_SIZE byte for 0-sized allocations (and so round_size
// will tell us that the minimum size is MIN_ALLOC_SIZE), but the system
// allocator may return a 0-sized allocation.
if (size == 0)
{
expected_size = 0;
}
#else
const auto exact_size = align == 1;
#endif
#ifdef __CHERI_PURE_CAPABILITY__
const auto cheri_size = __builtin_cheri_length_get(p);
if (cheri_size != alloc_size && (size != 0))
@@ -376,6 +362,6 @@ int main(int argc, char** argv)
our_malloc_usable_size(nullptr) == 0,
"malloc_usable_size(nullptr) should be zero");
snmalloc::debug_check_empty<snmalloc::Alloc::Config>();
snmalloc::debug_check_empty();
return 0;
}

13
src/test/func/memcpy/func-memcpy.cc
View File

@@ -19,12 +19,7 @@ int main()
# endif
# define SNMALLOC_FAIL_FAST false
# define SNMALLOC_STATIC_LIBRARY_PREFIX my_
# ifndef SNMALLOC_PASS_THROUGH
# include "snmalloc/override/malloc.cc"
# else
# define my_malloc(x) malloc(x)
# define my_free(x) free(x)
# endif
# include "snmalloc/override/malloc.cc"
# include "snmalloc/override/memcpy.cc"
# include "test/helpers.h"
@@ -150,9 +145,6 @@ void check_bounds(size_t size, size_t out_of_bounds)
int main()
{
// Skip the checks that expect bounds checks to fail when we are not the
// malloc implementation.
# if !defined(SNMALLOC_PASS_THROUGH)
// Some sizes to check for out-of-bounds access. As we are only able to
// catch overflows past the end of the sizeclass-padded allocation, make
// sure we don't try to test on smaller allocations.
@@ -173,10 +165,9 @@ int main()
// Check one object out of bounds
check_bounds(sz, sz);
}
# endif
for (size_t x = 0; x < 2048; x++)
{
check_size(x);
}
}
#endif
#endif

174
src/test/func/memory/memory.cc
View File

@@ -69,10 +69,9 @@ void test_limited(rlim64_t as_limit, size_t& count)
upper_bound, static_cast<unsigned long long>(info.freeram >> 3u));
std::cout << "trying to alloc " << upper_bound / KiB << " KiB" << std::endl;
# endif
auto& alloc = ThreadAlloc::get();
std::cout << "allocator initialised" << std::endl;
auto chunk = alloc.alloc(upper_bound);
alloc.dealloc(chunk);
auto chunk = snmalloc::alloc(upper_bound);
snmalloc::dealloc(chunk);
std::cout << "success" << std::endl;
std::exit(0);
}
@@ -91,8 +90,6 @@ void test_limited(rlim64_t as_limit, size_t& count)
void test_alloc_dealloc_64k()
{
auto& alloc = ThreadAlloc::get();
constexpr size_t count = 1 << 12;
constexpr size_t outer_count = 12;
void* garbage[count];
@@ -104,26 +101,25 @@ void test_alloc_dealloc_64k()
// This will fill the short slab, and then start a new slab.
for (size_t i = 0; i < count; i++)
{
garbage[i] = alloc.alloc(16);
garbage[i] = snmalloc::alloc(16);
}
// Allocate one object on the second slab
keep_alive[j] = alloc.alloc(16);
keep_alive[j] = snmalloc::alloc(16);
for (size_t i = 0; i < count; i++)
{
alloc.dealloc(garbage[i]);
snmalloc::dealloc(garbage[i]);
}
}
for (size_t j = 0; j < outer_count; j++)
{
alloc.dealloc(keep_alive[j]);
snmalloc::dealloc(keep_alive[j]);
}
}
void test_random_allocation()
{
auto& alloc = ThreadAlloc::get();
std::unordered_set<void*> allocated;
constexpr size_t count = 10000;
@@ -146,13 +142,13 @@ void test_random_allocation()
if (cell != nullptr)
{
allocated.erase(cell);
alloc.dealloc(cell);
snmalloc::dealloc(cell);
cell = nullptr;
alloc_count--;
}
if (!just_dealloc)
{
cell = alloc.alloc(16);
cell = snmalloc::alloc(16);
auto pair = allocated.insert(cell);
// Check not already allocated
SNMALLOC_CHECK(pair.second);
@@ -170,20 +166,18 @@ void test_random_allocation()
// Deallocate all the remaining objects
for (size_t i = 0; i < count; i++)
if (objects[i] != nullptr)
alloc.dealloc(objects[i]);
snmalloc::dealloc(objects[i]);
}
void test_calloc()
{
auto& alloc = ThreadAlloc::get();
for (size_t size = 16; size <= (1 << 24); size <<= 1)
{
void* p = alloc.alloc(size);
void* p = snmalloc::alloc(size);
memset(p, 0xFF, size);
alloc.dealloc(p, size);
snmalloc::dealloc(p, size);
p = alloc.alloc<YesZero>(size);
p = snmalloc::alloc<YesZero>(size);
for (size_t i = 0; i < size; i++)
{
@@ -191,10 +185,10 @@ void test_calloc()
abort();
}
alloc.dealloc(p, size);
snmalloc::dealloc(p, size);
}
snmalloc::debug_check_empty<snmalloc::Alloc::Config>();
snmalloc::debug_check_empty();
}
void test_double_alloc()
@@ -227,48 +221,46 @@ void test_double_alloc()
while (!set1.empty())
{
auto it = set1.begin();
a2->dealloc(*it, 20);
a2->dealloc(*it);
set1.erase(it);
}
while (!set2.empty())
{
auto it = set2.begin();
a1->dealloc(*it, 20);
a1->dealloc(*it);
set2.erase(it);
}
}
}
snmalloc::debug_check_empty<snmalloc::Alloc::Config>();
snmalloc::debug_check_empty();
}
void test_external_pointer()
{
// Malloc does not have an external pointer querying mechanism.
auto& alloc = ThreadAlloc::get();
for (snmalloc::smallsizeclass_t sc = size_to_sizeclass(MIN_ALLOC_SIZE);
sc < NUM_SMALL_SIZECLASSES;
sc++)
{
size_t size = sizeclass_to_size(sc);
void* p1 = alloc.alloc(size);
void* p1 = snmalloc::alloc(size);
if (size != alloc.alloc_size(p1))
if (size != snmalloc::alloc_size(p1))
{
std::cout << "Requested size: " << size
<< " alloc_size: " << alloc.alloc_size(p1) << std::endl;
<< " alloc_size: " << snmalloc::alloc_size(p1) << std::endl;
abort();
}
for (size_t offset = 0; offset < size; offset += 17)
{
void* p2 = pointer_offset(p1, offset);
void* p3 = alloc.external_pointer(p2);
void* p4 = alloc.external_pointer<End>(p2);
void* p3 = snmalloc::external_pointer(p2);
void* p4 = snmalloc::external_pointer<End>(p2);
if (p1 != p3)
{
std::cout << "size: " << size << " alloc_size: " << alloc.alloc_size(p1)
std::cout << "size: " << size
<< " alloc_size: " << snmalloc::alloc_size(p1)
<< " offset: " << offset << " p1: " << p1 << " p3: " << p3
<< std::endl;
}
@@ -282,16 +274,15 @@ void test_external_pointer()
SNMALLOC_CHECK((size_t)p4 == (size_t)p1 + size - 1);
}
alloc.dealloc(p1, size);
snmalloc::dealloc(p1, size);
}
snmalloc::debug_check_empty<snmalloc::Alloc::Config>();
snmalloc::debug_check_empty();
};
void check_offset(void* base, void* interior)
{
auto& alloc = ThreadAlloc::get();
void* calced_base = alloc.external_pointer((void*)interior);
void* calced_base = snmalloc::external_pointer((void*)interior);
if (calced_base != (void*)base)
{
std::cout << "Calced base: " << calced_base << " actual base: " << base
@@ -315,8 +306,6 @@ void test_external_pointer_large()
{
xoroshiro::p128r64 r;
auto& alloc = ThreadAlloc::get();
constexpr size_t count_log = DefaultPal::address_bits > 32 ? 5 : 3;
constexpr size_t count = 1 << count_log;
// Pre allocate all the objects
@@ -331,9 +320,9 @@ void test_external_pointer_large()
size_t size = (1 << 24) + rand;
total_size += size;
// store object
objects[i] = (size_t*)alloc.alloc(size);
objects[i] = (size_t*)snmalloc::alloc(size);
// Store allocators size for this object
*objects[i] = alloc.alloc_size(objects[i]);
*objects[i] = snmalloc::alloc_size(objects[i]);
check_external_pointer_large(objects[i]);
if (i > 0)
@@ -351,33 +340,32 @@ void test_external_pointer_large()
// Deallocate everything
for (size_t i = 0; i < count; i++)
{
alloc.dealloc(objects[i]);
snmalloc::dealloc(objects[i]);
}
}
void test_external_pointer_dealloc_bug()
{
std::cout << "Testing external pointer dealloc bug" << std::endl;
auto& alloc = ThreadAlloc::get();
constexpr size_t count = MIN_CHUNK_SIZE;
void* allocs[count];
for (size_t i = 0; i < count; i++)
{
allocs[i] = alloc.alloc(MIN_CHUNK_BITS / 2);
allocs[i] = snmalloc::alloc(MIN_CHUNK_BITS / 2);
}
for (size_t i = 1; i < count; i++)
{
alloc.dealloc(allocs[i]);
snmalloc::dealloc(allocs[i]);
}
for (size_t i = 0; i < count; i++)
{
alloc.external_pointer(allocs[i]);
snmalloc::external_pointer(allocs[i]);
}
alloc.dealloc(allocs[0]);
snmalloc::dealloc(allocs[0]);
std::cout << "Testing external pointer dealloc bug - done" << std::endl;
}
@@ -387,15 +375,12 @@ void test_external_pointer_stack()
std::array<int, 2000> stack;
auto& alloc = ThreadAlloc::get();
for (size_t i = 0; i < stack.size(); i++)
{
if (alloc.external_pointer(&stack[i]) > &stack[i])
if (snmalloc::external_pointer(&stack[i]) > &stack[i])
{
std::cout << "Stack pointer: " << &stack[i]
<< " external pointer: " << alloc.external_pointer(&stack[i])
<< std::endl;
std::cout << "Stack pointer: " << &stack[i] << " external pointer: "
<< snmalloc::external_pointer(&stack[i]) << std::endl;
abort();
}
}
@@ -405,92 +390,97 @@ void test_external_pointer_stack()
void test_alloc_16M()
{
auto& alloc = ThreadAlloc::get();
// sizes >= 16M use large_alloc
const size_t size = 16'000'000;
void* p1 = alloc.alloc(size);
SNMALLOC_CHECK(alloc.alloc_size(alloc.external_pointer(p1)) >= size);
alloc.dealloc(p1);
void* p1 = snmalloc::alloc(size);
SNMALLOC_CHECK(snmalloc::alloc_size(snmalloc::external_pointer(p1)) >= size);
snmalloc::dealloc(p1);
}
void test_calloc_16M()
{
auto& alloc = ThreadAlloc::get();
// sizes >= 16M use large_alloc
const size_t size = 16'000'000;
void* p1 = alloc.alloc<YesZero>(size);
SNMALLOC_CHECK(alloc.alloc_size(alloc.external_pointer(p1)) >= size);
alloc.dealloc(p1);
void* p1 = snmalloc::alloc<YesZero>(size);
SNMALLOC_CHECK(snmalloc::alloc_size(snmalloc::external_pointer(p1)) >= size);
snmalloc::dealloc(p1);
}
void test_calloc_large_bug()
{
auto& alloc = ThreadAlloc::get();
// Perform large calloc, to check for correct zeroing from PAL.
// Some PALS have special paths for PAGE aligned zeroing of large
// allocations. This is a large allocation that is intentionally
// not a multiple of page size.
const size_t size = (MAX_SMALL_SIZECLASS_SIZE << 3) - 7;
void* p1 = alloc.alloc<YesZero>(size);
SNMALLOC_CHECK(alloc.alloc_size(alloc.external_pointer(p1)) >= size);
alloc.dealloc(p1);
void* p1 = snmalloc::alloc<YesZero>(size);
SNMALLOC_CHECK(snmalloc::alloc_size(snmalloc::external_pointer(p1)) >= size);
snmalloc::dealloc(p1);
}
template<size_t asz, int dealloc>
template<size_t asz, int dealloc = 2>
void test_static_sized_alloc()
{
auto& alloc = ThreadAlloc::get();
auto p = alloc.alloc<asz>();
auto p = snmalloc::alloc<asz>();
static_assert((dealloc >= 0) && (dealloc <= 2), "bad dealloc flavor");
switch (dealloc)
{
case 0:
alloc.dealloc(p);
snmalloc::dealloc(p);
break;
case 1:
alloc.dealloc(p, asz);
snmalloc::dealloc(p, asz);
break;
case 2:
alloc.dealloc<asz>(p);
snmalloc::dealloc<asz>(p);
break;
}
if constexpr (dealloc != 0)
test_static_sized_alloc<asz, dealloc - 1>();
}
template<size_t max_size = bits::one_at_bit(23)>
void test_static_sized_allocs()
{
// For each small, medium, and large class, do each kind dealloc. This is
// mostly to ensure that all of these forms compile.
for (size_t sc = 0; sc < NUM_SMALL_SIZECLASSES; sc++)
{
// test_static_sized_alloc<sc, 0>();
// test_static_sized_alloc<sc, 1>();
// test_static_sized_alloc<sc, 2>();
}
// test_static_sized_alloc<sizeclass_to_size(NUM_SMALL_CLASSES + 1), 0>();
// test_static_sized_alloc<sizeclass_to_size(NUM_SMALL_CLASSES + 1), 1>();
// test_static_sized_alloc<sizeclass_to_size(NUM_SMALL_CLASSES + 1), 2>();
if (max_size < 16)
return;
// test_static_sized_alloc<large_sizeclass_to_size(0), 0>();
// test_static_sized_alloc<large_sizeclass_to_size(0), 1>();
// test_static_sized_alloc<large_sizeclass_to_size(0), 2>();
constexpr size_t next_size = max_size >> 1;
test_static_sized_allocs<next_size>();
test_static_sized_alloc<max_size * 3>();
test_static_sized_alloc<max_size * 5>();
test_static_sized_alloc<max_size * 7>();
test_static_sized_alloc<max_size * 1>();
test_static_sized_alloc<max_size * 3 - 1>();
test_static_sized_alloc<max_size * 5 - 1>();
test_static_sized_alloc<max_size * 7 - 1>();
test_static_sized_alloc<max_size * 1 - 1>();
test_static_sized_alloc<max_size * 3 + 1>();
test_static_sized_alloc<max_size * 5 + 1>();
test_static_sized_alloc<max_size * 7 + 1>();
test_static_sized_alloc<max_size * 1 + 1>();
}
void test_remaining_bytes()
{
auto& alloc = ThreadAlloc::get();
for (snmalloc::smallsizeclass_t sc = size_to_sizeclass(MIN_ALLOC_SIZE);
sc < NUM_SMALL_SIZECLASSES;
sc++)
{
auto size = sizeclass_to_size(sc);
char* p = (char*)alloc.alloc(size);
char* p = (char*)snmalloc::alloc(size);
for (size_t offset = 0; offset < size; offset++)
{
auto rem = alloc.remaining_bytes(address_cast(pointer_offset(p, offset)));
auto rem =
snmalloc::remaining_bytes(address_cast(pointer_offset(p, offset)));
if (rem != (size - offset))
{
printf(
@@ -503,7 +493,7 @@ void test_remaining_bytes()
abort();
}
}
alloc.dealloc(p);
snmalloc::dealloc(p);
}
}
@@ -513,18 +503,16 @@ void test_consolidaton_bug()
* Check for consolidation of various sizes, but allocating and deallocating,
* then requesting larger sizes. See issue #506
*/
auto& alloc = ThreadAlloc::get();
for (size_t i = 0; i < 27; i++)
{
std::vector<void*> allocs;
for (size_t j = 0; j < 4; j++)
{
allocs.push_back(alloc.alloc(bits::one_at_bit(i)));
allocs.push_back(snmalloc::alloc(bits::one_at_bit(i)));
}
for (auto a : allocs)
{
alloc.dealloc(a);
snmalloc::dealloc(a);
}
}
}
@@ -557,7 +545,6 @@ int main(int argc, char** argv)
test_random_allocation();
test_calloc();
test_double_alloc();
#ifndef SNMALLOC_PASS_THROUGH // Depends on snmalloc specific features
test_remaining_bytes();
test_static_sized_allocs();
test_calloc_large_bug();
@@ -567,7 +554,6 @@ int main(int argc, char** argv)
test_external_pointer();
test_alloc_16M();
test_calloc_16M();
#endif
test_consolidaton_bug();
return 0;
}

2
src/test/func/memory_usage/memory_usage.cc
View File

@@ -79,7 +79,6 @@ int main(int argc, char** argv)
{
UNUSED(argc);
UNUSED(argv);
#ifndef SNMALLOC_PASS_THROUGH // Depends on snmalloc specific features
setup();
add_n_allocs(5);
@@ -103,5 +102,4 @@ int main(int argc, char** argv)
remove_n_allocs(3);
std::cout << "Teardown complete!" << std::endl;
#endif
}

17
src/test/func/miracle_ptr/miracle_ptr.cc
View File

@@ -25,8 +25,8 @@ namespace snmalloc
{
// Instantiate the allocator with a client meta data provider that uses an
// atomic size_t to store the reference count.
using Alloc = snmalloc::LocalAllocator<snmalloc::StandardConfigClientMeta<
ArrayClientMetaDataProvider<std::atomic<size_t>>>>;
using Config = snmalloc::StandardConfigClientMeta<
ArrayClientMetaDataProvider<std::atomic<size_t>>>;
}
# define SNMALLOC_PROVIDE_OWN_CONFIG
@@ -58,7 +58,7 @@ namespace snmalloc::miracle
if (SNMALLOC_UNLIKELY(p == nullptr))
return nullptr;
snmalloc::libc::get_client_meta_data(p) = 1;
snmalloc::get_client_meta_data(p) = 1;
return p;
}
@@ -68,8 +68,7 @@ namespace snmalloc::miracle
return;
// TODO could build a check into this that it is the start of the object?
auto previous =
snmalloc::libc::get_client_meta_data(ptr).fetch_add((size_t)-1);
auto previous = snmalloc::get_client_meta_data(ptr).fetch_add((size_t)-1);
if (SNMALLOC_LIKELY(previous == 1))
{
@@ -88,8 +87,7 @@ namespace snmalloc::miracle
inline void acquire(void* p)
{
auto previous =
snmalloc::libc::get_client_meta_data(p).fetch_add((size_t)2);
auto previous = snmalloc::get_client_meta_data(p).fetch_add((size_t)2);
// Can we take new pointers to a deallocated object?
check((previous & 1) == 1, "Acquiring a deallocated object");
@@ -97,8 +95,7 @@ namespace snmalloc::miracle
inline void release(void* p)
{
auto previous =
snmalloc::libc::get_client_meta_data(p).fetch_add((size_t)-2);
auto previous = snmalloc::get_client_meta_data(p).fetch_add((size_t)-2);
if (previous > 2)
return;
@@ -185,7 +182,6 @@ void operator delete(void* p, size_t)
int main()
{
# ifndef SNMALLOC_PASS_THROUGH
snmalloc::miracle::raw_ptr<int> p;
{
auto up1 = std::make_unique<int>(41);
@@ -199,7 +195,6 @@ int main()
// raw_ptr has kept the memory live.
// Current implementation zeros the memory when the unique_ptr is destroyed.
check(*p == 0, "Failed to keep memory live");
# endif
return 0;
}
#endif

6
src/test/func/sandbox/sandbox.cc
View File

@@ -1,4 +1,4 @@
#if defined(SNMALLOC_PASS_THROUGH) || true
#if true
/*
* This test does not make sense with malloc pass-through, skip it.
*/
@@ -244,7 +244,7 @@ namespace
// Use the outside-sandbox snmalloc to allocate memory, rather than using
// the PAL directly, so that our out-of-sandbox can amplify sandbox
// pointers
return ThreadAlloc::get().alloc(sb_size);
return snmalloc::alloc(sb_size);
}
};
}
@@ -260,7 +260,7 @@ int main()
auto check = [](Sandbox& sb, auto& alloc, size_t sz) {
void* ptr = alloc.alloc(sz);
SNMALLOC_CHECK(sb.is_in_sandbox_heap(ptr, sz));
ThreadAlloc::get().dealloc(ptr);
snmalloc::dealloc(ptr);
};
auto check_with_sb = [&](Sandbox& sb) {
// Check with a range of sizes

41
src/test/func/statistics/stats.cc
View File

@@ -1,23 +1,16 @@
#ifdef SNMALLOC_PASS_THROUGH // This test depends on snmalloc internals
int main()
{
return 0;
}
#else
# include <iostream>
# include <snmalloc/snmalloc.h>
# include <vector>
#include <iostream>
#include <snmalloc/snmalloc.h>
#include <vector>
template<size_t size>
void debug_check_empty_1()
{
std::cout << "debug_check_empty_1 " << size << std::endl;
snmalloc::Alloc& a = snmalloc::ThreadAlloc::get();
bool result;
auto r = a.alloc(size);
auto r = snmalloc::alloc(size);
snmalloc::debug_check_empty<snmalloc::Alloc::Config>(&result);
snmalloc::debug_check_empty(&result);
if (result != false)
{
std::cout << "debug_check_empty failed to detect leaked memory:" << size
@@ -25,18 +18,18 @@ void debug_check_empty_1()
abort();
}
a.dealloc(r);
snmalloc::dealloc(r);
snmalloc::debug_check_empty<snmalloc::Alloc::Config>(&result);
snmalloc::debug_check_empty(&result);
if (result != true)
{
std::cout << "debug_check_empty failed to say empty:" << size << std::endl;
abort();
}
r = a.alloc(size);
r = snmalloc::alloc(size);
snmalloc::debug_check_empty<snmalloc::Alloc::Config>(&result);
snmalloc::debug_check_empty(&result);
if (result != false)
{
std::cout << "debug_check_empty failed to detect leaked memory:" << size
@@ -44,9 +37,9 @@ void debug_check_empty_1()
abort();
}
a.dealloc(r);
snmalloc::dealloc(r);
snmalloc::debug_check_empty<snmalloc::Alloc::Config>(&result);
snmalloc::debug_check_empty(&result);
if (result != true)
{
std::cout << "debug_check_empty failed to say empty:" << size << std::endl;
@@ -58,7 +51,6 @@ template<size_t size>
void debug_check_empty_2()
{
std::cout << "debug_check_empty_2 " << size << std::endl;
snmalloc::Alloc& a = snmalloc::ThreadAlloc::get();
bool result;
std::vector<void*> allocs;
// 1GB of allocations
@@ -70,9 +62,9 @@ void debug_check_empty_2()
{
std::cout << "." << std::flush;
}
auto r = a.alloc(size);
auto r = snmalloc::alloc(size);
allocs.push_back(r);
snmalloc::debug_check_empty<snmalloc::Alloc::Config>(&result);
snmalloc::debug_check_empty(&result);
if (result != false)
{
std::cout << "False empty after " << i << " allocations of " << size
@@ -88,17 +80,17 @@ void debug_check_empty_2()
{
std::cout << "." << std::flush;
}
snmalloc::debug_check_empty<snmalloc::Alloc::Config>(&result);
snmalloc::debug_check_empty(&result);
if (result != false)
{
std::cout << "False empty after " << i << " deallocations of " << size
<< std::endl;
abort();
}
a.dealloc(allocs[i]);
snmalloc::dealloc(allocs[i]);
}
std::cout << std::endl;
snmalloc::debug_check_empty<snmalloc::Alloc::Config>();
snmalloc::debug_check_empty();
}
int main()
@@ -115,4 +107,3 @@ int main()
return 0;
}
#endif

100
src/test/func/teardown/teardown.cc
View File

@@ -13,43 +13,25 @@
void trigger_teardown()
{
auto& a = snmalloc::ThreadAlloc::get();
// Trigger init
void* r = a.alloc(16);
a.dealloc(r);
void* r = snmalloc::alloc(16);
snmalloc::dealloc(r);
// Force teardown
a.teardown();
snmalloc::debug_teardown();
}
void alloc1(size_t size)
{
trigger_teardown();
void* r = snmalloc::ThreadAlloc::get().alloc(size);
snmalloc::ThreadAlloc::get().dealloc(r);
void* r = snmalloc::alloc(size);
snmalloc::dealloc(r);
}
void alloc2(size_t size)
{
trigger_teardown();
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc(size);
a.dealloc(r);
}
void alloc3(size_t size)
{
trigger_teardown();
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc(size);
a.dealloc(r, size);
}
void alloc4(size_t size)
{
trigger_teardown();
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc(size);
a.dealloc(r);
void* r = snmalloc::alloc(size);
snmalloc::dealloc(r, size);
}
void check_calloc(void* p, size_t size)
@@ -77,86 +59,46 @@ void check_calloc(void* p, size_t size)
void calloc1(size_t size)
{
trigger_teardown();
void* r =
snmalloc::ThreadAlloc::get().alloc<snmalloc::ZeroMem::YesZero>(size);
void* r = snmalloc::alloc<snmalloc::ZeroMem::YesZero>(size);
check_calloc(r, size);
snmalloc::ThreadAlloc::get().dealloc(r);
snmalloc::dealloc(r);
}
void calloc2(size_t size)
{
trigger_teardown();
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc<snmalloc::ZeroMem::YesZero>(size);
void* r = snmalloc::alloc<snmalloc::ZeroMem::YesZero>(size);
check_calloc(r, size);
a.dealloc(r);
}
void calloc3(size_t size)
{
trigger_teardown();
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc<snmalloc::ZeroMem::YesZero>(size);
check_calloc(r, size);
a.dealloc(r, size);
}
void calloc4(size_t size)
{
trigger_teardown();
auto& a = snmalloc::ThreadAlloc::get();
void* r = a.alloc<snmalloc::ZeroMem::YesZero>(size);
check_calloc(r, size);
a.dealloc(r);
snmalloc::dealloc(r, size);
}
void dealloc1(void* p, size_t)
{
trigger_teardown();
snmalloc::ThreadAlloc::get().dealloc(p);
snmalloc::dealloc(p);
}
void dealloc2(void* p, size_t size)
{
trigger_teardown();
snmalloc::ThreadAlloc::get().dealloc(p, size);
}
void dealloc3(void* p, size_t)
{
trigger_teardown();
snmalloc::ThreadAlloc::get().dealloc(p);
}
void dealloc4(void* p, size_t size)
{
trigger_teardown();
snmalloc::ThreadAlloc::get().dealloc(p, size);
snmalloc::dealloc(p, size);
}
void f(size_t size)
{
auto t1 = std::thread(alloc1, size);
auto t2 = std::thread(alloc2, size);
auto t3 = std::thread(alloc3, size);
auto t4 = std::thread(alloc4, size);
auto t5 = std::thread(calloc1, size);
auto t6 = std::thread(calloc2, size);
auto t7 = std::thread(calloc3, size);
auto t8 = std::thread(calloc4, size);
auto t3 = std::thread(calloc1, size);
auto t4 = std::thread(calloc2, size);
{
auto a = snmalloc::get_scoped_allocator();
auto p1 = a->alloc(size);
auto p2 = a->alloc(size);
auto p3 = a->alloc(size);
auto p4 = a->alloc(size);
auto t9 = std::thread(dealloc1, p1, size);
auto t10 = std::thread(dealloc2, p2, size);
auto t11 = std::thread(dealloc3, p3, size);
auto t12 = std::thread(dealloc4, p4, size);
auto t5 = std::thread(dealloc1, p1, size);
auto t6 = std::thread(dealloc2, p2, size);
t1.join();
t2.join();
@@ -164,14 +106,8 @@ void f(size_t size)
t4.join();
t5.join();
t6.join();
t7.join();
t8.join();
t9.join();
t10.join();
t11.join();
t12.join();
} // Drops a.
// snmalloc::current_alloc_pool()->debug_in_use(0);
snmalloc::debug_in_use(0);
printf(".");
fflush(stdout);
}

12
src/test/func/thread_alloc_external/thread_alloc_external.cc
View File

@@ -13,8 +13,8 @@
namespace snmalloc
{
using Alloc = snmalloc::LocalAllocator<
snmalloc::StandardConfigClientMeta<NoClientMetaDataProvider>>;
using Config = snmalloc::StandardConfigClientMeta<NoClientMetaDataProvider>;
using Alloc = snmalloc::LocalAllocator<Config>;
}
using namespace snmalloc;
@@ -65,16 +65,14 @@ int main()
setup();
allocator_thread_init();
auto& a = ThreadAlloc::get();
for (size_t i = 0; i < 1000; i++)
{
auto r1 = a.alloc(i);
auto r1 = snmalloc::alloc(i);
a.dealloc(r1);
snmalloc::dealloc(r1);
}
ThreadAlloc::get().teardown();
snmalloc::debug_teardown();
// This checks that the scoped allocator does not call
// register clean up, as this configuration will fault

6
src/test/func/two_alloc_types/alloc1.cc
View File

@@ -13,8 +13,7 @@
namespace snmalloc
{
using CustomGlobals = FixedRangeConfig<PALNoAlloc<DefaultPal>>;
using Alloc = LocalAllocator<CustomGlobals>;
using Config = FixedRangeConfig<PALNoAlloc<DefaultPal>>;
}
#define SNMALLOC_NAME_MANGLE(a) enclave_##a
@@ -22,6 +21,5 @@ namespace snmalloc
extern "C" void oe_allocator_init(void* base, void* end)
{
snmalloc::CustomGlobals::init(
nullptr, base, address_cast(end) - address_cast(base));
snmalloc::Config::init(nullptr, base, address_cast(end) - address_cast(base));
}

13
src/test/perf/contention/contention.cc
View File

@@ -75,13 +75,12 @@ size_t swapcount;
void test_tasks_f(size_t id)
{
auto& a = ThreadAlloc::get();
xoroshiro::p128r32 r(id + 5000);
for (size_t n = 0; n < swapcount; n++)
{
size_t size = 16 + (r.next() % 1024);
size_t* res = (size_t*)(use_malloc ? malloc(size) : a.alloc(size));
size_t* res = (size_t*)(use_malloc ? malloc(size) : snmalloc::alloc(size));
if (res != nullptr)
{
@@ -102,7 +101,7 @@ void test_tasks_f(size_t id)
if (use_malloc)
free(out);
else
a.dealloc(out, size);
snmalloc::dealloc(out, size);
}
}
};
@@ -111,8 +110,6 @@ void test_tasks(size_t num_tasks, size_t count, size_t size)
{
std::cout << "Sequential setup" << std::endl;
auto& a = ThreadAlloc::get();
contention = new std::atomic<size_t*>[size];
xoroshiro::p128r32 r;
@@ -120,7 +117,7 @@ void test_tasks(size_t num_tasks, size_t count, size_t size)
{
size_t alloc_size = 16 + (r.next() % 1024);
size_t* res =
(size_t*)(use_malloc ? malloc(alloc_size) : a.alloc(alloc_size));
(size_t*)(use_malloc ? malloc(alloc_size) : snmalloc::alloc(alloc_size));
*res = alloc_size;
contention[n] = res;
}
@@ -146,7 +143,7 @@ void test_tasks(size_t num_tasks, size_t count, size_t size)
if (use_malloc)
free(contention[n]);
else
a.dealloc(contention[n], *contention[n]);
snmalloc::dealloc(contention[n], *contention[n]);
}
}
@@ -154,7 +151,7 @@ void test_tasks(size_t num_tasks, size_t count, size_t size)
}
#ifndef NDEBUG
snmalloc::debug_check_empty<snmalloc::Alloc::Config>();
snmalloc::debug_check_empty();
#endif
};

23
src/test/perf/external_pointer/externalpointer.cc
View File

@@ -13,7 +13,7 @@ namespace test
// Pre allocate all the objects
size_t* objects[count];
NOINLINE void setup(xoroshiro::p128r64& r, Alloc& alloc)
NOINLINE void setup(xoroshiro::p128r64& r)
{
for (size_t i = 0; i < count; i++)
{
@@ -31,28 +31,27 @@ namespace test
if (size < 16)
size = 16;
// store object
objects[i] = (size_t*)alloc.alloc(size);
objects[i] = (size_t*)snmalloc::alloc(size);
if (objects[i] == nullptr)
abort();
// Store allocators size for this object
*objects[i] = alloc.alloc_size(objects[i]);
*objects[i] = snmalloc::alloc_size(objects[i]);
}
}
NOINLINE void teardown(Alloc& alloc)
NOINLINE void teardown()
{
// Deallocate everything
for (size_t i = 0; i < count; i++)
{
alloc.dealloc(objects[i]);
snmalloc::dealloc(objects[i]);
}
snmalloc::debug_check_empty<snmalloc::Alloc::Config>();
snmalloc::debug_check_empty();
}
void test_external_pointer(xoroshiro::p128r64& r)
{
auto& alloc = ThreadAlloc::get();
// This is very slow on Windows at the moment. Until this is fixed, help
// CI terminate.
#if defined(NDEBUG) && !defined(_MSC_VER)
@@ -66,7 +65,7 @@ namespace test
static constexpr size_t iterations = 100000;
# endif
#endif
setup(r, alloc);
setup(r);
{
MeasureTime m;
@@ -76,12 +75,12 @@ namespace test
size_t rand = (size_t)r.next();
size_t oid = rand & (((size_t)1 << count_log) - 1);
size_t* external_ptr = objects[oid];
if (!alloc.is_snmalloc_owned(external_ptr))
if (!snmalloc::is_owned(external_ptr))
continue;
size_t size = *external_ptr;
size_t offset = (size >> 4) * (rand & 15);
void* interior_ptr = pointer_offset(external_ptr, offset);
void* calced_external = alloc.external_pointer(interior_ptr);
void* calced_external = snmalloc::external_pointer(interior_ptr);
if (calced_external != external_ptr)
{
abort();
@@ -89,13 +88,12 @@ namespace test
}
}
teardown(alloc);
teardown();
}
}
int main(int, char**)
{
#ifndef SNMALLOC_PASS_THROUGH // Depends on snmalloc specific features
setup();
xoroshiro::p128r64 r;
@@ -105,5 +103,4 @@ int main(int, char**)
for (size_t n = 0; n < nn; n++)
test::test_external_pointer(r);
return 0;
#endif
}

4
src/test/perf/low_memory/low-memory.cc
View File

@@ -20,7 +20,7 @@ class Queue
Node* new_node(size_t size)
{
auto result = (Node*)ThreadAlloc::get().alloc(size);
auto result = (Node*)snmalloc::alloc(size);
result->next = nullptr;
return result;
}
@@ -44,7 +44,7 @@ public:
return false;
Node* next = head->next;
ThreadAlloc::get().dealloc(head);
snmalloc::dealloc(head);
head = next;
return true;
}

12
src/test/perf/memcpy/memcpy.cc
View File

@@ -35,7 +35,7 @@ void shape(size_t size)
// the memcpys. constexpr size_t alignment = 16; offset = (my_random() %
// size / alignment) * alignment;
Shape s;
s.object = ThreadAlloc::get().alloc(rsize);
s.object = snmalloc::alloc(rsize);
s.dst = static_cast<unsigned char*>(s.object) + offset;
// Bring into cache the destination of the copy.
memset(s.dst, 0xFF, size);
@@ -47,7 +47,7 @@ void unshape()
{
for (auto& s : allocs)
{
ThreadAlloc::get().dealloc(s.object);
snmalloc::dealloc(s.object);
}
allocs.clear();
}
@@ -68,7 +68,7 @@ void test(
Memcpy mc,
std::vector<std::pair<size_t, std::chrono::nanoseconds>>& stats)
{
auto src = ThreadAlloc::get().alloc(size);
auto src = snmalloc::alloc(size);
shape(size);
for (size_t i = 0; i < 10; i++)
{
@@ -77,7 +77,7 @@ void test(
auto time = m.get_time();
stats.push_back({size, time});
}
ThreadAlloc::get().dealloc(src);
snmalloc::dealloc(src);
unshape();
}
@@ -108,7 +108,6 @@ void memcpy_platform_checked(void* dst, const void* src, size_t size)
int main(int argc, char** argv)
{
opt::Opt opt(argc, argv);
#ifndef SNMALLOC_PASS_THROUGH
bool full_test = opt.has("--full_test");
// size_t size = 0;
@@ -182,8 +181,5 @@ int main(int argc, char** argv)
stats_platform.clear();
stats_platform_checked.clear();
}
#else
snmalloc::UNUSED(opt);
#endif
return 0;
}

54
src/test/perf/msgpass/msgpass.cc
View File

@@ -36,46 +36,6 @@ void chatty(const char* p, ...)
}
}
/*
* Interpret SNMALLOC_PASS_THROUGH ourselves to make this a bit more fair of a
* comparison, since relying of snmalloc itself to do the passing through
* results in it imposing its own idea of alignment onto the underlying
* allocator, which might result in it taking less optimized paths.
*/
#ifdef SNMALLOC_PASS_THROUGH
struct MyAlloc
{
MyAlloc() {}
void* alloc(size_t sz)
{
return malloc(sz);
}
void dealloc(void* p)
{
free(p);
}
};
#else
struct MyAlloc
{
snmalloc::Alloc& a;
MyAlloc() : a(ThreadAlloc::get()) {}
void* alloc(size_t sz)
{
return a.alloc(sz);
}
void dealloc(void* p)
{
a.dealloc(p);
}
};
#endif
/*
* FreeListMPSCQ make for convenient MPSC queues, so we use those for sending
* "messages". Each consumer or proxy has its own (source) queue.
@@ -106,7 +66,6 @@ freelist::HeadPtr domesticate_nop(freelist::QueuePtr p)
void consumer(const struct params* param, size_t qix)
{
MyAlloc a{};
auto& myq = param->msgqueue[qix];
chatty("Cl %zu q is %p\n", qix, &myq);
@@ -118,13 +77,11 @@ void consumer(const struct params* param, size_t qix)
if (myq.can_dequeue(domesticate_nop, domesticate_nop))
{
myq.dequeue(
domesticate_nop,
domesticate_nop,
[qix, &a, &reap](freelist::HeadPtr o) {
domesticate_nop, domesticate_nop, [qix, &reap](freelist::HeadPtr o) {
UNUSED(qix);
auto p = o.as_void().unsafe_ptr();
chatty("Cl %zu free %p\n", qix, p);
a.dealloc(p);
snmalloc::dealloc(p);
reap++;
return true;
});
@@ -145,7 +102,7 @@ void consumer(const struct params* param, size_t qix)
producers_live || (queue_gate > param->N_CONSUMER));
chatty("Cl %zu fini\n", qix);
a.dealloc(myq.destroy().unsafe_ptr());
snmalloc::dealloc(myq.destroy().unsafe_ptr());
}
void proxy(const struct params* param, size_t qix)
@@ -178,13 +135,12 @@ void proxy(const struct params* param, size_t qix)
chatty("Px %zu fini\n", qix);
MyAlloc().dealloc(myq.destroy().unsafe_ptr());
snmalloc::dealloc(myq.destroy().unsafe_ptr());
queue_gate--;
}
void producer(const struct params* param, size_t pix)
{
MyAlloc a{};
static constexpr size_t msgsizes[] = {48, 64, 96, 128};
static constexpr size_t nmsgsizes = sizeof(msgsizes) / sizeof(msgsizes[0]);
@@ -206,7 +162,7 @@ void producer(const struct params* param, size_t pix)
/* Allocate batch and form list */
for (size_t msgix = 0; msgix < nmsg; msgix++)
{
auto msg = a.alloc(msgsize);
auto msg = snmalloc::alloc(msgsize);
chatty("Pd %zu make %p\n", pix, msg);
auto msgc = capptr::Alloc<void>::unsafe_from(msg)

12
src/test/perf/singlethread/singlethread.cc
View File

@@ -8,8 +8,6 @@ using namespace snmalloc;
template<ZeroMem zero_mem>
void test_alloc_dealloc(size_t count, size_t size, bool write)
{
auto& alloc = ThreadAlloc::get();
{
MeasureTime m;
m << "Count: " << std::setw(6) << count << ", Size: " << std::setw(6)
@@ -20,7 +18,7 @@ void test_alloc_dealloc(size_t count, size_t size, bool write)
// alloc 1.5x objects
for (size_t i = 0; i < ((count * 3) / 2); i++)
{
void* p = alloc.alloc<zero_mem>(size);
void* p = snmalloc::alloc<zero_mem>(size);
SNMALLOC_CHECK(set.find(p) == set.end());
if (write)
@@ -36,13 +34,13 @@ void test_alloc_dealloc(size_t count, size_t size, bool write)
void* p = *it;
set.erase(it);
SNMALLOC_CHECK(set.find(p) == set.end());
alloc.dealloc(p, size);
snmalloc::dealloc(p, size);
}
// alloc 1x objects
for (size_t i = 0; i < count; i++)
{
void* p = alloc.alloc<zero_mem>(size);
void* p = snmalloc::alloc<zero_mem>(size);
SNMALLOC_CHECK(set.find(p) == set.end());
if (write)
@@ -55,12 +53,12 @@ void test_alloc_dealloc(size_t count, size_t size, bool write)
while (!set.empty())
{
auto it = set.begin();
alloc.dealloc(*it, size);
snmalloc::dealloc(*it, size);
set.erase(it);
}
}
snmalloc::debug_check_empty<snmalloc::Alloc::Config>();
snmalloc::debug_check_empty();
}
int main(int, char**)

3
src/test/perf/startup/startup.cc
View File

@@ -77,8 +77,7 @@ int main()
ParallelTest test(
[](size_t id) {
auto start = Aal::tick();
auto& alloc = snmalloc::ThreadAlloc::get();
alloc.dealloc(alloc.alloc(1));
snmalloc::dealloc(snmalloc::alloc(1));
auto end = Aal::tick();
counters[id] = end - start;
},