Resurrecting this in case someone bumps into this question.
When to use ArraySegment over Memory?
Never, unless you need to call something old that expects an ArraySegment<T>, which I doubt will be the case as it was never that popular.
ArraySegment<T> is just an array, an offset, and a length, which are all exposed directly where you can choose to ignore the offset and length and access the entirety of the array if you want to. There’s also no read-only version of ArraySegment<T>.
Span<T> and Memory<T> can be backed by arrays, similar to ArraySegment<T>, but also by strings and unmanaged memory (in the form of a pointer in Span<T>’s case, and by using a custom MemoryManager<T> in Memory<T>’s case). They provide better encapsulation by not exposing their underlying data source and have read-only versions for immutable access.
Back then, we had to pass the array/offset/count trio to a lot of APIs (APIs that needed a direct reference of an array), but now that Span<T> and Memory<T> exist and are widely supported by most, if not all, .NET APIs that need to interact with continuous blocks of memory, you should have no reason to use an ArraySegment<T>.
See also: Memory- and span-related types - MS Docs