14

I ran into strange behavior when using the Aztec linear system solver library. Using valgrind, I found out that this library does a memcpy on overlapping buffers. Specification says that behavior of memcpy on overlapping buffers is not defined.

It turns out that memcpy on many machines has the same behavior as if you would do it with a for loop and therefore you can safely copy from a higher source to a lower destination:

for(int i = 0; i < len; i ++)
  dest[i] = source[i];

BUT on our large cluster, memcpy of overlapping buffers has a different behavior which leads to problems.

Now I wonder whether the overlapping memcpy in the library is normal or just caused by another bug in my code. Since the library is widely used I assume that the memcpy issue should have been discovered earlier. On the other hand, it's still possible that the vast majority of the memcpy implementations behave like the for loop and therefore nobody ever encountered this problem.

  • Can anyone tell me about his experiences with overlapping memcpy on various machines?
  • Which part of my computer system does actually provide memcpy?

I'd like to point out that question is about the practical experience with various implementations, not about what the specification says.

Michael
  • 7,407
  • 8
  • 41
  • 84
  • Please provide a minimal example, so that we can reproduce your issue and let you know if the bug is in the library or in your code. – OmnipotentEntity Sep 02 '14 at 18:24
  • I've never understood this issue with memcpy. Surely, it's only one compare to find out if you need to copy from the top or bottom of the source buffer? – Martin James Sep 02 '14 at 18:27
  • 2
    @Martin James Long ago and far away, the priorities of _heavily_ used `memcpy()` obliged it to be exceedingly lean. By the standards of the time, the "one compare" was bloat-ware and _typically_ not needed - further it prevents certain optimizations. It simply slowed the system. Hence the birth of `memmove()`. – chux - Reinstate Monica Sep 02 '14 at 19:08

3 Answers3

17

I've done some research on this in the past... on Linux, up until fairly recently, the implementation of memcpy() worked in a way that was similar enough to memmove() that overlapping memory wasn't an issue, and in my experience, other UNIXs were the same. This doesn't change the fact that this is undefined behavior according to the standard, and you are just lucky that on some platforms it sometimes works -- and memmove() is the standard-supported right answer.

However, in 2010, the glibc maintainers rolled out a new, optimized memcpy() that changed the behavior of memcpy() for some Intel core types where the C standard library is compiled to be faster, but no longer works like memmove() [1]. (I seem to recall also that this is new code triggered only for memory segments larger than 80 bytes). Interestingly, this caused things like the Linux version of Adobe's Flash player to break[2], as well as several other open-source packages (back in 2010 when Fedora Linux became the first to adopt the changed memcpy() in glibc).

JohnH
  • 2,713
  • 12
  • 21
  • That's exactly the answer I was looking for. So therefore it's perfectly possible that this behavior has been in that library for many years, never causing problems... – Michael Sep 02 '14 at 18:51
  • 1
    Exactly right. I was bitten by it when moving old code to a new server with the new changes. – JohnH Sep 02 '14 at 18:57
  • 1
    For further reading: "[The memcpy vs. memmove saga](https://www.win.tue.nl/~aeb/linux/misc/gcc-semibug.html)". There was, and still is today, some fallout as a result of that glibc change. – jwd Dec 07 '21 at 01:40
  • Thanks @jwd - Fantastic read... this one just keeps on giving. *sigh* – JohnH Dec 09 '21 at 16:08
14

memcpy() doesn't support overlapping memory. This allows for optimizations that won't work if the buffers do overlap.

There's not much to really look into, however, because C provides an alternative that does support overlapping memory: memmove(). Its usage is identical to memcpy(). You should use it if the regions might overlap, as it accounts for that possibility.

FatalError
  • 52,695
  • 14
  • 99
  • 116
  • The specification is clear about that. But I wonder whether the library regularly does such `memcpy` and nobody ever encountered this problem because the vast majority of implementations behave like the for loop OR whether I just have to assume a bug in the code driving the library. – Michael Sep 02 '14 at 18:20
  • 2
    If you found the library making such a call it's either 1) A bug in the library itself, or 2) A bug in how the library was used such that memory that shouldn't have overlapped did. The fact that there are cases where it *does* work, doesn't make it not a bug, however. – FatalError Sep 02 '14 at 18:22
0

memmove() can be used for the purpose. memcpy() [ man-page ] is defined with the condition that "source" should not overlap with destination.

To understand better, you can attempt to understand the memcpy() and memmove(), by an attempt to define your version of it. [ more info ]

parasrish
  • 3,864
  • 26
  • 32
  • The implementation your link provides for memmove is not optimal. You don't have to use an intermediate buffer whatsoever. – Raslanove Jul 02 '22 at 22:57
  • @Raslanove, bringing the context from above, that case here is to first reveal the catch around it, if that can be simplified. "Optimal" is a second step. More so, query above doesn't hint on those lines as yet. Thanks. – parasrish Jul 06 '22 at 07:14