std::malloc returns, first and foremost, a pointer. nullptr is not a pointer (that is literally the whole point of it). Therefore, std::malloc cannot return nullptr.
On failure, std::malloc will return a pointer value which is defined to be the "null pointer constant". nullptr is a non-pointer type that is implicitly convertible to the "null pointer constant", and is equality-comparable to any pointer value (returning equals if the pointer value is the "null pointer constant").
So if you test the return value against nullptr, it will test true if and only if std::malloc failed.
The macro NULL results (for C++) in the integer literal 0 (note: it can also result in nullptr, but pretty much no implementation does so). By C++'s rules, the integer literal 0 is implicit convertible to the null pointer constant, and is comparable to any pointer value (resulting in equals if the pointer value is the "null pointer constant").
So if you test the return value against NULL, it will test true if and only if std::malloc failed.