In your question, you don't precise the type of sigmask or set, I will assume that they are 32 bits integer (int) and pointer on 32 bits integer (int *).
The line
curproc->sigmask &= ~(*set);
can be rewritten:
// get current mask
int mask = cupproc->sigmask;
int temp;
// get the signal(s) to set
temp = *set;
// invert the signal(s) to set to convert it signal to mask
temp = ~temp;
// update the mask with the signal to mask:
mask = mask & temp;
// save the new mask
cupproc->sigmask = mask;
So for instance, if you want to set the signal whose code is 3 (SIGINT in linux), you must first raise the third bit of an integer:
sig = 1 << ( 3 - 1) // to set only the third bit in sig.
Then call the function, set will be equal to &sig,
After line temp = ~temp;, temp will have all bits set but third
After line mask = mask & temp; all bits in mask will be kept but third that will be clear.