/[xy]/g is a RegEx literal. String.prototype.replace(..) takes RegEx or a substring as its first argument, and new substring or a function as its second.
/[xy]/g is a regular expression that matches a single character x or y in the original string.
replace(..) will replace all substrings that match the regex ('x' or 'y') to another string using the given function.
The function basically gives a random character for replace(..) to replace 'x' or 'y' with, and it does additional & operation for 'y' (which still gives a random character, but a different one).
If it's 'y', it is assigned (r&0x3|0x8).
In (r&0x3|0x8), r is the random number generated earlier. 0x3 is 11 in binary(base 2). r&0x3 extracts the lowest two bits of r. For example, if r, the random number, was 6. 6 in base 2 is 110. (110)&(11) extracts the lowest bit, so the result is (10) in base 2, which translate to 2 in base 10. So 2 is assigned. If the value of r&0x3 is 0, the |0x8 part makes it fall back to 0x8, which is 8. To learn more about this, please read up on bitwise operators
In summary, the 'x' and 'y' in the original string xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx is a placeholder, and it generates some kind of random id such as b9e08d48-0dfc-4a27-ba77-4a94b363b311, by replacing them with random characters.