The resolver acted properly with the following minimum set of code:
public class XsdUtils {
    static {
        System.setProperty("java.protocol.handler.pkgs", "org.fao.oek.protocols");
    }
    private static XMLCatalogResolver cr;
    public static synchronized XMLCatalogResolver getResolver() {
        if (cr == null) {
            cr = new XMLCatalogResolver(new String[] { "classpath:xml-catalog.xml" });
        }
        return cr;
    }
    public static void main(String[] args) throws MalformedURLException, IOException {
        XMLCatalogResolver resolver = getResolver();
        URL url0 = new URL("classpath:xml-catalog.xml");
        URL url1 = new URL(resolver.resolveURI("http://www.loc.gov/mods/v3"));
        url0.openConnection();
        url1.openConnection();
    }
}
You can alternatively specify java.protocol.handler.pkgs as a JVM argument:
java -Djava.protocol.handler.pkgs=org.fao.oek.protocols ...
The Handler class was implemented as follows:
package org.fao.oek.protocols.classpath;
import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;
public class Handler extends java.net.URLStreamHandler {
    @Override
    protected URLConnection openConnection(URL u) throws IOException {
        String resource = u.getPath();
        if (!resource.startsWith("/")) resource = "/" + resource;
        System.out.println(getClass().getResource(resource));
        return getClass().getResource(resource).openConnection();
    }
}
It is important to have the forward slash ("/") when requesting the resource, as answered by this Stack Overflow question: "open resource with relative path in java."
Note the main method in XsdUtils. The output to the program when xml-catalog.xml and mods-3.3.xsd are on the classpath but not in a JAR is:
file:/workspace/8412798/target/classes/xml-catalog.xml
file:/workspace/8412798/target/classes/org/me/myapp/xsd/mods-3.3.xsd
The output to the program when the files are in a JAR is:
jar:file:/workspace/8412798/target/stackoverflow.jar!/xml-catalog.xml
jar:file:/workspace/8412798/target/stackoverflow.jar!/org/me/myapp/xsd/mods-3.3.xsd
With respect to this code in the original question:
new org.fao.oek.protocols.classpath.Handler(XsdUtils.class.getClassLoader())
your Handler does not need a specific class loader unless you have configured your application to use a special class loader, like one extended from URLClassLoader.
"A New Era for Java Protocol Handlers" is a good resource about protocol handlers.
Just to bring everything full circle, the following class uses XsdUtils.getResolver() to parse XML. It validates against the schemas specified in the XMLCatalogResolver:
public class SampleParser {
    public static void main(String[] args) throws Exception {
        String xml = "<?xml version=\"1.0\"?>" + //
                "<mods ID=\"id\" version=\"3.3\" xmlns=\"http://www.loc.gov/mods/v3\">" + //
                "<titleInfo></titleInfo>" + //
                "</mods>";
        ByteArrayInputStream is = new ByteArrayInputStream(xml.getBytes());
        XMLReader parser = XMLReaderFactory.createXMLReader(org.apache.xerces.parsers.SAXParser.class.getName());
        parser.setFeature("http://xml.org/sax/features/validation", true);
        parser.setFeature("http://apache.org/xml/features/validation/schema", true);
        parser.setFeature("http://apache.org/xml/features/validation/schema-full-checking", true);
        parser.setProperty("http://apache.org/xml/properties/internal/entity-resolver", XsdUtils.getResolver());
        parser.setErrorHandler(new ErrorHandler() {
            @Override
            public void error(SAXParseException exception) throws SAXException {
                System.out.println("error: " + exception);
            }
            @Override
            public void fatalError(SAXParseException exception) throws SAXException {
                System.out.println("fatalError: " + exception);
            }
            @Override
            public void warning(SAXParseException exception) throws SAXException {
                System.out.println("warning: " + exception);
            }
        });
        parser.parse(new InputSource(is));
    }
}