What format is "d"? PEM? or binary?
The format is Base64URL encoding or "Base 64 Encoding with URL and Filename Safe Alphabet" of RFC 4648 (see section 5, Table 2 on page 7).
how do I load it into a rsa struct?
OK, so OpenSSL is painful. To load it in a RSA struct, you need no convert n, e and d from Base64URL to Base64. Here's how I did it in Crypto++ (you can do it in OpenSSL, but its going to hurt):
string nz = "ofgWCuLjybRlzo0tZWJjNiuSfb4p4fAkd_wWJcyQoTbji9k0l8W26mPddx"
            "HmfHQp-Vaw-4qPCJrcS2mJPMEzP1Pt0Bm4d4QlL-yRT-SFd2lZS-pCgNMs"
            "D1W_YpRPEwOWvG6b32690r2jZ47soMZo9wGzjb_7OMg0LOL-bSf63kpaSH"
            "SXndS5z5rexMdbBYUsLA9e-KXBdQOS-UTo7WTBEMa2R2CapHg665xsmtdV"
            "MTBQY4uDZlxvb3qCo5ZwKh9kG4LT6_I5IhlJH7aGhyxXFvUK-DWNmoudF8"
            "NAco9_h9iaGNj8q2ethFkMLs91kzk2PAcDTW9gb54h4FRWyuXpoQ";
string ez = "AQAB";
string dz = "Eq5xpGnNCivDflJsRQBXHx1hdR1k6Ulwe2JZD50LpXyWPEAeP88vLNO97I"
            "jlA7_GQ5sLKMgvfTeXZx9SE-7YwVol2NXOoAJe46sui395IW_GO-pWJ1O0"
            "BkTGoVEn2bKVRUCgu-GjBVaYLU6f3l9kJfFNS3E0QbVdxzubSu3Mkqzjkn"
            "439X0M_V51gfpRLI9JYanrC4D4qAdGcopV_0ZHHzQlBjudU2QvXt4ehNYT"
            "CBr6XCLQUShb1juUO1ZdiYoFaFQT5Tw8bGUl_x_jTj3ccPDVZFD9pIuhLh"
            "BOneufuBiB4cS98l2SR_RQyGWSeWjnczT0QU91p1DhOVRuOopznQ";
string nn, ee, dd;
// First, convert Base64URL encoding to Base64
std::replace(nz.begin(), nz.end(), '-', '+');
std::replace(ez.begin(), ez.end(), '-', '+');
std::replace(dz.begin(), dz.end(), '-', '+');
std::replace(nz.begin(), nz.end(), '_', '/');
std::replace(ez.begin(), ez.end(), '_', '/');
std::replace(dz.begin(), dz.end(), '_', '/');
// Now, Base64 decode
StringSource ss1(nz, true, new Base64Decoder(new StringSink(nn)));
StringSource ss2(ez, true, new Base64Decoder(new StringSink(ee)));
StringSource ss3(dz, true, new Base64Decoder(new StringSink(dd)));
EDIT: Crypto++ now has the Base64URLEncoder and Base64URLDecoder classes, so you don't need the find/replace operations.
After the code above runs, nn, ee and dd are binary strings (i.e., non-ASCII characters). From there, you can load them into Integer's and get the Base 10 string with:
Integer n((byte*)nn.data(), nn.size());
Integer e((byte*)ee.data(), ee.size());
Integer d((byte*)dd.data(), dd.size());
cout << "N: " << endl << n << endl << endl;
cout << "E: " << endl << e << endl << endl;
cout << "D: " << endl << d << endl << endl;
$ ./cryptopp-test.exe
N: 
20446702916744654562596343388758805860065209639960173505037453331270270518732245
08977372301204320323609709562340204469011575537734525469644875960570778896584888
95017468362112062706438336639499925362469853626937363871851454247879222415857219
92924045675229348655595626434390043002821512765630397723028023792577935108185822
75369257422156693093780503115582009714681996492027000881132703628678639279359312
17624250488602118597634417704467037220158572506211078553986931332640811506974231
88751482418465308470313958250757758547155699749157985955379381294962058862159085
915015369381046959790476428631998204940879604226680285601.
E: 
65537.
D:
23583109899396195101799862623499368829246520235662137651186064319555667005065389
11356936879137503597382515919515633242482643314423192704128296593672966061810149
31632061789402182278402640746140338406535182197235078430096761014345948432406842
76746396884059179774424728049430754391920261073195321175575450790865379829879825
22396626690057355718157403493216553255260857777965627529169195827622139772389760
13057175483467867984218114225248961766503010944557397801270779301059273764049922
00150833924259148778478404572782464027609558833769999511998277062853834711506435
61410605789710883438795588594095047409018233862167884701.
OpenSSL needs n, e, d, p and q for private key operations. d mod p-1, d mod q-1 and inv q mod p are optional. With only n, e, d, you need to solve for the missing parameters (at minimum p and q). The two tough ones are p and q. Here's the Crypto++ code to solve for them (feel free to convert to OpenSSL):
Integer p, q;
RSA_solve(n, e, d, p, q);
cout << "P: " << endl << p << endl << endl;
cout << "Q: " << endl << q << endl << endl;
And:
void RSA_solve(const Integer& n, const Integer& e, const Integer& d, Integer& p, Integer& q)
{
    AutoSeededRandomPool prng;
    Integer g = 1;
    unsigned int SAFETY = 0;
STEP_1:
    const Integer k = e * d - 1;
    if(!k.IsEven())
        throw runtime_error("e * d - 1 is not even");
STEP_2:
    // g = 3, 5, 7, ...
    g += 2; while(!VerifyPrime(prng, g)) g += 2;
    Integer t = k;
STEP_3:
    if(SAFETY++ > 128)
        throw runtime_error("could not factor n");
    if(!t.IsEven())
        goto STEP_2;
    t /= 2;
    Integer x = a_exp_b_mod_c(g, t, n);
STEP_4:
    if(!(x > 1))
        goto STEP_3;
    Integer y = GCD(x-1, n);
    if(!(y > 1))
        goto STEP_3;
    p = std::max(y, n/y);
    q = std::min(y, n/y);
    Integer check = p * q;
    if(n != check)
        throw runtime_error("n != p * q");
}
That results in:
P: 
15737705590244743839558616502896029191493197327877753279847020015603526753735923
90718294084119093232085749598005372477289597182368848096852332845373492076546615
30801859889389455120932077199406250387226339056140578989122526711937239401762061
949364440402067108084155200696015505170135950332209194782224750221639.
Q: 
12992175256740635899099334754006444501823007340248226099417932857332386190837921
12746269565434716649972371852989646481333243433270528522640603220881224011247812
49085873464824282666514908127141915943024862618996371026577302203267804867959037
802770797169483022132210859867700312376409633383772189122488119155159.
d mod p-1, d mod q-1 and inv q mod p are left as an exercise to the reader (but they are easy, especially in Crypto++). Your modified RSA_solve might look like:
void RSA_solve(const Integer& n, const Integer& e, const Integer& d,
               Integer& p, Integer& q,
               Integer& dmodp1, Integer& dmodq1, Integer& invqmodp)
Now, switch to OpenSSL with your Base 10 (decimal) strings:
const char nz[] =
    "20446702916744654562596343388758805860065209639960173505037453331270270518732245"
    "08977372301204320323609709562340204469011575537734525469644875960570778896584888"
    "95017468362112062706438336639499925362469853626937363871851454247879222415857219"
    "92924045675229348655595626434390043002821512765630397723028023792577935108185822"
    "75369257422156693093780503115582009714681996492027000881132703628678639279359312"
    "17624250488602118597634417704467037220158572506211078553986931332640811506974231"
    "88751482418465308470313958250757758547155699749157985955379381294962058862159085"
    "915015369381046959790476428631998204940879604226680285601";
const char ez[] = "65537";
const char dz[] =
    "23583109899396195101799862623499368829246520235662137651186064319555667005065389"
    "11356936879137503597382515919515633242482643314423192704128296593672966061810149"
    "31632061789402182278402640746140338406535182197235078430096761014345948432406842"
    "76746396884059179774424728049430754391920261073195321175575450790865379829879825"
    "22396626690057355718157403493216553255260857777965627529169195827622139772389760"
    "13057175483467867984218114225248961766503010944557397801270779301059273764049922"
    "00150833924259148778478404572782464027609558833769999511998277062853834711506435"
    "61410605789710883438795588594095047409018233862167884701";
const char pz[] =
    "15737705590244743839558616502896029191493197327877753279847020015603526753735923"
    "90718294084119093232085749598005372477289597182368848096852332845373492076546615"
    "30801859889389455120932077199406250387226339056140578989122526711937239401762061"
    "949364440402067108084155200696015505170135950332209194782224750221639";
const char qz[] =
    "12992175256740635899099334754006444501823007340248226099417932857332386190837921"
    "12746269565434716649972371852989646481333243433270528522640603220881224011247812"
    "49085873464824282666514908127141915943024862618996371026577302203267804867959037"
    "802770797169483022132210859867700312376409633383772189122488119155159";
using BN_ptr = std::unique_ptr<BIGNUM, decltype(&::BN_free)>;
using RSA_ptr = std::unique_ptr<RSA, decltype(&::RSA_free)>;
using EVP_PKEY_ptr = std::unique_ptr<EVP_PKEY, decltype(&::EVP_PKEY_free)>;
using EVP_MD_CTX_ptr = std::unique_ptr<EVP_MD_CTX, decltype(&::EVP_MD_CTX_destroy)>;
#define UNUSED(x) ((void)x)
int main(int argc, char* argv[])
{
    UNUSED(argc); UNUSED(argv);
    int rc;
    long err;
    RSA_ptr rsa(RSA_new(), ::RSA_free);
    BIGNUM *n = NULL, *e = NULL, *d = NULL, *p = NULL, *q = NULL;
    rc = BN_dec2bn(&n, nz);
    if(rc == 0 || n == NULL) {
        cerr << "BN_dec2bn failed for n" << endl;
        exit(1);
    }
    rsa->n = n;
    rc = BN_dec2bn(&e, ez);
    if(rc == 0 || e == NULL) {
        cerr << "BN_dec2bn failed for e" << endl;
        exit(1);
    }
    rsa->e = e;
    rc = BN_dec2bn(&d, dz);
    if(rc == 0 || d == NULL) {
        cerr << "BN_dec2bn failed for d" << endl;
        exit(1);
    }
    rsa->d = d;
    rc = BN_dec2bn(&p, pz);
    if(rc == 0 || p == NULL) {
        cerr << "BN_dec2bn failed for p" << endl;
        exit(1);
    }
    rsa->p = p;
    rc = BN_dec2bn(&q, qz);
    if(rc == 0 || q == NULL) {
        cerr << "BN_dec2bn failed for q" << endl;
        exit(1);
    }
    rsa->q = q;
    [Exercise left to the reader]
    rc = RSA_check_key(rsa.get());
    err = ERR_get_error();
    if(rc != 1) {
        cerr << "RSA_check_key failed, error 0x" << std::hex << err << endl;
        exit(1);
    }
    [Continues at next question below]
    ...
}
Here are the fields you need to provide in struct rsa (from <openssl dir>/crypto/rsa/rsa.h):
struct rsa_st
    {
    ...
    /* functional reference if 'meth' is ENGINE-provided */
    ENGINE *engine;
    BIGNUM *n;
    BIGNUM *e;
    BIGNUM *d;
    BIGNUM *p;
    BIGNUM *q;
    BIGNUM *dmp1;
    BIGNUM *dmq1;
    BIGNUM *iqmp;
    ...
    };
so that I can feed this into an EVP_SignFinal call...
EVP_SignFinal needs an EVP_PKEY and you have an RSA. So:
EVP_PKEY_ptr pkey(EVP_PKEY_new(), ::EVP_PKEY_free);
rc = EVP_PKEY_set1_RSA(pkey.get(), rsa.get());
err = ERR_get_error();
if(rc != 1) {
    cerr << "EVP_PKEY_set1_RSA failed, error 0x" << std::hex << err << endl;
    exit(1);
}
The set1 means the reference count is bumped on the RSA*. That's OK. If it s was set0, you would have had to release your copy (that is, use rsa.release() rather than rsa.get()) to avoid a double free.
EVP_MD_CTX_ptr ctx(EVP_MD_CTX_create(), ::EVP_MD_CTX_destroy);
EVP_MD_CTX_init(ctx.get());
const EVP_MD* md = EVP_sha256();
rc = EVP_SignInit(ctx.get(), md);
err = ERR_get_error();
if(rc != 1) {
    cerr << "EVP_SignInit_ex failed, error 0x" << std::hex << err << endl;
    exit(1);
}
const char message[] = "Now is the time for all good men...";
rc = EVP_SignUpdate(ctx.get(), message, (unsigned int)sizeof(message));
err = ERR_get_error();
if(rc != 1) {
    cerr << "EVP_SignUpdate failed, error 0x" << std::hex << err << endl;
    exit(1);
}
const unsigned int req = std::max(EVP_MD_size(md), EVP_PKEY_size(pkey.get()));
unique_ptr<unsigned char[]> signature(new unsigned char[req]);
unsigned int size = req;
rc = EVP_SignFinal(ctx.get(), signature.get(), &size, pkey.get());
err = ERR_get_error();
if(rc != 1) {
    cerr << "EVP_SignFinal failed, error 0x" << std::hex << err << endl;
    exit(1);
}
size = std::min(size, (unsigned int)EVP_MD_size(md));
cout << "Signature: ";
for(unsigned i = 0; i < size; i++)
    cout << std::hex << (signature[i] & 0xFF);
cout << endl;
Here's the Pastebin of the Crypto++ code used above: http://pastebin.com/9Rm7bxZp.
Here's the Pastebin of the OpenSSL code used above: http://pastebin.com/aGVpj4FW.
Here's the output of the OpenSSL program:
$ ./openssl-test.exe 
Signature: 78f2c9af23b9a2a42e3b57dec454fa43ea6627992f48d40a33da6a7c93f98b4