GAE Application needing to create Expiring Signed URL's to Google Cloud Storage

Question

I was successfully able to create a stand-alone Java Application that creates Expiring Signed URL's to assets sitting in Google Cloud Storage. However, I have been unsuccessful in figuring out how to create Expiring Signed URL's to these same assets through AppEngine.

How can I create a Expiring Signed URL to Google Cloud Storage Assets that I can return to client applications?

Here is my Java Application that works:

import java.io.FileInputStream;
import java.io.IOException;
import java.net.URLEncoder;
import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.Signature;
import java.security.UnrecoverableKeyException;
import java.util.Calendar;

import org.apache.commons.codec.binary.Base64;

public class GCSSignedURL {

public static void main(String[] args) throws Exception {

    final String googleAccessId = "XXXXXXXXXXXX@developer.gserviceaccount.com";
    final String keyFile = "D:\\XXXXXXXXXXXXXXXXXXXXXXXXXXXXX-privatekey.p12";
    final  String keyPassword = "notasecret";
    Calendar calendar = Calendar.getInstance();
    calendar.add(Calendar.MINUTE, 30);
    String httpVerb = "GET";
    String contentMD5 = "";         
    String contentType = "";
    long expiration = calendar.getTimeInMillis();
    String canonicalizedExtensionHeaders = ""; 
    String canonicalizedResource = "/myproj/foo.txt";

    String stringToSign = 
            httpVerb + "\n" + 
            contentMD5 + "\n" + 
            contentType + "\n" + 
            expiration + "\n" + 
            canonicalizedExtensionHeaders + 
            canonicalizedResource;

    PrivateKey pkcsKey = loadKeyFromPkcs12(keyFile, keyPassword.toCharArray());
    String signature = signData(pkcsKey, stringToSign);     
    String baseURL = "https://storage.cloud.google.com/myproj/foo.txt";     
    String urlEncodedSignature = URLEncoder.encode(signature, "UTF-8");
    String url = baseURL + "?GoogleAccessId=" + googleAccessId + "&Expires=" + expiration + "&Signature=" + urlEncodedSignature;

    System.out.println(url);
}

private static PrivateKey loadKeyFromPkcs12(String filename, char[] password)
        throws Exception {
    FileInputStream fis = new FileInputStream(filename);
    KeyStore ks = KeyStore.getInstance("PKCS12");
    try {
        ks.load(fis, password);
    } catch (IOException e) {
        if (e.getCause() != null
                && e.getCause() instanceof UnrecoverableKeyException) {
            System.err.println("Incorrect password");
        }
        throw e;
    }
    return (PrivateKey) ks.getKey("privatekey", password);
}

private static String signData(PrivateKey key, String data)
        throws Exception {
    Signature signer = Signature.getInstance("SHA256withRSA");
    signer.initSign(key);
    signer.update(data.getBytes("UTF-8"));
    byte[] rawSignature = signer.sign();
    String encodedSignature = new String(Base64.encodeBase64(rawSignature,
            false), "UTF-8");
    return encodedSignature;
}

}

Here is what I have tried:

public String signUrl(Long _songId, String _format) throws ResourceNotFoundException
{
    final String googleAccessId = "XXXXXXXXXXXXXXXXXX@developer.gserviceaccount.com";

    AppIdentityService service = AppIdentityServiceFactory.getAppIdentityService();

    Calendar calendar = Calendar.getInstance();
    calendar.add(Calendar.MINUTE, 5);       
    String httpVerb = "GET";
    String contentMD5 = "";
    String contentType = "";
    long expiration = calendar.getTimeInMillis();       
    String canonicalizedExtensionHeaders = "";      
    String canonicalizedResource = "/myproj/foo.txt";
    String stringToSign = 
            httpVerb + "\n" + 
            contentMD5 + "\n" + 
            contentType + "\n" + 
            expiration + "\n" + 
            canonicalizedExtensionHeaders + 
            canonicalizedResource;  

    SigningResult key = service.signForApp(stringToSign.getBytes());
    String baseURL = "https://storage.cloud.google.com/myproj/foo.txt";
    String encodedUrl = baseURL + "?GoogleAccessId=" + googleAccessId + "&Expires=" + expiration
            + "&Signature=" + key.getKeyName();

    return encodedUrl;
}

The result is an expiring URL but requires me to authenticate with my google email / password so the signing isn't working properly.

I was able to finally generate an encoded URL using Fabio's suggestion, however, I now get:

<?xml version="1.0" encoding="UTF-8"?>
-<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated 
does not match the signature you provided. Check your Google secret key and signing 
method.</Message>
<StringToSign>
GET 1374729586 /[my_bucket]/[my_folder]/file.png</StringToSign>    
</Error>

The code I am using to generate the URL is:

AppIdentityService service = AppIdentityServiceFactory.getAppIdentityService();
    final String googleAccessId = service.getServiceAccountName();
    String url = songUrl(_songId, _format);  
    Calendar calendar = Calendar.getInstance();
    calendar.add(Calendar.MINUTE, 10);
    String httpVerb = "GET";
    String contentMD5 = ""; 
    String contentType = ""; 
    long expiration = calendar.getTimeInMillis()/1000L;
    String canonicalizedExtensionHeaders = "";      
    String canonicalizedResource = "/[my_bucket]/[my_folder]/file.png";
    String stringToSign = 
            httpVerb + "\n" + 
            contentMD5 + "\n" + 
            contentType + "\n" + 
            expiration + "\n" + 
            canonicalizedExtensionHeaders + 
            canonicalizedResource;

    try 
    {
        String baseURL = http://[my_bucket].commondatastorage.googleapis.com/[my_folder]/file.png;  
        SigningResult signingResult = service.signForApp(stringToSign.getBytes());
        String encodedSignature = new String(Base64.encodeBase64(signingResult.getSignature(), false), "UTF-8");
        String encodedUrl = baseURL + "?GoogleAccessId=" + googleAccessId + "&Expires=" + expiration
                + "&Signature=" + encodedSignature;

        return encodedUrl;
    } 
    catch (UnsupportedEncodingException e) 
    {
        throw new ResourceNotFoundException("Unable to encode URL.  Unsupported encoding exception.", e);
    }       

Answer 1

Two things:

For googleAccessId use:

String googleAccessId = service.getServiceAccountName();

And for Signature use:

SigningResult signingResult = service
            .signForApp(stringToSign.getBytes());
String encodedSignature = new String(Base64.encodeBase64(
            signingResult.getSignature(), false), "UTF-8");

That's what worked for me. See below a sample signer class:

public class GcsAppIdentityServiceUrlSigner  {

    private static final int EXPIRATION_TIME = 5;
    private static final String BASE_URL = "https://storage.googleapis.com";
    private static final String BUCKET = "my_bucket";
    private static final String FOLDER = "folder";


    private final AppIdentityService identityService = AppIdentityServiceFactory.getAppIdentityService();

    public String getSignedUrl(final String httpVerb, final String fileName) throws Exception {
        final long expiration = expiration();
        final String unsigned = stringToSign(expiration, fileName, httpVerb);
        final String signature = sign(unsigned);

        return new StringBuilder(BASE_URL).append("/")
                .append(BUCKET)
                .append("/")
                .append(FOLDER)
                .append("/")
                .append(fileName)
                .append("?GoogleAccessId=")
                .append(clientId())
                .append("&Expires=")
                .append(expiration)
                .append("&Signature=")
                .append(URLEncoder.encode(signature, "UTF-8")).toString();
    }

    private static long expiration() {
        final long unitMil = 1000l;
        final Calendar calendar = Calendar.getInstance();
        calendar.add(Calendar.MINUTE, EXPIRATION_TIME);
        final long expiration = calendar.getTimeInMillis() / unitMil;
        return expiration;
    }

    private String stringToSign(final long expiration, String filename, String httpVerb) {
        final String contentType = "";
        final String contentMD5 = "";
        final String canonicalizedExtensionHeaders = "";
        final String canonicalizedResource = "/" + BUCKET + "/" + FOLDER + "/" + filename;
        final String stringToSign = httpVerb + "\n" + contentMD5 + "\n" + contentType + "\n"
                + expiration + "\n" + canonicalizedExtensionHeaders + canonicalizedResource;
        return stringToSign;
    }

    protected String sign(final String stringToSign) throws UnsupportedEncodingException {
        final SigningResult signingResult = identityService
                .signForApp(stringToSign.getBytes());
        final String encodedSignature = new String(Base64.encodeBase64(
                signingResult.getSignature(), false), "UTF-8");
        return encodedSignature;
    }

    protected String clientId() {
        return identityService.getServiceAccountName();
    }
}

Answer 2

1/ Add the google api google-api-services-storage to your dependencies

2/ Then you need to create ServiceAccountAuthCredentials object from your service account ID and primary key :

serviceAccountAuthCredentials = AuthCredentials.createFor(resources.getString("authentication.p12.serviceAccountId"), pk);

3/ Finally you generate the signed URL from the ServiceAccountAuthCredentials object and the bucket and file name (no need to generate a String to sign) :

public String getSignedURL(String bucket, String fileName) throws IOException, GeneralSecurityException {
    BlobId blobId = BlobId.of(bucket, fileName);
    Blob blob = cloudStorageService.get(blobId);
    URL signedURL = blob.signUrl(durationSignedURLAvailable, TimeUnit.MINUTES, com.google.cloud.storage.Storage.SignUrlOption.signWith(serviceAccountAuthCredentials));
    return signedURL.toString();
}

This works fine for me.

Answer 3

Using storage.cloud.google.com is requesting cookie based authenticated downloads. Changing:

String baseURL = "https://storage.cloud.google.com/myproj/foo.txt"

to

String baseURL = "https://storage.googleapis.com/myproj/foo.txt"

should work better.

I'm not sure why you're only seeing this when using the URL created via App Engine. Maybe you're not logged in to Google when testing the App Engine application? Or running it in the local dev server?

See the reference URIs section of the docs for details on possible request URIs.