I've successfully built an application that fetches an access and refresh token.
In my script I check if the access token is valid and if not I then use the refresh token to gain access $client->refreshToken($refreshToken);
Code in full,
    $refreshToken = '<REFRESH_TOKEN>';
    $client_id = '<CLIENT_ID>';
    $client_secret = '<CLIENT_SECRET>';
    // Setup infomation
    $client = new Google_Client();
    $client->setClientId($client_id);
    $client->setClientSecret($client_secret);
    $client->setAccessType("offline");
    $client->addScope("https://mail.google.com/");
    // If access token is not valid use refresh token
    if($client->isAccessTokenExpired()) {
        // Use refresh token
        $client->refreshToken($refreshToken);
    } else {
        // Use access token
        echo $client->setAccessToken($accessToken);
    }
However when trying to use the refresh token I get an excpetion :
Fatal error: Uncaught exception 'Google_Auth_Exception' with message 'Error refreshing the OAuth2 token, message: '{ "error" : "invalid_grant" }''