.NET Users: I hope this answer saves someone a ton of grief.
As @Christophe Fondacci noted on 2015, the accepted solution worked great a few years ago.
Now it's 2017 2020 and the process is far easier and faster.
My use case is to validate in-app subscriptions, where my mobile app sends subscription purchase information to my RESTful server, which in turn contacts Google to validate a subscription purchase.
The strategy is to create a Service Account that will operate on your behalf.
- Sign into your Google Play Dev Console and click the app you're setting up. 
- Visit Settings->API access 
- Under Service Accounts, hit the Create Service Account button. 
- As of Jan 2017 a dialog with directions on setting up a service account appears.  The dialog takes you to the Google API Console; from there, - A) Click Create Service Account - B) Create the service account name that makes sense. Since we're interested in accessing Android Publisher Services, I chose "publisher". 
C) For Role, just choose something - you can change this later.
D) Choose "Furnish New private key" and choose P12 for .Net implementations. Don't lose this file!
- Now you're done with #4, you'll see your new Service Account listed; click "Grant Access" to enable it. 
- Tap on the link to "View permissions".  You should modify permissions based on your needs and API. 
To validate in-app purchases, visit the Cog->Change Permissions and enable the GLOBAL "Visibility" and "Manage Orders" permissions.
OK at this point you have configured everything on Google's end. Now to setup your server to server stuff.  I recommend creating
a .Net Console App to test out your implementation then offload it where needed.
- Add the Android Publisher Client Library from Nuget[1]
    PM> Install-Package Google.Apis.AndroidPublisher.v3
- Add the P12 file to your project root 
- Change the P12 Properties so "Build Action" is "Content" and "Copy To Output Directory" to "Copy if newer". 
- Implement something like this to test your access and fine tune [1] . 
    using System.Threading.Tasks;
    using System.Security.Cryptography.X509Certificates;
    using Google.Apis.Services;
    using Google.Apis.Auth.OAuth2;
    using Google.Apis.AndroidPublisher.v3;
    ...
    public Task<SubscriptionPurchase> GetSubscriptionPurchase(string packageName, string productId, string purchaseToken)
    {
        var certificate = new X509Certificate2(
                        "{{your p12 file name}}",
                        "{{ your p12 secret }}",
                        X509KeyStorageFlags.Exportable
                    );
        var credentials = new ServiceAccountCredential(
                new ServiceAccountCredential.Initializer("{{ your service account email }}")
                {
                    Scopes = new[] { AndroidPublisherService.Scope.Androidpublisher }
                }.FromCertificate(certificate));
    
        var service = new AndroidPublisherService(new BaseClientService.Initializer()
        {
            HttpClientInitializer = credentials,
            ApplicationName = "my server app name",
        });
        return service.Purchases.Subscriptions.Get(packageName, productId, purchaseToken).ExecuteAsync();
    }
Good luck, hope this helps someone.
Sources:
Using OAuth 2.0 for Server to Server Applications
.Net Client Library for Google.Apis.AndroidPublisher.v3[1]
1 
Updated 04/11/2020 - Google.Apis.AndroidPublisher.v2 EOL'd, use Google.Apis.AndroidPublisher.v3.
**, **grant_type=authorization_code**, redirect_uri=...– Brian White Aug 09 '14 at 18:04