10

background

For two years I've been happily accessing my Gmail accounts with neomutt.

I'm sync'ing between neomutt locally and my online Gmail account with mbsync and a "2-Step Verification" app password (Sign in with App Passwords).

sending email with msmtp, until now

To send a Gmail with neomutt is trickier, because msmtp requires an unexpired token from the Gmail API. Fortunately GitHub user tenllado provided the only working open-source solution that I've been able to find, his script oauth2token. I adapted it as oauth2tool.sh. The steps for this to function are:

1 prepare - get my Gmail OAuth 2.0 credentials

  1. Use Gmail API's Python Quickstart to get my credentials, which look like this:
    • my Client ID: xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
    • my Client Secret: xxxxxxxxxxxxxxxxxxxxxxxx
  2. Grab a copy of oauth2.py (Code "the refresh token lasts indefinitely").
  3. Get the immortal refresh token: $ python2 oauth2.py --user=my@gmail.com --client_id=<myCI> --client_secret=<myCS> --generate_oauth2_token and follow the instructions. It looks like this:
    • refresh token: 1//03xxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxx

2 prepare - configure msmtprc

account my
auth oauthbearer
host smtp.gmail.com
port 587
from my@gmail.com
user my@gmail.com
passwordeval bash oauth2tool.sh my

3 use - send emails from the command line, until now

Then, when I send an email echo "test" | msmtp -a my <target_email>, my oauth2tool.sh pulls up a valid token. The way it does this is it grabs the token with pass if it's not expired, otherwise it grabs a new one with python2 oauth2.py --user=my@gmail.com --client_id=<myCI> --client_secret=<myCS> --refresh_token=<myRT>.

With all this, I could easily send emails from my Gmail accounts from the command line, until now.

now, oob is no longer allowed

Now my once immortal refresh tokens are being expired, and I can't renew them because Gmail's oauth2.py is using redirect_uri = urn:ietf:wg:oauth:2.0:oob, which is deprecated.

Making Google OAuth interactions safer by using more secure OAuth flows "OAuth out-of-band (oob) flow will be deprecated".

How to continue sending with msmtp?

OAuth 2.0 for Mobile & Desktop Apps "Loopback IP address (macOS, Linux, Windows desktop)" seems to be the way forward, but I'd need a few weeks of free time, which I don't have, to figure out how. Any ideas out there?

Related question: Google Cloud: OAuth clients in test mode that are using the OAuth OOB flow.

joharr
  • 425

2 Answers2

0

getmail6 solution

getmail6 have somewhat solved this problem for Gmail (and Microsoft Office 365) with their Python script getmail-gmail-xoauth-tokens, though with the huge caveat that the Gmail API now limits your refresh token to just 1 hour, so the last step described below, involving brief browser interaction and generating Google security alerts, would need to be repeated regularly, thus making this a solution of little practical worth. (getmailrc-examples does warn of this - "Unfortunately... needs to be repeated regularly.") In discovering this, I successfully worked through Akkana Peck's instructions, Sending Mail via Gmail using OAuth2 (2022 Edition) (as suggested in the comment from @GuenterRoeck).

1 get your OAuth 2.0 Client ID & Secret

(Note: This might not be the only or best path, but it works. In 2020 I got these credentials more easily by the now obsolete route described in the OP, and they're still valid.)

  1. Go to Create a Google Cloud project and, as instructed there, open the Google Cloud console which tells you how to navigate on to New Project where you need to choose a Project name - I chose getmail6.
  2. In APIs & Services select Enabled APIs and services and click on + ENABLE APIS AND SERVICES.
  3. Select the Gmail API, and ENABLE it.
  4. Select Credentials and click on + CREATE CREDENTIALS to get to Create OAuth client ID where you click on Configure consent screen, which is where, it seems, we need to now continue as if we're app creators, which seems to me absurd, but there's no choice. Not being a Google Workspace user, I was obliged to select External ("Available to any test user with a Google Account. Your app will start in testing mode and will only be available to users you add to the list of test users. Once your app is ready to push to production, you may need to verify your app."). We're obliged then to specify our imaginary App name - I chose "msmtpGmail", and our User support email, which is that of the account that you're doing all this from, same for our Developer contact information. Then Save and continue.
  5. Back to Create OAuth client ID for Application type - I chose Desktop app, and Name - I chose "msmtp", which, finally, got me my Client ID and Client Secret. I downloaded them as client_secret_nnnnnnnnnnnn-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com.json, and filed that away safely.
  6. Essential last step is to add yourself in Test users (because Google didn't do that for you) at OAuth consent screen.

2 get your OAuth 2.0 access_token (and refresh_token)

Here we leverage some of the hard work of getmail6. You're going to need their getmail-gmail-xoauth-tokens Python script.

1 make your (secret) json

As instructed in getmail6's getmailrc-examples (- search for "mail.google.com"), make this JSON - I named mine getmail6.json:

{"scope": "https://mail.google.com/",
"user": "yours@gmail.com",
"client_id": "yours",
"client_secret": "yours",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs"}

- fill in yours...

2 generate your access_token (and refresh_token)

python3 getmail-gmail-xoauth-tokens --init your.json (fill in your) > "Google hasn’t verified this app" > Continue > "<yourApp> wants access to your Google Account" > Continue, which appends into your.json these lines:

"access_token": "<extremely_long_key>",
"expires_at": <now+3600s>,
"refresh_token": "<another_extremely_long_key>"}

- which are what we've been looking for, but that 1h expiry is prohibitively awkward...

3 adjust your ~/.msmtprc

Again, leveraging getmail-gmail-xoauth-tokens:

tls on
tls_trust_file  /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log
account your
auth oauthbearer
host smtp.gmail.com
port 587
from your@gmail.com
user your@gmail.com
passwordeval python3 getmail-gmail-xoauth-tokens your.json

- fill in instances of your.

4 send a test mail

Quickly, you've got an hour before your refresh token has gone stale (change your):

echo "test send from msmtp using Gmail's OAuth 2.0" | msmtp -a your <a_target_email>
joharr
  • 425
0

mutt_oauth2.py possible route doesn't work either

I'm including this 2nd answer for those that might be tempted to try this route.

Following the relatively recent advice posted by Simon Dobson Local email from Office365 using OAUTH2 with mbsync and with some clues from Brett Presnell Reading Email with Emacs who uses pizauth, I got myself through to these commands that make use of neomutt/contrib/oauth2/mutt_oauth2.py:

for my ordinary Gmail account

  1. Using Thunderbird's credentials as exposed in releases-comm-central/mailnews/base/src/OAuth2Providers.sys.mjs, python <path_to_local_copy_of_mutt_oauth2.py --provider microsoft --client-id 406964657835-aq8lmia8j95dhl1a2bvharmfk3t1hgqj.apps.googleusercontent.com --client-secret kSmqreRr0qwBWJgbf5Y-PjSU --authflow localhostauthcode --email name@gmail.com -a tokens_Google.gpg --encryption-pipe 'gpg --encrypt --recipient <gpg_id> gets a URL for a Google Sign-in page which, once completed, goes on to a page that warns "Mozilla Thunderbird Email wants access to your Google Account", but, in my cases, clicking through then gets me "Unable to connect" to server at localhost:36697, and my terminal (Alacritty) reports "Did not obtain an authcode." and becomes unusable, requiring an OS reboot!
  2. Using, instead, my credentials created over two years ago when I tried the getmail6 route, I got through to a page stating "Google hasn't verified this app". Advanced > Go to Quickstart (unsafe) got me a page stating "Quickstart wants access to your Google Account". Continue got me a page stating "Unable to connect" to server at localhost:36697, and Alacritty again reported "Did not obtain an authcode."

for my ordinary Microsoft account

Using Microsoft credentials published by Simon Dobson and others, python <path_to_local_copy_of_mutt_oauth2.py --provider microsoft --client-id 08162f7c-0fd2-4200-a84a-f25a4db0b584 --client-secret 'TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82' --authflow localhostauthcode --email <address> -a tokens_Microsoft.gpg --encryption-pipe 'gpg --encrypt --recipient <gpg_id> gets a URL for a Microsoft Sign-in page but it tells me "You can't sign in here with a personal account. Use your work or school account instead." Same with all my other aliases current and defunct.

conclusion: mutt is no longer an option for ordinary Google or Microsoft accounts

- because mbsync (& consequently msmtp) can no longer work with an ordinary Google or Microsoft account. I wish this wasn't so, and I'm guessing all this heavy security is because they both provide so many services other than simply email.

joharr
  • 425