43

The documentation describing how to connect to a kerberos secured endpoint shows the following:

curl -i --negotiate -u : "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=..."

The -u flag has to be provided but is ignored by curl.

Does the --negotiate option cause curl to look for a keytab that was created beforehand with the kinit command, or will curl prompt for credentials?

If it looks for a keytab file, what filename will the command be looking for?

John R Smith
  • 848
  • 7
  • 18
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
  • 2
    curl tries to get a valid ticket **in the cache** (cf. env variable `KRB5CCNAME` or by default `FILE:/tmp/krb5cc_$(id -u)` on Unix / `API:` on Windows) -- it does not care how the ticket was created. – Samson Scharfrichter Jul 21 '16 at 16:49
  • Caveat: the Windows versions of curl are compiled to support the Microsoft SSPI libraries, not the GSSAPI libraries that are required by Hadoop REST services. – Samson Scharfrichter Jul 21 '16 at 16:52
  • BTW a keytab is a file containing a hashed password, created by `ktutil` on Linux or by an Active Directory utility like `ktpass.exe`. It is **used** by `kinit`, not **created**. – Samson Scharfrichter Jul 21 '16 at 16:54
  • 2
    Even better: if you update your question so that it is more specific, I can make a more specific answer *(e.g. do you want to SPNEGO each time, or just once to get the reusable Delegation Token? do you want to know why HttpFs uses a signed cookie instead of the actual Token like WebHDFS does? are you worried about the lack of consistency checks using WebHDFS vs. using the Java API? etc.)* – Samson Scharfrichter Jul 21 '16 at 21:47
  • An extra hint at no additional cost: http://stackoverflow.com/questions/37459073/accessing-kerberos-secured-webhdfs-without-spnego/37480236#37480236 – Samson Scharfrichter Jul 21 '16 at 21:54
  • Thanks Samsom. I've just started exploring this area, so I don't have the knowledge to make my question more specific yet. I will post more as my knowledge builds. – Chris Snow Jul 22 '16 at 07:39
  • @SHC Were you able to properly resolve the issue? Otherwise I would add a qualified answer. – Michael-O Jul 28 '16 at 17:53
  • Thanks, do you know how can we do the same in a Java http client, ie. Authorize the proxy using a keytab file? – Arik Pamnani Apr 28 '23 at 19:18

3 Answers3

73

Being a once-in-a-while-contributor to curl in that area. Here is what you need to know:

curl(1) itself knows nothing about Kerberos and will not interact neither with your credential cache nor your keytab file. It will delegate all calls to a GSS-API implementation which will do the magic for you. What magic depends on the library, Heimdal and MIT Kerberos.

Based on your question, I assume that you have little knowledge about Kerberos and want simply automate API calls to a REST endpoints secured by SPNEGO.

Here is what you need to do:

  1. Have a Unix-like OS
  2. Install at least MIT Kerberos 1.11
  3. Install at least curl 7.38.0 against MIT Kerberos
  4. Verify this with curl --version mentioning GSS-API and SPNEGO and with ldd linked against your MIT Kerberos version.
  5. Create a client keytab for the service principal with ktutil or mskutil
  6. Try to obtain a TGT with that client keytab by kinit -k -t <path-to-keytab> <principal-from-keytab>
  7. Verify with klist that you have a ticket cache

Environment is now ready to go:

  1. Export KRB5CCNAME=<some-non-default-path>
  2. Export KRB5_CLIENT_KTNAME=<path-to-keytab>
  3. Invoke curl --negotiate -u : <URL>

MIT Kerberos will detect that both environment variables are set, inspect them, automatically obtain a TGT with your keytab, request a service ticket and pass to curl. You are done.

Note: this will not work with Heimdal.

Michael-O
  • 18,123
  • 6
  • 55
  • 121
  • 2
    Can you detail how exactly in step-3 do we install curl *against* MIT Kerberos? – displayName Sep 27 '19 at 15:22
  • 2
    @displayName at best, your OS package manager will do this for you. At worst, you need to pass the pass the path of `krb5-config` to the configure script of curl. That's it. – Michael-O Sep 30 '19 at 08:08
  • What is `KRB5CCNAME` supposed to be set to? – wheeler Mar 13 '21 at 07:28
  • @wheeler See https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html – Michael-O Mar 13 '21 at 19:48
  • That doesn't really answer my question. Does it need to point to an existing file or something? What, _specifically_ do I set the `KRB5CCNAME` environment variable to? – wheeler Mar 14 '21 at 19:05
  • 2
    @wheeler A late answer, but you set it with path to kerberos cache file, e.g. `export KRB5CCNAME=/tmp/krb5cc_1000`. After using kinit, the cache is created automatically, you can find it using `klist`: __Ticket cache: FILE:/tmp/krb5cc_1000__ – akoz Jul 13 '21 at 18:40
  • +9000 for this post. This is a Top 5 G.O.A.T. for me. Before I read this post, I never knew that Linux/Cygwin /usr/bin/curl with GSS-API + SPNEGO was using MIT Kerberos lib for auth. Blew my mind. There is no hint from the curl error. Only: `gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate.` Another good tip: Enable MIT Kerberos debug logs: `env KRB5_TRACE=/dev/stdout curl ...` – kevinarpe May 19 '23 at 07:26
18

  1. Check curl version

    $ curl -V - It should support the feature "GSS-Negotiate"

  2. Login using kinit

    $ kinit <user-id>

  3. Use curl

    $ curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:14000/webhdfs/v1/?op=liststatus

    "--negotiate" option enables SPNEGO

    "-u" option is required but ignored (the principle specified during kinit is used)

    "-b" & "-c" options are used to store and send http cookies.

Avinash Reddy
  • 1,153
  • 7
  • 23
  • It should not support `GSS-Negotiate` because all these version are broken prior 7.38.0. It has to support `SPNEGO`. – Michael-O Jul 27 '16 at 14:32
  • 2
    This works fine for me, and is a lot less setup/trouble than the accepted answer. curl 7.35.0, no SPNEGO, only GSS-Negotiate. – BobDoolittle Apr 25 '17 at 18:58
  • Are the cookie options necessary for a basic test of connectivity? If yes, why? – peedee Mar 06 '18 at 10:21
  • 2
    Answering myself: without cookiejar the http will redirect infinitely even though the Kerberos part was successful. – peedee Mar 07 '18 at 14:02
  • @Michael-O I, too, am using a pre-7.38 version with SPNEGO with no issue. `curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets`. `GSS-Negotiate` appears to be the ticket. – WattsInABox Jun 12 '18 at 15:33
  • I'm having the same problem but none of the answers seem to help. Would you mind having a look at https://stackoverflow.com/questions/59898525/curl-with-negotiate-kerberos-doesnt-seem-to-work maybe you can spot a mistake :-) – Michel Jung Jan 24 '20 at 14:35
0

I am using version 7.54.0. The following command works fine for me. (please remember/pay attention there are double dashes before the negotiate option).

curl --negotiate -u: '<the_full_url>'
rayryeng
  • 102,964
  • 22
  • 184
  • 193
Mubai
  • 21
  • 3