How to get python package `awswranger` to accept a custom `endpoint_url`

Question

I'm attempting to use the python package awswrangler to access a non-AWS S3 service.

The AWS Data Wranger docs state that you need to create a boto3.Session() object.

The problem is that the boto3.client() supports setting the endpoint_url, but boto3.Session() does not (docs here).

In my previous uses of boto3 I've always used the client for this reason.

Is there a way to create a boto3.Session() with a custom endpoint_url or otherwise configure awswrangler to accept the custom endpoint?

score 5 · Accepted Answer · answered Mar 25 '21 at 02:27

5

I finally found the configuration for awswrangler:

import awswrangler as wr

wr.config.s3_endpoint_url = 'https://custom.endpoint'

answered Mar 25 '21 at 02:27

David Parks

score 3 · Answer 2 · answered Dec 02 '21 at 22:08

In that case, simply set WR_S3_ENDPOINT_URL to your custom endpoint, and the configuration will reflect that when you import the library.

score 2 · Answer 3 · edited Mar 25 '21 at 00:49

2

Once you create your session, you can use client as well. For example:

import boto3

session = boto3.Session()
s3 = session.client('s3', endpoint_url='<custom-endpoint>')

edited Mar 25 '21 at 00:49

jellycsc

answered Mar 25 '21 at 00:45

Marcin

2

This didn't get the `endpoint_url` to pass through to `awswrangler` when `session` is handed to `awswrangler`. – David Parks Mar 25 '21 at 02:26
1

@DavidParks I think you should re-phrase your question, that you are asking only about `awswrangler as wr`. I answered to your original question regarding `Is there a way to create a boto3.Session() with a custom endpoint_url?` and the use of its client, not how do I modify `awswrangler` default endpoint. – Marcin Mar 25 '21 at 02:29
2

I agree, I was ambiguous about that, my fault. The `awswrangler` docs seemed very explicit that I should handle those matters in the session. I will update the wording. I do very much appreciate your answer and help. It got me past what I was stuck on and looking in the right area for the answer. – David Parks Mar 25 '21 at 02:32

3 Answers3