django and python requests - getting a 403 on a post request

Question

I am using requests to log into my Django site for testing (and yes, I know about the Django TestClient, but I need plain http here). I can log in and, as long as I do get requests, everything is OK.

When I try to use post instead, I get a 403 from the csrf middleware. I've worked around that for now by using a @crsf_exempt on my view, but would prefer a longer term solution.

This is my code:

with requests.Session() as ses:

    try:

        data = {
            'username': self.username,
            'password': self.password,
        }

        ses.get(login_url)
        try:
            csrftoken = ses.cookies["csrftoken"]
        except Exception, e:
            raise
        data.update(csrfmiddlewaretoken=csrftoken)

        _login_response = ses.post(login_url, data=data)

        logger.info("ses.cookies:%s" % (ses.cookies))

        assert 200 <= _login_response.status_code < 300, "_login_response.status_code:%s" % (_login_response.status_code)

        response = ses.post(
            full_url,
            data=data,
            )

        return self._process_response(response)

The login works fine, and I can see the csrf token here.

 INFO:tests.helper_fetch:ses.cookies:<RequestsCookieJar[<Cookie csrftoken=TmM97gnNHs4YCgQPzfNztrAWY3KcysAg for localhost.local/>, <Cookie sessionid=kj6wfmta

However, the middleware sees cookies as empty.

INFO:django.middleware.csrf:request.COOKIES:{}

I've added the logging code to it:

def process_view(self, request, callback, callback_args, callback_kwargs):

    if getattr(request, 'csrf_processing_done', False):
        return None

    try:
        csrf_token = _sanitize_token(
            request.COOKIES[settings.CSRF_COOKIE_NAME])
        # Use same token next time
        request.META['CSRF_COOKIE'] = csrf_token
    except KeyError:
        # import pdb
        # pdb.set_trace()
        import logging
        logger = logging.getLogger(__name__)
        logger.info("request.COOKIES:%s" % (request.COOKIES))

Am I missing something with way I call request's session.post? I tried adding cookie to it, made no difference. But I can totally see why crsf middleware is bugging out. I thought the cookies were part of the session, so why are they missing in my second post?

            response = ses.post(
                self.res.full_url,
                data=data,
                cookies=ses.cookies,
                )

This variation, inspired by How to send cookies in a post request with the Python Requests library?, also did not result in anything being passed to csrf middleware:

            response = ses.post(
                self.res.full_url,
                data=data,
                cookies=dict(csrftoken=csrftoken),
                )

score 6 · Accepted Answer · answered Apr 13 '16 at 02:28

6

For subsequent requests after the login, try supplying it as header X-CSRFToken instead.

The following worked for me:

with requests.Session() as sesssion:
    response = session.get(login_url)
    response.raise_for_status()  # raises HTTPError if: 400 <= status_code < 600

    csrf = session.cookies['csrftoken']
    data = {
        'username': self.username,
        'password': self.password,
        'csrfmiddlewaretoken': csrf
    }


    response = session.post(login_url, data=data)
    response.raise_for_status()

    headers = {'X-CSRFToken': csrf, 'Referer': url}
    response = session.post('another_url', data={}, headers=headers)
    response.raise_for_status()

    return response  # At this point we probably made it

Docs reference: https://docs.djangoproject.com/en/dev/ref/csrf/#csrf-ajax

answered Apr 13 '16 at 02:28

fips

4,319
5
26
42

Thanks for your help. Got it working OK, I had to do some changes, which took a long time to figure out, because the csrf token **changes** after `session.post(login_url, data=data)`. And that changed token is what needs to go into `X-CSRFToken`. But it looks pretty good and X-CSRFToken matches what I do for JS Ajax posts. What I'll do is to write up my answer at the end of the week - with the cleaned-up code - and accept your answer for pointing me in the right direction. Keeping it open for now because that was not a cheap bounty and someone may have another great answer. Take care. – JL Peyret Apr 14 '16 at 18:02
If I haven't accepted it by Monday, gimme a buzz. I *assume* that SO would still award you the 100 points if I overshot the bounty duration, but don't want to have you miss out on the bounty if that is not the case. – JL Peyret Apr 14 '16 at 18:04
Don't worry about the bounty! I just answered this question cos I happened to have solved it a while ago. You are right the csrf token changes after post to login but if you are using sessions it's handled transparently for you. For a complete solution feel free to check out my library on github (that does access control checks) by exactly the way you are doing it: https://github.com/stphivos/fnval/blob/master/fnval/net.py#L41 – fips Apr 14 '16 at 18:20
the bounty is non-refundable, so I've already paid for it. might as well award it. yeah, maybe sessions was supposed to handle it, but it didn't. or at least didn't seem to. plus, i have seen people remark elsewhere on SO that sometimes `requests`'s sessions can let cookies fall through the cracks. funny how sometimes writing 30 lines of code (that work) can take you days ;-) – JL Peyret Apr 14 '16 at 19:16
cool idea, `fnval`. I am doing something probably somewhat related to it. django testcases run against the TestClient, fire off an URL with some data and then you can parse responses. but they *only* work against the local test client, even though really most of the work is defining the test parameters and the unittest assertions to run against the response. my code reuses all that, but fires it off using `requests` against an arbitrary hostname+port. In my case, my itch to scratch is that I need to test a VM-based appliance. I log in @ every url call, so sessions aren't a big deal. – JL Peyret Apr 14 '16 at 19:23
Cool I'm a big fun of tests. Py.test has a plugin to do distributed tests from remote hosts but that's the opposite of what you're doing if I understood correctly. Feel free to share a link to your library when you're done! – fips Apr 14 '16 at 23:43

score 2 · Answer 2 · answered Apr 13 '16 at 03:13

2

You could also try to use this decorator on your view, instead of the csrf_exempt. I tried to reproduce your issue, and this worked as well for me.

from django.views.decorators.csrf import ensure_csrf_cookie`

@ensure_csrf_cookie
def your_login_view(request):
    # your view code

answered Apr 13 '16 at 03:13

Vladir Parrado Cruz

2,301
21
27

`ensure_csrf_cookie` job is to make sure that a csrf token goes out, even if there is no form on the page. I don't have that problem - my write-up specifically says that I can see the csrf_token, I just don't know how to pass it back in to the actual target (not the login) POST. – JL Peyret Apr 14 '16 at 17:55
Ok, thanks for your comment, the most important thing here is that you find a solution that works for you ;), and that we, who want to help to reach that, have that in mind, and it was my goal ;). Thanks again, and good luck! – Vladir Parrado Cruz Apr 14 '16 at 18:19

django and python requests - getting a 403 on a post request

2 Answers2