Trying to GET from api after posting API issues

Question

Hi developers I am back again with a question,

I am trying to get some data from this website https://www.iamsterdam.com/nl/uit-in-amsterdam/uit/agenda. First did I crawl the website but when doing that it came to mind that they have a api and that will be a lot faster. So I tried to get the data from the api I tried this:

get-website.js:

var webPage = require('webpage');
var page = webPage.create();
var settings = {
  operation: "POST",
  encoding: "utf8",
  headers: {
    "Content-Type": "application/json"
  },
  data: JSON.stringify({
    DateFilter: 03112016,
    PageId: "3418a37d-b907-4c80-9d67-9fec68d96568",
    Take: 2,
    Skip: 12,
    ViewMode: 1
  })
};

page.open('https://www.iamsterdam.com/api/AgendaApi/', settings, function(status) {
  console.log(page.content);
  phantom.exit();
});

get-website.php

$phantom_script= 'get-website.js'; 

$response =  exec ('phantomjs ' . $phantom_script);

echo  $response;

But what I get back doesn't look good:

Message":"An error has occurred.","ExceptionMessage":"Page could not be found","ExceptionType":"System.ApplicationException","StackTrace":" at Axendo.SC.AM.Iamsterdam.Controllers.Api.AgendaApiController.GetResultsInternal(RequestModel requestModel)\r\n at lambda_method(Closure , Object , Object[] )\r\n
etc.

Here is a picture of firebug:

I hope someone can help me.

Your PageId seems to be wrong, just used postman to try the same, got the same response from the page. — Ann-Sophie Angermüller, Nov 03 '16 at 08:58
@LuciaAngermüller I got the PageId by looking add what the website was sending used firebug for it so should be good. — N. Smeding, Nov 03 '16 at 09:14
on of your data is obviously wrong, and because he says page not found, i thought that the PageID might be the problem, but without further information it's like guessing into the dark — Ann-Sophie Angermüller, Nov 03 '16 at 09:15
@AnnSophieAngermüller I have add a firebug image you can see that the browser send the same data as i am sending. — N. Smeding, Nov 03 '16 at 10:58
Who or how do you send the request? Maybe some other script is executed before? — Ann-Sophie Angermüller, Nov 03 '16 at 11:12

score 1 · Accepted Answer · answered Nov 03 '16 at 12:32

Interesting question. I was a bit surprised that the site would honor AJAX-request in a browser and even in cURL, but not in PhantomJS. In such cases you have to study and replicate request very carefully, because one of little details probably greatly affects the server's response.

Turned out, it was a cookie and form content-type that had to be set accordingly.

var webPage = require('webpage');
var page = webPage.create();

// courtesy of http://stackoverflow.com/a/1714899/2715393
var serialize = function(obj) {
  var str = [];
  for(var p in obj)
    if (obj.hasOwnProperty(p)) {
      str.push(encodeURIComponent(p) + "=" + encodeURIComponent(obj[p]));
    }
  return str.join("&");
}

var settings = {
    operation: "POST",
    encoding: "utf8",
    headers: {
        "accept-encoding" : "identity", // https://github.com/ariya/phantomjs/issues/10930#issuecomment-81541618
        "x-requested-with" : "XMLHttpRequest",
        "accept-language" : "en;q=0.8,en-US;q=0.6",
        "authority" : "www.iamsterdam.com",
        "accept":"application/json, text/javascript, */*; q=0.01",
        "content-type" : "application/x-www-form-urlencoded; charset=UTF-8",
        "cookie" : "website#lang=nl"        
    },
    data: serialize({
        Genre: '',
        DateFilter: '03112016',
        DayPart: '',
        SearchTerm: '', 
        Neighbourhoud: '',
        DayRange: '',
        ViewMode: 1, 
        LastMinuteTickets : '',
        PageId: '3418a37d-b907-4c80-9d67-9fec68d96568',
        Skip: 0,
        Take: 12
    }) 
};

page.open('https://www.iamsterdam.com/api/AgendaApi/', settings, function(status) {
    console.log(page.content);
    phantom.exit();
});

Thx man you are a legend all I need to do now is encode it right? #mvp — N. Smeding, Nov 03 '16 at 13:23
I mean with decode instead of encode mb, because I get the response in plaintext and not as code so I except that the response is json right? — N. Smeding, Nov 04 '16 at 08:20
No, the response is actually prerendered HTML, so you will have to parse it somehow. Actually you're not gaining much by connecting to that API endpoint directly. — Vaviloff, Nov 04 '16 at 11:38

Trying to GET from api after posting API issues

1 Answers1

Linked