NodeJS - How to get response of URL?

Question

I have an url adress. When I go that adress in browser it shows a json file. I need to get that json file in NodeJs.

How can I do that?

Edit: This is my current code:

const got = require('got');   

var url = "https://www.instagram.com/p/CFcyO54Hc7k/?__a=1";  

got(url).json().then(result => {     
    console.log(result); 
}).catch(err => {     
    console.log(err); 
});

And This is the result I get:

Promise { <pending> }
Hint: hit control+c anytime to enter REPL.
RequestError: Unexpected token < in JSON at position 0 in "https://www.instagram.com/accounts/login/?next=/p/CFcyO54Hc7k/%3F__a%3D1"
    at Object.parseBody [as default] (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/parse-body.js:22:15)
    at /home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/index.js:157:40
    at JSON.parse (<anonymous>)
    at parseJson (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/index.js:118:35)
    at Object.parseBody [as default] (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/parse-body.js:11:48)
    at /home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/index.js:157:40
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  name: 'ParseError',
  code: undefined,
  timings: {
    start: 1601274389096,
    socket: 1601274389096,
    lookup: 1601274389106,
    connect: 1601274389117,
    secureConnect: 1601274389128,
    upload: 1601274389128,
    response: 1601274389467,
    end: 1601274389511,
    error: undefined,
    abort: undefined,
    phases: {
      wait: 0,
      dns: 10,
      tcp: 11,
      tls: 11,
      request: 0,
      firstByte: 339,
      download: 44,
      total: 415
    }
  }
}

My node version is 12.16.1

Does this answer your question? [How to download a file with Node.js (without using third-party libraries)?](https://stackoverflow.com/questions/11944932/how-to-download-a-file-with-node-js-without-using-third-party-libraries) — tomerpacific, Sep 27 '20 at 18:43
@UzaySan what do you mean it doesn't work? There are a ton of answers on that post. How many have you tried? I don't think many, coz you replied after a mere 3 minutes after that link was shared. — Sнаđошƒаӽ, Sep 27 '20 at 18:55
@Sнаđошƒаӽ Yes bunch of answers but they all suggest using http library. And When I use that method, I get response headers. not the actual json. — uzaysan, Sep 27 '20 at 19:00
@SridharChidurala No ı dont want to write it to a file. I need to extract few objects. and thats it — uzaysan, Sep 28 '20 at 05:59

score 0 · Answer 1 · answered Sep 27 '20 at 18:49

0

If you want to get it as json you can use the fetch api (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) then in the promises you can read the json as an object in your js code.

fetch('http://server/file.json').then(resp=>resp.json()).then(data=>{do something with data});

answered Sep 27 '20 at 18:49

user3252327

617
5
9

`fetch()` works in the browser. The OP asked for a solution in node.js. – jfriend00 Sep 28 '20 at 01:05

jfriend00 · Answer 2 · 2020-09-28T06:12:38.880

0

You can either parse the JSON yourself or you can get a library for making an Http request that parses the JSON automatically for you. There are lots of libraries to choose from here. My favorite is got() and it will let you parse the JSON with a simple .json() added to it so the resolved value you get back is a Javascript object.

const got = require('got');

got("https://www.instagram.com/p/CFcyO54Hc7k/?__a=1").json().then(result => {
    console.log(result);
}).catch(err => {
    console.log(err);
});

When I run the above code with your instagram URL, I get this Javascript object back:

{
  graphql: {
    shortcode_media: {
      __typename: 'GraphSidecar',
      id: '2404017227651993316',
      shortcode: 'CFcyO54Hc7k',
      dimensions: [Object],
      gating_info: null,
      fact_check_overall_rating: null,
      fact_check_information: null,
      sensitivity_friction_info: null,
      media_overlay_info: null,
      media_preview: null,
      display_url: 'https://scontent-sjc3-1.cdninstagram.com/v/t51.2885-15/e35/119980944_890957481429986_8757958615208227911_n.jpg?_nc_ht=scontent-sjc3-1.cdninstagram.com&_nc_cat=102&_nc_ohc=w7zBW_FdRNgAX9BYknl&_nc_tp=18&oh=11424efc118b55180e4d3302d75181ca&oe=5F99EB23',
      display_resources: [Array],
      is_video: false,
      tracking_token: 'eyJ2ZXJzaW9uIjo1LCJwYXlsb2FkIjp7ImlzX2FuYWx5dGljc190cmFja2VkIjp0cnVlLCJ1dWlkIjoiZGIyMzdkMzgzYWVhNDY5YmI0ZWVkZWI4YzlhZDY0YWUyNDA0MDE3MjI3NjUxOTkzMzE2In0sInNpZ25hdHVyZSI6IiJ9',
      edge_media_to_tagged_user: [Object],
      edge_media_to_caption: [Object],
      caption_is_edited: false,
      has_ranked_comments: false,
      edge_media_to_parent_comment: [Object],
      edge_media_to_hoisted_comment: [Object],
      edge_media_preview_comment: [Object],
      comments_disabled: false,
      commenting_disabled_for_viewer: false,
      taken_at_timestamp: 1600801207,
      edge_media_preview_like: [Object],
      edge_media_to_sponsor_user: [Object],
      location: null,
      viewer_has_liked: false,
      viewer_has_saved: false,
      viewer_has_saved_to_collection: false,
      viewer_in_photo_of_you: false,
      viewer_can_reshare: true,
      owner: [Object],
      is_ad: false,
      edge_web_media_to_related_media: [Object],
      edge_sidecar_to_children: [Object],
      edge_related_profiles: [Object]
    }
  }
}

If you'd rather modify your existing code, then you will probably need to use JSON.parse(). You will have to add your existing code to your question for us to help with that.

edited Sep 28 '20 at 06:12

answered Sep 28 '20 at 01:03

jfriend00

683,504
96
985
979

I don't want to write file. I need to extract few objects in it and save it to database. I guess i need to explain. This is my url: `https://www.instagram.com/p/CFcyO54Hc7k/?__a=1` I want to get json from this url. When I go to this url in browser it shows json file. But ın node js, ıt just returns html with javascript methods. I tried request library. But I guess I need to add session token header. – uzaysan Sep 28 '20 at 05:48
I also mention that, when I use request library with other url like `reddit.com/popular.json` It returns json. So I guess my problem is with instagram. But I dont know How to solve it – uzaysan Sep 28 '20 at 05:50
@UzaySan - My mistake. I left a `.json()` out of my code which I have now added. When I insert that, your instagram URL gives me a nice Javascript object. In the future, please include things like a specific URL in your question so people can test out their answers against your actual situation. – jfriend00 Sep 28 '20 at 06:07
I tried latest code. I get this error. `RequestError: Unexpected token < in JSON at position 0 in "https://www.instagram.com/accounts/login/?next=/p/CFcyO54Hc7k/%3F__a%3D1" ` I think I need to add a session token. How can I add this? Or how to solve this? Edit: I see you added a perfect json file. I cant get that. I'm using repl.it online node editor. – uzaysan Sep 28 '20 at 06:12
@UzaySan - When I plug your instagram URL into my code above, I get the Javascript object now shown in my answer. Is that not what you want? Please note that the %3F and %3D should not be in your URL. That indicates something was encoding the ? and the = and should not have been doing that. You need to real characters in the URL. – jfriend00 Sep 28 '20 at 06:15
That chars are not in my URL. This is my URL: `https://www.instagram.com/p/CFcyO54Hc7k/?__a=1` I think instagram redirects. But I dont know why. This is my full code: `const got = require('got'); var url = "https://www.instagram.com/p/CFcyO54Hc7k/?__a=1"; got(url).json().then(result => { console.log(result); }).catch(err => { console.log(err); });` – uzaysan Sep 28 '20 at 06:17
@UzaySan - And, what exactly is the result you get? What version of node.js? What version of got? – jfriend00 Sep 28 '20 at 06:24
I have edited my question. I added my current code, node version and result I get. Please take a look – uzaysan Sep 28 '20 at 06:28
@UzaySan - I don't know what to say. The same code works just fine for me in node `v14.10.0` and I'm using got 11.7.0 which is current. Are you running this in REPL or running it from a file via the command line. I'm running it from a file via the command line. I never use REPL as I've found too many differences between it and real nodejs. – jfriend00 Sep 28 '20 at 06:32
Ok. Thank you. I'm using this code in repl online editor. I didnt use it with command line. I will try it in server to see i it makes any difference. Thanks again – uzaysan Sep 28 '20 at 06:37
@UzaySan - Did it work in a regular node.js script file? – jfriend00 Sep 29 '20 at 00:09
I didnt try actually. I found a npm module that does exactly what I want. I'm using it. – uzaysan Sep 29 '20 at 15:03

score 0 · Answer 3 · edited Sep 28 '20 at 06:09

0

You can use axios in NodeJS.

axios.get('https://someapiendpoint.domain.com/v1/api')
  .then((response) => {
    console.log(response.data);
  });

edited Sep 28 '20 at 06:09

Tenten Ponce

2,436
1
13
40

answered Sep 28 '20 at 05:46

axysharma

113
1
8

I tried axios. But getting same result. `https://www.instagram.com/p/CFcyO54Hc7k/?__a=1` doesnt return a json. But in browser it shows a json. – uzaysan Sep 28 '20 at 05:49
You cannot access Instagram APIs directly as Instagram Legacy API permission was disabled on June 29, 2020 in favor of new Instagram graph API. You can try the wrapper https://www.npmjs.com/package/instagram-graph-sdk – axysharma Sep 28 '20 at 06:18

NodeJS - How to get response of URL?

3 Answers3