Github scraping for all javascript files

Question

I need to extract all available javascript files over all the projects available on github using python.I looked for an API in github and I found this : https://developer.github.com/v3/ I don't understand what kind of requests do I have to send and how do I compose the URL.I would prefer not depending on another 3rd party API if possible. Please guide me in the right direction,Any help would be appreciated!!

That's going to take weeks https://developer.github.com/v3/#rate-limiting — Vinnie James, Jan 20 '17 at 17:32
I could use different auths...Even if it takes time I have to do it...Is there a way around? And I dont understand how do i compose the URL...I am not familiar with curl — Sharang Gupta, Jan 20 '17 at 17:36
Are you looking to do this for a single organization/user; or all users on GitHub? — Vinnie James, Jan 20 '17 at 20:17
Voting to close as *too broad* because `Please guide me` is not a question. — bahrep, Jan 23 '17 at 17:37

score 1 · Accepted Answer · edited May 23 '17 at 12:33

1

To gather the files you can use this in your python script:

import os 
os.system("curl -o https://github.com/file.js")

Replace the URL with the individual file name, or in your case a variable in your loop to grab all files from a repo. You will need to repeat this for each org/user/repo/etc

Download remote files using curl

Running shell commands from python

edited May 23 '17 at 12:33

Community

1
1

answered Jan 20 '17 at 21:40

Vinnie James

5,763
6
43
52

Thanks!!!!I'll try it out....Sorry for being unable to upvote,my reputation is quite low... – Sharang Gupta Jan 20 '17 at 22:20

Github scraping for all javascript files

1 Answers1