Selenium is not loading TikTok pages

Question

I'm implementing a TikTok crawler using selenium and scrapy

start_urls = ['https://www.tiktok.com/trending']
....
def parse(self, response):
    options = webdriver.ChromeOptions()
    from fake_useragent import UserAgent
    ua = UserAgent()
    user_agent = ua.random
    options.add_argument(f'user-agent={user_agent}')
    options.add_argument('window-size=800x841')
    driver = webdriver.Chrome(chrome_options=options)
    driver.get(response.url)

The crawler open Chrome but it does not load videos. Image loading

The same problem happens also using Firefox No loading page using Firefox

The same problem using a simple script using Selenium

from selenium import webdriver
import time


driver = webdriver.Firefox()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()

driver = webdriver.Chrome()
driver.get("https://www.tiktok.com/trending")
time.sleep(10)
driver.close()

The problem persists using a simple python script using Selenium — user12512567, Dec 11 '19 at 23:13
all of you saying you found it working, can you provide a link to the code you used? — wesley franks, Apr 08 '20 at 16:20

score 3 · Answer 1 · answered Mar 21 '20 at 10:29

3

Did u try to navigate further within the selenium browser window? If an error 404 appears on following sites, I have a solution that worked for me:

I simply changed my User-Agent to "Naverbot" which is "allowed" by the robots.txt file from Tik Tok

(Robots.txt)

After changing that all sites and videos loaded properly.

Other user-agents that are listed under the "allow" segment should work too, if you want to add a rotation.

answered Mar 21 '20 at 10:29

zebo

141
10

can you show with code please? been having trouble getting fake-useragent to load in VS code – wesley franks Apr 08 '20 at 15:41
2

@wesleyfranks check out this post [here](https://stackoverflow.com/questions/29916054/change-user-agent-for-selenium-driver) on how to set user-agents in Selenium. For me changing it to "Googlebot" or "Naverbot" does the job! – zebo Apr 08 '20 at 21:30

score 0 · Answer 2 · answered Dec 23 '19 at 14:09

0

You can use Windows IE. Instead of chrome or firefox

Videos will load in IE but IE's Layout of showing feed is somehow different from chrome and firefox.

Reasons, why your page, is not loading.

Few advance web apps check your browser history, profile data and cached to check the authentication of the user. One other thing you can do is run your default profile within your selenium It would be helpfull.

answered Dec 23 '19 at 14:09

Zeeshan Ahmad

39
1
8

Can't use IE in ubuntu. – wesley franks Apr 08 '20 at 15:41
There are only two things working right now. 1- Selenium with an app called "Multilogin". 2 - The drawrowfly's GitHub repo. – Zeeshan Ahmad Apr 13 '20 at 07:25
@ZeeshanAhmad does the page open as mobile browser only? – Marcelo Gazzola Jan 10 '21 at 14:26

Selenium is not loading TikTok pages

2 Answers2