1

I'm scraping a website, which for some reason takes 15 seconds to load, but the elements I need get loaded in the first 5 seconds.

The question is, can I stop loading a page, and move on with the code when the code is already present?

Current code:

driver.Navigate().GoToUrl(url);

new WebDriverWait(driver, TimeSpan.FromSeconds(20))
    .Until(ExpectedConditions.ElementIsVisible(By.XPath(xpath)));

Current behaviour:

  1. I call driver.Navigate().GoToUrl(url);
  2. The driver starts loading the page
  3. The element is present within approx. 5 seconds (that's when I want to move on to the next statement)
  4. The driver waits for the whole 15 seconds to fully load the page
  5. Then the driver finally moves to the WebDriverWait function

The problem:

All of the solutions in similar questions are called after the page fully loaded, making the answer useless.

Can I do something about it? Thank you.

Jakub Loksa
  • 537
  • 1
  • 14
  • 32

3 Answers3

4

Method .Navigate().GoToUrl is created such a way that next line is executed only when page gets fully loaded. so yes, it doesn't matter what you are writing in code, after navigate method, it will never work until the page gets fully loaded. As a workaround you can opt for timeout option, with this it will throw a exception as page remains to load so we should catch that and execute our next code.

//This is java code, so please pick only the logic from here:
// Set the page load timeout to 10 seconds. 

driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS);  

try {  
  driver.Navigate().GoToUrl(url);  
} catch (TimeoutException e) {  
  // Ignore the exception.  
}
Keshav
  • 415
  • 2
  • 12
1

I've got the same issue and googled a lot but found no workable solutions. The timeout solution is not applicable to my situation. At last it seems the only way that really works is to use a global hook.

For more information about hook, please refer to this excellent post: Global keyboard capture in C# application

So, in the hook handling code taken from above link, add something like this:

    private IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam)
    {
        if (nCode >= 0 && wParam == (IntPtr)WM_KEYDOWN)
        {
            int vkCode = Marshal.ReadInt32(lParam);
            if (vkCode == 121)  //F10 key
            {
                try
                {
                    UnhookWindowsHookEx(_hookID);//avoid ESC key to be captured
                    SetForegroundWindow(_handle);
                    SendKeys.Send("{ESC}");
                    _hookID = SetHook(_proc);

                }
                catch (Exception ex)
                {
                }
            }
        }

        return CallNextHookEx(_hookID, nCode, wParam, lParam);
    }

related code below:

    using System.Windows.Automation; //you need to reference UIAutomationClient and UIAutomationTypes

    [DllImport("user32.dll")]
    [return: MarshalAs(UnmanagedType.Bool)]
    static extern bool SetForegroundWindow(IntPtr hWnd);

    private IntPtr _handle;

    //I forget where I got the code of the function below, probably also from stackoverflow. Thanks to the original author!
    private IntPtr getIntPtrHandle(IWebDriver driver, int timeoutSeconds = 30)
    {
        var end = DateTime.Now.AddSeconds(timeoutSeconds);
        while (DateTime.Now < end)
        {
            // Searching by AutomationElement is a bit faster (can filter by children only)
            var ele = AutomationElement.RootElement;
            foreach (AutomationElement child in ele.FindAll(TreeScope.Children, Condition.TrueCondition))
            {
                if (!child.Current.Name.Contains(driver.Title)) continue;
                return new IntPtr(child.Current.NativeWindowHandle); ;
            }
        }
        return IntPtr.Zero;
    }

also put _handle = getIntPtrHandle(webdriver); somewhere BEFORE your webdriver.Navigate().GoToUrl() statement.

I tested the above. After excecuting GoToUrl, press F10 somewhere (sometimes twice) and the page stops loading.

Community
  • 1
  • 1
Silent Sojourner
  • 146
  • 4
0

Yes it's possible by setting the pageLoadStrategy capability to none. Then wait for an element to be present to stop the loading: more informations :

https://www.selenium.dev/documentation/webdriver/capabilities/shared/

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;

namespace pageLoadStrategy {
  class pageLoadStrategy {
    public static void Main(string[] args) {
      var chromeOptions = new ChromeOptions();
      chromeOptions.PageLoadStrategy = PageLoadStrategy.None;
      IWebDriver driver = new ChromeDriver(chromeOptions);
      try {
        driver.Navigate().GoToUrl("https://example.com");
      } finally {
        driver.Quit();
      }
    }
  }
}
user18748309
  • 1