Fetch the div element having an attribute with a name

Question

<div class="resultbox_textbox" style="position:relative">
  <div class="resultbox_title font22 fw500 color111 complist_title" style="min-height:32px;position:relative" title="Nakshatra Hospital in D N Nagar Andheri West, mumbai">
    <a class="resultbox_title_anchor line_clamp_1 font22 fw500 color111" href="/Mumbai/Nakshatra-Hospital-Andheri-West/022PXX22-XX22-220812130845-C7T1_BZDET?ncatid=10892680&amp;area=&amp;search=%20General%20Physician%20Doctors%20in%20Mumbai&amp;mncatname=General%20Physician%20Doctors" target="_self" title="Nakshatra Hospital in D N Nagar Andheri West, mumbai">Nakshatra Hospital
      <!-- -->
      <span class="title_anchor_tag"></span>
    </a>
    <span class="jdicon results_fav_icon" style="position:absolute;right:0;top:0" title="Add to Favorites"></span>
  </div>
  <div class="font12 colorCCC fadeIn animated" style="position:absolute;right:9px;top:34px">P</div>
  <div class="resultbox_overall mt-10" style="min-height:24px" title="Ratings for Nakshatra Hospital in D N Nagar Andheri West, mumbai">
    <div class="resultbox_totalrate mr-6 font14 fw700 colorFFF" style="min-height:24px">4.5</div>
    <div class="resultbox_starrate">
      <svg height="100%" viewbox="0 0 1000 200" width="100%">
        <polygon fill="#FFE372" id="star9927" points="100,10 40,198 190,78 10,78 160,198"></polygon>
        <defs>
          <clippath id="stars9927">
            <use xlink:href="#star9927"></use>
            <use x="20%" xlink:href="#star9927"></use>
            <use x="40%" xlink:href="#star9927"></use>
            <use x="60%" xlink:href="#star9927"></use>
            <use x="80%" xlink:href="#star9927"></use>
          </clippath>
        </defs>
        <rect clip-path="url(#stars9927)" height="100%" style="fill:rgb(215, 215, 215);stroke:red;stroke-width:1;height:100%;width:100%" width="90%"></rect>
        <rect clip-path="url(#stars9927)" height="100%" style="fill:rgb(255, 110, 0);height:100%" width="90%"></rect>
      </svg>
    </div>
    <div class="resultbox_countrate ml-12 mr-12 font14 fw400 color777" style="min-height:22px">6
      <!-- --> Ratings
    </div>
  </div>
  <div class="resultbox_address mt-10" style="min-height:24px">
    <div class="font15 fw400 color111" style="min-height:23px">D N Nagar Andheri West</div>
  </div>
  <div class="resultbox_amenities mt-10" style="min-height:27px">
    <span class="amenities_tabs font12 fw500 color777">Cancer Pain Management</span>
    <span class="amenities_tabs font12 fw500 color777">Pulmonology</span>
    <span class="amenities_tabs font12 fw500 color777">Contraception Advice</span>
    <span class="amenities_tabs font12 fw500 color777">Diabetology</span>
  </div>
  <div class="callmn_0 font18 fw500 color111 mt-10" data-did="022PXX22.XX22.220812130845.C7T1" style="min-height:28px">
    <div>
      <span class="result_call_black jdicon mr-12"></span>
      <span class="cp" id="lessNum_0">
        <a class="anchor color111" href="tel:07947176733">07947176733</a>
      </span>
    </div>
  </div>
  <div class="resultbox_comment font14 fw400 color777" style="min-height:64px">
    <div class="reviewcomment_icon mt-15"></div>
    <span class="comment_text mt-10">
      <q class="comment_text_quote"></q>During the treatment, she makes sure that the patient is comfortable, she never rushes into the treatment, gives enough time to patients to ensure that patients are comfo... <span class="font14 anchor color007 pl-4">more</span>
    </span>
  </div>
  <div class="resultbox_btn_wpr" style="min-height:45px">
    <div class="resultbox_btnbox mt-10">
      <div class="dn greenfill_animate callbutton font15 fw500 colorFFF mr-15" id="listing_call_button" style="min-height:35px">
        <div class="button_flare"></div>
        <span class="callcontent">
          <span class="whitecall_icon jdicon mr-10"></span>Show Number </span>
        <div class="jsx-d8f990f1402ff3b8 ctatooltip ctatooltip_header ctatooltip_bottom callnow">
          <div class="jsx-d8f990f1402ff3b8 ctatooltip_textbox_1">
            <div class="jsx-d8f990f1402ff3b8 font13 fw400 colorFFF mt-5 nowrap">Click here to see the number</div>
            <div class="jsx-d8f990f1402ff3b8 ctatooltip_btnbox ml-20">
              <button class="jsx-d8f990f1402ff3b8 ctatooltip_buttton filled font14 fw400 color111">OK</button>
            </div>
          </div>
        </div>
      </div>
      <button class="blue_whitefill_animate onlinebtn font15 fw500 mr-15" style="min-height:35px">Send Enquiry</button>
    </div>
    <div class="resultbox_btnbox mt-10"></div>
  </div>
</div>

The above is one of the div's in the list of divs. One of the div above has the attribute data-did . I have the handle to above div, but how can I fetch this div having an attribute with name data-did

Here is the current code I have written to parse the contents of a URL:

driver = webdriver.Chrome()   
driver.get("https://www.justdial.com/Mumbai/General-Physician-Doctors/nct-10892680") 
html = driver.page_source    
soup = BeautifulSoup(html, "html.parser")  
ds= soup.find_all("div", class_="resultbox_textbox")  
for d in ds:
     d. # do something on d to fetch the div with attribute data-did

When you post HTML in the future, take a minute to use a beautifier. That was a massive amount of text all in a single line making it unreadable. — JeffC, Feb 25 '23 at 05:58

score 1 · Answer 1 · answered Feb 24 '23 at 15:33

1

Use any of the below relative XPath expressions:

//div[@data-did]

or:

//div/@data-did

or:

//*/@data-did

answered Feb 24 '23 at 15:33

Shawn

4,064
2
11
23

Don't use the last one if you know the desired element is a DIV. You might end up grabbing the wrong element. – JeffC Feb 25 '23 at 06:00

KunduK · Accepted Answer · 2023-02-24T17:22:20.837

1

If you would like to identify list of div elements with attribute name data-did you use any of the locator strategies.

CSS SELECTOR:

driver.find_elements(By.CSS_SELECTOR,"div[data-did]")

XPATH:

driver.find_elements(By.XPATH,"//div[@data-did]")

Ideally to handle dynamic element use webdriverwait()

WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div[data-did]")))

OR

WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.XPATH,"//div[@data-did]")))

you need to import below library

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

Update:

If you using soup, then use this below css selector.

driver = webdriver.Chrome()   
driver.get("url") 
html = driver.page_source    
soup = BeautifulSoup(html, "html.parser") 
ds= soup.select("div[data-did]")  
for d in ds:
    print(d.text)
    print(d['data-did'])

edited Feb 24 '23 at 17:22

answered Feb 24 '23 at 16:13

KunduK

32,888
5
17
41

I have updated the OP with some code that I already have. I am not sure how could I use your method using the variable `d` in the code. So `d` = div posted above – juan Feb 24 '23 at 17:15
@juan, Check my updated answer, Let me know how that goes? – KunduK Feb 24 '23 at 17:22
This gives me an error. Try replacing `url` = `https://www.justdial.com/Mumbai/General-Physician-Doctors/nct-10892680` This is the exact URL I am using and trying to fetch the name, phone number and address – juan Feb 24 '23 at 19:12
I was able to fetch the did elements using this approach. I added the wait logic too. But sometimes I am getting 9 phone numbers other times 10. What could be the reason for this? – juan Feb 26 '23 at 08:29

score 0 · Answer 3 · answered Feb 24 '23 at 20:23

To locate the <div> element having the attribute data-did you can use either of the following locator strategies:

Using CSS_SELECTOR:

element = driver.find_element(By.CSS_SELECTOR, "div[data-did]")

Using XPATH:

element = driver.find_element(By.XPATH, "//div[@data-did]")

To locate the element ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:

Using CSS_SELECTOR:

element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[data-did]")))

Using XPATH:

element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@data-did]")))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Fetch the div element having an attribute with a name

3 Answers3