Selenium get text in paragraph after div

Question

I have a page that looks like this:

When we look at the HTML code:

So it first gives the title of the paragraph in a div and then under it it will have the paragraph. So ideally I want to do something like driver.find_element_by_link_text('Objectives of the Course') and then say "get next element" (i.e. the paragraph under it).

How can this be done using selenium or any other library?

Can you share the url and does solution have to be selenium? — QHarr, Sep 06 '19 at 07:49

score 3 · Answer 1 · answered Sep 06 '19 at 07:18

3

You may use XPATH or CSS Selector with find_element_by_css_selector method

in this HTML:

<div class="title"> title </div>
<p> content </p>

you can select next sibling with this:

div.title + p {
  color: red;
}

so in your case, driver.find_element_by_css_selector('div.FieldsetBaslik+p') will work

check this link: https://developer.mozilla.org/en-US/docs/Web/CSS/Adjacent_sibling_combinator

answered Sep 06 '19 at 07:18

cereme

280
1
18

Dammn this was an easy solution!, awesome how you can just add `+p` – CodeNoob Sep 06 '19 at 08:06

undetected Selenium · Accepted Answer · 2019-09-06T07:16:36.333

1

To extract the text within <p> tag which is just under the title Objectives of the Course you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using xpath and get_attribute():

print(WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='FieldsetBaslik' and contains(., 'Objectives of the Course')]//following::p[1]"))).get_attribute("innerHTML"))

Using xpath and text:

print(WebDriverWait(browser, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='FieldsetBaslik' and contains(., 'Objectives of the Course')]//following::p[1]"))).text)

Outro

As per the documentation:

get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

edited Sep 06 '19 at 07:16

answered Sep 06 '19 at 07:11

undetected Selenium

183,867
41
278
352

1

Although the other answer is way easier I think this one is more suited for more complex tasks, for example the other answer won't work when there would be `div`s with the same name (`FieldsetBaslik`) but a different text – CodeNoob Sep 06 '19 at 08:49
Looks like the xpath you're suggesting is the answer to the question. IMHO, the WebDriverWait stuff is unrelated to the question. For example, if that HTML is statically loaded with the page then there's no reason to wait for it. – Arnon Axelrod Sep 07 '19 at 05:05
@ArnonAxelrod You are correct. Did you observe the `== $0` present in the html? – undetected Selenium Sep 07 '19 at 18:39

Selenium get text in paragraph after div

2 Answers2

Outro