How can I get only visible text from some HTML node in Python

Question

How can I get only visible text from some HTML node in Python?

Suppose that I have a node like this:

<span>
   <style>.vAnH{display:none}.vsP6{display:inline}</style>
   <span class="vAnH">34</span>
   <span />
   <span style="display: inline">111</span>
   <span style="display:none">120</span>
   <span class="vAnH">120</span>
   <div style="display:none">120</div>
   <span class="78">.</span>
   <span class="vAnH">100</span>
   <div style="display:none">100</div>
   161
   <span style="display: inline">.</span>
   <span class="174">126</span>
   <span class="vAnH">159</span>
   <div style="display:none">159</div>
   <span />
   <span class="vsP6">.</span>
   <span style="display:none">5</span>
   <span class="vAnH">5</span>
   <div style="display:none">5</div>
   <span style="display:none">73</span>
   <span class="vAnH">73</span>
   <div style="display:none">73</div>
   <span class="221">98</span>
   <span style="display:none">194</span>
   <div style="display:none">194</div>
</span>

Is there any third-party libraries to do it or should I parse it manually?

Answer 1

There are multiple ways to make a node visible/hidden for the end user in the browser. BeautifulSoup is an HTML Parser, it doesn't know if an element would be shown or not. Though, there was an attempt here:

• BeautifulSoup Grab Visible Webpage Text

It would not work if, for example, an element is hidden by a CSS rule, but might work for your use case.

The easiest option would be to switch to selenium. .text here returns only visible text of an element:

from selenium import webdriver

driver = webdriver.Firefox() 
driver.get('http://domain.com')

element = driver.find_element_by_id('id_of_an_element')
print(element.text)

Answer 2

If you don't want to go the Selenium way, you can get something with BeautifulSoup:

from bs4 import BeautifulSoup

def is_visible_span_or_div(tag, is_parent=False):
    """ This function checks if the element is a span or a div,
    and if it is visible. If so, it recursively checks all the parents
    and returns False is one of them is hidden """

    # loads the style attribute of the element
    style = tag.attrs.get('style', False)

    # checks if element is div or span, if it's not a parent
    if not is_parent and tag.name not in ('div', 'span'):
        return False

    # checks if the element is hidden
    if style and ('hidden' in style or 'display: none' in style):
        return False

    # makes a recursive call to check the parent as well
    parent = tag.parent
    if parent and not is_visible_span_or_div(parent, is_parent=True):
        return False

    # neither the element nor its parent(s) are hidden, so return True
    return True

html = """
    <span style="display: none;">I am not visible</span>
    <span style="display: inline">I am visible</span>
    <div style="display: none;">
        <span>I am a visible span inside a hidden div</span>
    </div>
"""

soup = BeautifulSoup(html)

visible_elements = soup.find_all(is_visible_span_or_div)

print(visible_elements)

Keep in mind that it's not going to exactly reflect the way a browser would display or hide the elements, though, because other factors could decide for the visibility of an element (such as width, height, opacity, absolute positioning outside the window...).

Despite of that, this script is quite reliable because it recursively checks for all the element's parents as well and returns False as soon as it finds a hidden parent.

The only problem I see with this function is that it has quite an overhead, because it has to check for all the parents for every element, even if those elements happen to be just aside in the DOM tree. It could be easily optimised for that, but perhaps at the cost of readability.

Answer 3

You'll need to write a custom filter function. A working example:

from bs4 import BeautifulSoup
import re

data = '''<span>
   <style>.vAnH{display:none}.vsP6{display:inline}</style>
   <span class="vAnH">34</span>
   <span />
   <span style="display: inline">111</span>
   <span style="display:none">120</span>
   <span class="vAnH">120</span>
   <div style="display:none">120</div>
   <span class="78">.</span>
   <span class="vAnH">100</span>
   <div style="display:none">100</div>
   161
   <span style="display: inline">.</span>
   <span class="174">126</span>
   <span class="vAnH">159</span>
   <div style="display:none">159</div>
   <span />
   <span class="vsP6">.</span>
   <span style="display:none">5</span>
   <span class="vAnH">5</span>
   <div style="display:none">5</div>
   <span style="display:none">73</span>
   <span class="vAnH">73</span>
   <div style="display:none">73</div>
   <span class="221">98</span>
   <span style="display:none">194</span>
   <div style="display:none">194</div>
</span>'''

soup = BeautifulSoup(data)
no_disp = re.search(r'\.(.+?){display:none}', soup.style.string).group(1)

def find_visible(tag):
    return (not tag.name == 'style') and (not no_disp in tag.get('class', '')) and (not 'display:none' in tag.get('style', ''))

for tag in soup.find_all(find_visible, text=True):
    print tag.string