how to extract the value fom script using python regex in webscraping?

Question

I am trying to learn how to use regex to extract values using python. here is the script,so how to get the salesprice,seller_name and skuId

<script>
define('app/pc', ['//laz-g-cdn.alicdn.com/lzdfe/pdp-platform/0.1.8/pc.js'], function(app) {
try {
    app.run( {
        "data": {
            "root": {
                "fields": {"skuInfos": {
                        "0": {
                            "categoryId":"8711", "dataLayer": {
                                "pdt_category":["Mother & Baby", "Feeding", "Milk Formula", "Follow On (6 - 12 mnths)"], "pagetype":"pdp", "pdt_discount":"-8%", "pdt_photo":"//laz-img-sg.alicdn.com/original/6bdf9b4b759b97f57b438a605f0e37e7.jpg", "v_voya":1, "brand_name":"Dumex", "brand_id":"30360", "pdt_sku":153105871, "core": {
                                    "country": "SG", "layoutType": "desktop", "language": "en", "currencyCode": "SGD"
                                }
                                , "seller_name":"Dumex", "pdt_simplesku":191142180, "pdt_name":"Dumex Mamil Gold Stage 2 Follow On Baby Milk Formula (850g)", "page": {
                                    "regCategoryId": "180101030000", "xParams": "_p_typ=pdp&_p_ispdp=1&_p_item=DU741TBAATAO7DSGAMZ-61110782&_p_prod=153105871&_p_sku=191142180&_p_slr=100047849"
                                }
                                , "supplier_id":100047849, "pdt_price":"47.9"
                            }
                            , "image":"//laz-img-sg.alicdn.com/original/6bdf9b4b759b97f57b438a605f0e37e7.jpg", "inWishlist":false, "itemId":"153105871", "operation": {
                                "operationWeight": 6, "text": "Add to Cart", "type": "default"
                            }
                            ,  "price": {
                                "discount":"-8%", "originalPrice": {
                                    "text": "SGD47.90", "value": 47.9
                                }
                                , "salePrice": {
                                    "text": "SGD44.29", "value": 44.29
                                }
                            } ,
                            ], "sellerId":"100047849", "skuId":"191142180", "stock":18, "stockList":[ {
                                "stoock": 18, "type": "default"
                            }
                            ]
                        }
</script>

Why do you need regex? https://stackoverflow.com/a/1732454/7570485 — Xnkr, Aug 29 '18 at 10:52
Possible duplicate of [Python Regex - How to Get Positions and Values of Matches](https://stackoverflow.com/questions/250271/python-regex-how-to-get-positions-and-values-of-matches) — er.irfankhan11, Aug 29 '18 at 10:56

score 0 · Accepted Answer · answered Aug 29 '18 at 11:01

One approach would be based on a combination of HTML parsing, regex and json loading:

locate the desired script element with BeautifulSoup (you've shown a single script element, but I assume it is actually inside a bigger HTML)
extract the desired Javascript object using regular expressions
use json.loads() to load it into a Python dict/list
get the desired things from this Python object

Smth along these lines:

import json
import re
from pprint import pprint

from bs4 import BeautifulSoup


data = """
<script>
define('app/pc', ['//laz-g-cdn.alicdn.com/lzdfe/pdp-platform/0.1.8/pc.js'], function(app) {
try {
    app.run( {
        "data": {
            "root": {
                "fields": {"skuInfos": {
                        "0": {
                            "categoryId":"8711", "dataLayer": {
                                "pdt_category":["Mother & Baby", "Feeding", "Milk Formula", "Follow On (6 - 12 mnths)"], "pagetype":"pdp", "pdt_discount":"-8%", "pdt_photo":"//laz-img-sg.alicdn.com/original/6bdf9b4b759b97f57b438a605f0e37e7.jpg", "v_voya":1, "brand_name":"Dumex", "brand_id":"30360", "pdt_sku":153105871, "core": {
                                    "country": "SG", "layoutType": "desktop", "language": "en", "currencyCode": "SGD"
                                }
                                , "seller_name":"Dumex", "pdt_simplesku":191142180, "pdt_name":"Dumex Mamil Gold Stage 2 Follow On Baby Milk Formula (850g)", "page": {
                                    "regCategoryId": "180101030000", "xParams": "_p_typ=pdp&_p_ispdp=1&_p_item=DU741TBAATAO7DSGAMZ-61110782&_p_prod=153105871&_p_sku=191142180&_p_slr=100047849"
                                }
                                , "supplier_id":100047849, "pdt_price":"47.9"
                            }
                            , "image":"//laz-img-sg.alicdn.com/original/6bdf9b4b759b97f57b438a605f0e37e7.jpg", "inWishlist":false, "itemId":"153105871", "operation": {
                                "operationWeight": 6, "text": "Add to Cart", "type": "default"
                            }
                            ,  "price": {
                                "discount":"-8%", "originalPrice": {
                                    "text": "SGD47.90", "value": 47.9
                                }
                                , "salePrice": {
                                    "text": "SGD44.29", "value": 44.29
                                }
                            } ,
                            "sellerId":"100047849", "skuId":"191142180", "stock":18, "stockList":[ {
                                "stoock": 18, "type": "default"
                            }
                            ]
                        }
</script>"""


soup = BeautifulSoup(data, "html.parser")

pattern = re.compile(r'"skuInfos": {\s+"0": ({.*})$', re.MULTILINE | re.DOTALL)

script = soup.find("script", text=pattern)

json_string = pattern.search(script.get_text()).group(1)
data = json.loads(json_string)


print(data['price']['salePrice']['value'])
print(data['skuId'])
print(data['dataLayer']['pdt_category'])

Prints:

44.29
191142180
['Mother & Baby', 'Feeding', 'Milk Formula', 'Follow On (6 - 12 mnths)']

Note that for this to work I had to fix a syntax error in the JS itself as I think this is not a full script you actually have. In any case, I can imagine you would need to adjust the pattern to better match the desired JS object(s) for your specific use case.

you are right the script is in bigger html,so this script is in 124 th in html page so how to i select the 124th position script and pass the patten? — , Aug 29 '18 at 11:44

how to extract the value fom script using python regex in webscraping?

1 Answers1