I develop django website on cpanel with MySQL database. I have a function that pull feeds from this website https://travelcommunication.net/feed/ and create an object from that (web scraping using beautifulsoup4).
But when I try to grab the content section, the error appears. But that only happens with certain items, not all.
I try on my local (with sqlite database) and all working fine. I have also tried on heroku (with PostgreSQL database) and all working fine.
Here is my code:
#views.py
def pull_feeds(request, pk):
if request.user.is_superuser:
source = Autoblogging.objects.get(pk=pk)
url = requests.get(source.url)
soup = BeautifulSoup(url.content, "html.parser")
length = source.items
items = soup.find_all('item')[:length]
contents = soup.find_all('content:encoded')[:length]
for i in range(length-1, -1, -1):
content = contents[i].text
title = items[i].title.text
body = content[content.find('<p>'):] #the problem is here .. when i comment this, everything works fine
category = Category.objects.get(pk=source.category.id)
if not Post.objects.filter(title=title).exists():
post = Post(title=title,
body=body, #the problem is here .. when i comment this, everything works fine
category=category)
link = content[content.find('src=')+5:content.find('alt')-2]
img_data = requests.get(link).content
with open('temp_image.jpg', 'wb') as handler:
handler.write(img_data)
with open('temp_image.jpg', 'rb') as handler:
file_name = link.split("/")[-1]
post.cover.save(file_name, files.File(handler))
os.remove("temp_image.jpg")
return redirect("news:autoblogging")
else:
return HttpResponse("Sorry you are not allowed to access this page")
Does anyone know how to fix this error? Thanks.
