Is requests-html's render unable to load soundcloud.com?
Hello,
I'm trying to do the Scraping SoundCloud assigment, and here are the Jupyter cells I'm running:
from bs4 import BeautifulSoup
from requests_html import AsyncHTMLSession
r = await session.get("https://soundcloud.com/discover")
r.status_code
200
await r.html.arender()
soup = BeautifulSoup(r.html.html, 'lxml')
with open('soundcloud_pretty.html', 'wb') as f:
f.write(soup.prettify('utf-8'))
However the webpage doesn't seem to render, since it doesn't recognize the request as performed by a valid browser. I have walk through again every lesson that may cover such issue, but I couldn't find a solution. Here's the part of the webpage that informs the error:
<noscript class="errorPage__inner">
<div class="errorPage__inner">
<p class="errorTitle">
JavaScript is disabled
</p>
<p class="errorText sc-font-light">
You need to enable JavaScript to use SoundCloud
</p>
<div class="errorButtons">
<a class="sc-button sc-button-medium" href="[http://www.enable-javascript.com/](http://www.enable-javascript.com/)" target="_blank">
Show me how to enable it
</a>
</div>
</div>
</noscript>
1 answers ( 0 marked as helpful)
same problem