Last answered:

10 Dec 2022

Posted on:

14 Nov 2022

0

Is requests-html's render unable to load soundcloud.com?

Hello,
I'm trying to do the Scraping SoundCloud assigment, and here are the Jupyter cells I'm running:

from bs4 import BeautifulSoup
from requests_html import AsyncHTMLSession


r = await session.get("https://soundcloud.com/discover")
r.status_code

200

await r.html.arender()


soup = BeautifulSoup(r.html.html, 'lxml')
with open('soundcloud_pretty.html', 'wb') as f:
    f.write(soup.prettify('utf-8'))

However the webpage doesn't seem to render, since it doesn't recognize the request as performed by a valid browser. I have walk through again every lesson that may cover such issue, but I couldn't find a solution. Here's the part of the webpage that informs the error:

<noscript class="errorPage__inner">
  <div class="errorPage__inner">
    <p class="errorTitle">
     JavaScript is disabled
    </p>
    <p class="errorText sc-font-light">
     You need to enable JavaScript to use SoundCloud
    </p>
    <div class="errorButtons">
      <a class="sc-button sc-button-medium" href="[http://www.enable-javascript.com/](http://www.enable-javascript.com/)" target="_blank">
        Show me how to enable it
      </a>
    </div>
  </div>
</noscript>
1 answers ( 0 marked as helpful)
Posted on:

10 Dec 2022

0

same problem

Submit an answer