Web Scraping and API Fundamentals in Python
Web Scraping and API Fundamentals in Python offers an introduction to the techniques of data extraction from the web. In this course, you will learn how to use one of the most powerful tools on the Internet – APIs. We will also discuss in depth how to obtain information directly from websites using the BeautifulSoup Python package. There will be a short HTML crash course for those not familiar with it. Finally, we will introduce the Requests-HTML package in order to extract dynamically generated JavaScript content.
4 hours
8 exercises
26 quizzes
44 downloadables Sign up to
preview the program
for FREE!
Create a free account and start learning data science today.
create free account
Our graduates work at exciting places:
Section 1
Course Introduction
In this first section, we will discuss what the course covers, why you need to learn Web Scraping and give you some notes on the ethics of scraping.
What does the course cover? What is Web Scraping? Ethics of Scraping Section 2
Setting Up the Environment
In this part of the course, we will explain to you how to set up Python 3 and then load up Jupyter. We’ll also show you what the Anaconda Prompt is and how we use it to download and import new modules.
Setting up the environment - Do not skip, please! Why Python and why Jupyter? Installing Anaconda Jupyter Dashboard - Part 1 Jupyter Dashboard - Part 2 Installing the packages Section 3
Working with APIs
Here we will introduce what APIs are and how to use them. In order to do that, we will discuss the popular data exchange format JSON, as well as HTTP requests and the Python library to submit them – ‘requests’. At the end of the section, we will show you how to deal with an API that requires registration.
API overview HTTP requests: GET and POST requests JSON: preferred data exchange format for APIs Exchange rates API: GETting a JSON response Incorporating parameters in a GET request Additional API functionalities Creating a simple currency converter iTunes API iTunes API: Structuring and exporting the data GitHub API: Pagination EDAMAM API: Initial setup and registration EDAMAM API: Sending a POST request Downloading files with Requests Section 4
HTML Overview
Web Scraping relies on extracting information from the source code of webpages. Thus, a general understanding of HTML is required. This section is a short crash course for those that are not familiar with HTML. It is meant as an intuitive look into the basics, not a comprehensive guide.
What is HTML? Structure of HTML Syntax of HTML. Tags Tag attributes Popular tags CSS and JavaScript Character encoding XHTML and code style Section 5
Web Scraping with Beautiful Soup
After familiarizing with HTML, we are ready to delve into the Web Scraping itself. We will now introduce the “Beautiful Soup” package and explore its capabilities.
Introduction to the Beautiful Soup package Workflow of Web Scraping Setting up your first scraper Searching and navigating the HTML tree Searching the HTML tree by attributes Exctracting data from the HTML tree Extracting text from an HTML tag Practical example: dealing with links Extracting data from nested HTML tags Scraping multiple pages automatically Section 6
Practical Project: Scraping Rotten Tomatoes
Now that we’ve seen what Beautiful Soup can do, we will devote this section to practicing our newly formed skills. We are going to obtain information about movies from a ‘Rotten Tomatoes’ rank list.
Setting up your scraper Extracting the title and year of each movie Extracting the rest of the information Dealing with the cast of the movies Storing and exporting the data in a structured form Section 7
Scraping HTML Tables
In this short section, we will discuss an easy way to scrape HTML tables.
Scraping HTML tables with the help of Pandas Section 8
Common Roadblocks when Scraping
Although we have done a decent amount of scraping so far in the course, this is one of those topics that can depend very much so on the website we choose. Different websites present specific problems. Thus, in this section, we will discuss what are the most common problems that you will have to deal with and give you solutions and workarounds.
Common roadblocks when Web Scraping Section 9
The Requests-HTML Package
Here, we will introduce another Web Scraping package – ‘Requests-HTML’. We are doing it because it has one big advantage over Beautiful Soup – the ability to execute JavaScript. Thus, this allows us to extract dynamically generated content which is exactly what we will do.
Introduction to the requests-html package Exploring the capabilities of requests-html for Web Scraping Searching for text CSS selectors Scraping JavaScript Advanced Specialization
This course is part of Module 4 of the 365 Data Science Program. The complete training consists of four modules, each building upon your knowledge from the previous one. Module 4 is focused on developing a specialized, industry-relevant skill set, and students are encouraged to complete Modules 1, 2, and 3 before they start this part of the training. Here, you will learn how to perform Credit Risk Modeling for banks, Customer Analytics for retail or other commercial companies, and Time Series Analysis for finance and stock data.
See All Modules Why Choose the 365 Data Science Program?
Practice
Real-life project and data. Solve them on your own computer as you would in the office.
Q&A Hub
Our expert instructors are happy to help. Post a question and get a personal answer by one of our instructors.
Certificates
Earn a verifiable certificate after each completed course. Celebrate your successes and share your progress with your professional network!
Trust the other 450,000 students
The course is in-depth and is delivered at a steady pace with eye catching visuals. The instructors go through all the basics really well. They try not to over-simplify the material, you get a good sense аof how deep Data Science is in the course. Great job!!!
This course is amazing! After watching the video carefully and doing all the exercises, I am even capable of having discussions with Machine learning major Master’s students! High standard course with reasonable pricing.
Very clear and in-depth explanation of data science and how all the inter-related concepts apply in real life business environment. Absolutely great for beginners! Best data science course I have come across so far!
I would highly recommend the course to any beginner who wants to venture into the world of Data Science. The concepts are very well explained and there is an emphasis on practical application which really helps create a better understanding of the concepts.
Ready to start?
Sign up today for FREE!
