Web Scraping and API Fundamentals in Python

4.8/5

(240)

Introducing you to the fundamentals of data extraction from the web using Python. We will learn about APIs, Beautiful Soup and Requests-HTML.

48 lessons 4h

Start course

Course Overview

Web Scraping and API Fundamentals in Python offers an introduction to the techniques of data extraction from the web. In this course, you will learn how to use one of the most powerful tools on the Internet – APIs. We will also discuss in depth how to obtain information directly from websites using the Beautiful Soup Python package. There will be a short HTML crash course for those not familiar with it. Finally, we will introduce the Requests-HTML package in order to extract dynamically generated JavaScript content.

48 High Quality Lessons

36 Practical Tasks

4 Hours of Video

Certificate of Achievement

Skills you will gain

JupyterProgrammingPythonTheory

What You'll Learn

Introducing you to the fundamentals of data extraction from the web using Python. We will first learn about the preferred method for program-to-program communication – APIs. Then we will delve into the web data extraction aspect with the help of Beautiful Soup.

Learn the fundamentals of web scraping

Implement APIs

Master working with Beautiful Soup

Start using requests-html

Create functioning scrapers

Understand website cookies

Curriculum

Introduction to the course Free

3 Lessons 10 Min

In this section, you will learn about the importance of data visualization, as well as some theoretical foundations for creating charts. We introduce popular frameworks for choosing an appropriate visualization for your data, discuss color theory, and show different approaches to selecting the colors for your graphic.
What does the course cover Free What is Web Scraping Free Ethics of Scraping Free
Setting Up the Environment Free

3 Lessons 4 Min

Here, we set up different environments for the course. First, we will guide you through the installation process for Tableau. Then, you will get familiar with the step-by-step process of installing Anaconda and Jupyter and an introductory tour of the Jupyter Dashboard for Python. Finally, you’ll learn how to install R and R studio, explore the latter’s main features and learn how to customize its appearance.
Setting up the Environment Free Installing the Necessary Packages Free API overview Free
Working with APIs Free

12 Lessons 41 Min

We dive straight into visualization with the bar chart! We will take a look at a data set for second-hand car advertisements and use it to create a bar chart in Excel, Tableau, Python, and R. We’ll also lift the curtain on the key elements to making an outstanding bar chart.
HTTP requests GET and POST requests Free JSON preferred data exchange format for APIs Free Exchange rates API GETting a JSON response Free Incorporating parameters in a GET request Free Additional API functionalities Free Creating a simple currency converter Free iTunes API Free iTunes API Structuring and exporting the data Free GitHub API Pagination Free EDAMAM API Initial setup and registration Free EDAMAM API Sending a POST request Free Downloading files with Requests Free
HTML overview

8 Lessons 38 Min

In this section, we explore pie charts, which, despite criticism, are among the most popular visualizations. You will learn how to create a pie chart of engine fuel types in Excel, Tableau, Python, and R, and discover what to avoid when making a pie chart.
What is HTML? Structure of HTML Syntax of HTML. Tags Tag attributes Popular tags CSS and JavaScript Character encoding XHTML and code style
Web Scraping with Beautiful Soup

10 Lessons 49 Min

Here, you will create your own stacked area chart. Once again, our data follows the automobile theme with one additional element - time series, as the chart follows the popularity of different engine fuel types across the years.
Introduction to the Beautiful Soup package Workflow of Web Scraping Setting up your first scraper Searching and navigating the HTML tree Searching the HTML tree by attributes Extracting data from the HTML tree Extracting text from an HTML tag Practical example dealing with links Extracting data from nested HTML tags Scraping multiple pages automatically
Practical project: Scraping Rotten Tomatoes

5 Lessons 25 Min

In this section, we continue discussing time series data. We will turn our attention to the financial world and explore the stock market returns for two major indices: S&P 500 and FTSE 100. In conclusion, you’ll find out the advantages of using a line chart and what you should be wary of when creating one.
Setting up your scraper Extracting the title and year of each movie Extracting the rest of the information Dealing with the cast of the movies Storing and exporting the data in a structured form
Scraping HTML tables

1 Lesson 5 Min

This section centers around the histogram – an integral part of the data analysis process. We will create a histogram of the price of California's real estate. Here, we devote an extra lecture and explore how to choose the right number of bins for your histogram.
Scraping HTML tables with the help of Pandas
Common roadblocks when scraping

1 Lesson 13 Min

We’ll explore a combination chart of a scatter and a regression line by using marketing data and a regression line to quantify the relationship between a company’s advertising budget and its sales. You will learn how to create a regression scatter in Excel, Tableau, Python, and R, and discover different types of relationships between features in data. the model residuals can be beneficial in model selection.
Common roadblocks when Web Scraping
The requests-html package

5 Lessons 25 Min

Here, we will introduce another Web Scraping package – ‘Requests-HTML’. We are doing it because it has one big advantage over Beautiful Soup – the ability to execute JavaScript. Thus, this allows us to extract dynamically generated content which is exactly what we will do.
Introduction to the requests-html package Exploring the capabilities of requests-html for Web Scraping Searching for text CSS selectors Scraping JavaScript

Student feedback

4.8/5

240 ratings

5 stars

206 (86%)

4 stars

32 (13%)

3 stars

2 (1%)

2 stars

0 (0%)

1 star

0 (0%)

Vagner Bessa

14.11.2022

Very concise clear lessons, covering so much. And if I may, I'd like to suggest improving this course with and additional session covering the use of selenium for scrapping webpages full of ajax which cannot be done with requests.

Frederick Zhang

16.11.2022

Given the fickle nature of the Internet, much of the material taught in this course no longer works as is. I therefore recommend supplemental tutorials on the Beautiful Soup, requests and requests_html libraries.