Chapter 17: Working with APIs

GitHub’s API lets you request a wide range of information through API calls. To see what an API call looks like, enter the following into your browser’s address bar and press ENTER:

This call returns the number of Python projects currently hosted on GitHub, as well as information about the most popular Python repositories. Let’s examine the call. The first part, api.github.com/, directs the request to the part of GitHub that responds to API calls. The next part, search/repositories, tells the API to conduct a search through all the repositories on GitHub.

The question mark after repositories signals that we’re about to pass an argument. The q stands for query, and the equal sign (=) lets us begin specifying a query (q=). By using language:python, we indicate that we want information only on repositories that have Python as the primary language. The final part, +sort:stars, sorts the projects by the number of stars they’ve been given.

The following snippet shows the first few lines of the response:

The Requests package allows a Python program to easily request information from a website and examine the response. Use pip to install Requests:

If you use a command other than python to run programs or start a terminal session, such as python3, your command will look like this:

Now we’ll write a program to automatically issue an API call and process the results:

Finally, we print the keys from response_dict and see the following output:

Because the status code is 200, we know that the request was successful. The response dictionary contains only three keys: 'total_count', 'incomplete_results', and 'items'. Let’s take a look inside the response dictionary.

With the information from the API call represented as a dictionary, we can work with the data stored there. Let’s generate some output that summarizes the information. This is a good way to make sure we received the information we expected, and to start examining the information we’re interested in:

The results give us a clearer picture of the actual data:

Let’s pull out the values for some of the keys in repo_dict:

The output should look something like this:

We can see that the most-starred Python project on GitHub as of this writing is public-apis. Its owner is an organization with the same name, and it has been starred by almost 200,000 GitHub users.

When we make a visualization for this data, we’ll want to include more than one repository. Let’s write a loop to print selected information about each repository the API call returns so we can include them all in the visualization:

Some interesting projects appear in these results, and it might be worth looking at a few. But don’t spend too much time here, because we’re about to create a visualization that will make the results much easier to read.

Most APIs have rate limits, which means there’s a limit to how many requests you can make in a certain amount of time. To see if you’re approaching GitHub’s limits, enter api.github.com/rate_limit into a web browser. You should see a response that begins like this:

Plotly supports a number of ways to style and customize the plots, once you know the information in the plot is correct. We’ll make some changes in the initial px.bar() call and then make some further adjustments to the fig object after it’s been created.

We’ll start styling the chart by adding a title and labels for each axis:

In Plotly, you can hover the cursor over an individual bar to show the information the bar represents. This is commonly called a tooltip, and in this case, it currently shows the number of stars a project has. Let’s create a custom tooltip to show each project’s description as well as the project’s owner.

We need to pull some additional data to generate the tooltips:

Because Plotly allows you to use HTML on text elements, we can easily add links to a chart. Let’s use the x-axis labels as a way to let the viewer visit any project’s home page on GitHub. We need to pull the URLs from the data and use them when generating the x-axis labels:

When we call px.bar(), we use repo_links for the x-values in the chart. The result looks the same as before, but now the viewer can click any of the project names at the bottom of the chart to visit that project’s home page on GitHub. Now we have an interactive, informative visualization of data retrieved through an API!

Once a chart has been created, almost any aspect of the chart can be customized through an update method. We’ve used the update_layout() method previously. Another method, update_traces(), can be used to customize the data that’s represented on a chart.

Let’s change the bars to a darker blue, with some transparency:

In Plotly, a trace refers to a collection of data on a chart. The update_traces() method can take a number of different arguments; any argument that starts with marker_ affects the markers on the chart. Here we set each marker’s color to 'SteelBlue'; any named CSS color will work here. We also set the opacity of each marker to 0.6. An opacity of 1.0 will be entirely opaque, and an opacity of 0 will be entirely invisible.

Plotly’s documentation is extensive and well organized; however, it can be hard to know where to start reading. A good place to start is with the article Plotly Express in Python at plotly.com/python/plotly-express. This is an overview of all the plots you can make with Plotly Express, and you can find links to longer articles about each individual chart type.

If you want to understand how to customize Plotly charts better, the article Styling Plotly Express Figures in Python will expand on what you’ve seen in Chapters 15-17. You can find this article at plotly.com/python/styling-plotly-express.

For more about the GitHub API, refer to its documentation at docs.github.com/en/rest. Here you’ll learn how to pull a wide variety of information from GitHub. If you have a GitHub account, you can work with your own data as well as the publicly available data from other users' repositories.

17-1. Other Languages: Modify the API call in python_repos.py so it generates a chart showing the most popular projects in other languages. Try languages such as JavaScript, Ruby, C, Java, Perl, Haskell, and Go.

17-2. Active Discussions: Using the data from hn_submissions.py, make a bar chart showing the most active discussions currently happening on Hacker News. The height of each bar should correspond to the number of comments each submission has. The label for each bar should include the submission’s title and act as a link to the discussion page for that submission. If you get a KeyError when creating a chart, use a try-except block to skip over the promotional posts.

17-3. Testing python_repos.py: In python_repos.py, we printed the value of status_code to make sure the API call was successful. Write a program called test_python_repos.py that uses pytest to assert that the value of status_code is 200. Figure out some other assertions you can make: for example, that the number of items returned is expected and that the total number of repositories is greater than a certain amount.

17-4. Further Exploration: Visit the documentation for Plotly and either the GitHub API or the Hacker News API. Use some of the information you find there to either customize the style of the plots we’ve already made or pull some different information and create your own visualizations. If you’re curious about exploring other APIs, take a look at the APIs mentioned in the GitHub repository at github.com/public-apis.

D17-1. Simulated GitHub API Response: Create a Python dict that mimics the GitHub API response structure — a 'total_count' integer, 'incomplete_results' boolean, and 'items' list of repo dicts (each with 'name', 'stargazers_count', 'html_url', 'owner' with 'login', and 'description'). Use at least 8 items representing homelab projects. Write a program that extracts name, stars, and description into separate lists and prints a summary for each.

D17-2. Homelab Repo Sorting: Extend D17-1: add 'open_issues_count' and 'updated_at' fields to each repo dict. Use itemgetter() to sort the list by 'open_issues_count' in descending order. Print the sorted list showing name, stars, and open issues.

D17-3. Rate Limit Simulator: Write a class called APIRateLimit with limit = 10, remaining = 10, and reset_time (Unix timestamp 60 seconds from now). Add a make_call() method that decrements remaining and raises a RuntimeError with a friendly message when remaining reaches 0. Add a time_until_reset() method. Simulate 12 calls and handle the error gracefully.

D17-4. Hover Text Builder: Using your simulated repo list from D17-1, build a hover_texts list for each repo in the format "{owner}<br />{description}". Also build a repo_links list using the HTML anchor tag format <a href='{url}'>{name}</a>. Print the first 3 items from each list.

D17-5. Multi-API Aggregator: Simulate two API responses — one for GitHub repos and one for Hacker News submissions — each as a Python dict. Write a function merge_results(github_items, hn_items) that returns a combined list of dicts with a source field ('github' or 'hn'), a title field, and a score field. Sort the combined list by score descending using itemgetter(). Print the merged results.

C17-1. Simulated ISE API Response: Create a Python dict that mimics a JSON REST API response from ISE — a 'total' count, 'page' info, and 'SearchResult' with a 'resources' list of endpoint dicts (each with 'id', 'name', 'mac', 'status'). Use at least 8 items. Write a program that extracts name, mac, and status into separate lists and prints a summary for each endpoint.

C17-2. Endpoint Sorting: Extend C17-1: add 'last_seen' (ISO timestamp) and 'auth_failures' integer fields to each endpoint dict. Use itemgetter() to sort the list by 'auth_failures' in descending order. Print the sorted list.

C17-3. Rate Limit for Security APIs: Write a class called SecAPILimit with limit = 100, remaining = 100, and a reset_time timestamp. Add make_call(endpoint) that decrements remaining, prints the endpoint called, and raises RuntimeError when exhausted. Simulate 12 calls to different ISE endpoints and handle the error.

C17-4. Alert Hover Text Builder: Using a simulated list of security alerts (each with 'name', 'severity', 'source', 'detail_url'), build a hover_texts list in the format "{severity}<br />{source}" and an alert_links list using the HTML anchor tag format. Print the first 3 items from each list.

C17-5. Monad Pipeline API Aggregator: Simulate two API responses — one for Monad pipeline stages and one for Sentinel alert rules — each as a Python dict. Write merge_pipeline_data(monad_items, sentinel_items) that returns a combined list with source, name, and event_count fields. Sort by event_count descending. Print the merged results.

L17-1. Simulated Package Repo API: Create a Python dict that mimics a package repository API response — a 'total_count', 'results' list of package dicts (each with 'name', 'version', 'downloads', 'description', 'maintainer'). Use at least 8 packages. Extract name, downloads, and description into separate lists. Print a summary for each.

L17-2. Package Sorting: Extend L17-1: add 'open_bugs' and 'last_updated' fields to each package dict. Use itemgetter() to sort by 'open_bugs' descending. Print the sorted list.

L17-3. Rate Limit for Package API: Write a class called PkgAPILimit with limit = 30, remaining = 30, and a reset_time. Add make_call(pkg_name) that decrements remaining and raises RuntimeError when exhausted. Simulate 35 calls and handle the error.

L17-4. Package Hover Text Builder: Using your simulated package list, build hover_texts in the format "{maintainer}<br />{description}" and pkg_links using HTML anchor tags. Print the first 3 items from each list.

L17-5. Multi-Repo Aggregator: Simulate two package repo API responses (e.g., PyPI and a private repo). Write merge_pkg_results(pypi_items, private_items) that returns a combined list with source, name, and downloads fields. Sort by downloads descending. Print the merged results.

E17-1. Simulated Vocabulary API Response: Create a Python dict that mimics a Spanish vocabulary API response — a 'total_count', and a 'words' list of word dicts (each with 'palabra', 'tipo', 'definicion', 'ejemplo', 'frecuencia'). Use at least 8 words from Don Quijote. Extract palabra, frecuencia, and definicion into separate lists. Print a summary for each word.

E17-2. Vocabulary Sorting: Extend E17-1: add 'dificultad' (1–5) and 'capitulo' integer fields to each word dict. Use itemgetter() to sort by 'dificultad' descending. Print the sorted list.

E17-3. Rate Limit for Language API: Write a class called LangAPILimit with limit = 20, remaining = 20, and a reset_time. Add make_call(word) that decrements remaining and raises RuntimeError when exhausted. Simulate 25 vocabulary lookups and handle the error gracefully with a user-friendly message.

E17-4. Word Hover Text Builder: Using your simulated word list, build hover_texts in the format "{tipo}<br />{ejemplo}" and word_links using HTML anchor tags pointing to a dictionary URL. Print the first 3 items from each list.

E17-5. Multi-Source Vocabulary Aggregator: Simulate two API responses — one for Don Quijote vocabulary and one for DELE C2 exam word lists. Write merge_vocab(quijote_items, dele_items) that returns a combined list with source, palabra, and frecuencia fields. Sort by frecuencia descending using itemgetter(). Print the merged results.