Chapter 15: Generating Data

Although the initial plot shows that the numbers are increasing, the label type is too small and the line is a little thin to read easily. Fortunately, Matplotlib allows you to adjust every feature of a visualization.

We’ll use a few of the available customizations to improve this plot’s readability. Let’s start by adding a title and labeling the axes:

Now that we can read the chart better, we can see that the data is not plotted correctly. Notice at the end of the graph that the square of 4.0 is shown as 25! Let’s fix that.

When you give plot() a single sequence of numbers, it assumes the first data point corresponds to an x-value of 0, but our first point corresponds to an x-value of 1. We can override the default behavior by giving plot() both the input and output values used to calculate the squares:

Now plot() doesn’t have to make any assumptions about how the output numbers were generated, and the resulting plot is correct.

You can specify a number of arguments when calling plot() and use a number of methods to customize your plots after generating them. We’ll continue to explore these approaches to customization as we work with more interesting datasets throughout this chapter.

Matplotlib has a number of predefined styles available. These styles contain a variety of default settings for background colors, gridlines, line widths, fonts, font sizes, and more. To see the full list of available styles, run the following lines in a terminal session:

To use any of these styles, add one line of code before calling subplots():

A wide variety of styles is available; play around with these styles to find some that you like.

Sometimes, it’s useful to plot and style individual points based on certain characteristics. For example, you might plot small values in one color and larger values in a different color. You could also plot a large dataset with one set of styling options and then emphasize individual points by replotting them with different options.

To plot a single point, pass the single x- and y-values of the point to scatter():

Let’s style the output to make it more interesting. We’ll add a title, label the axes, and make sure all the text is large enough to read:

To plot a series of points, we can pass scatter() separate lists of x- and y-values, like this:

The x_values list contains the numbers to be squared, and y_values contains the square of each number. When these lists are passed to scatter(), Matplotlib reads one value from each list as it plots each point. The points to be plotted are (1, 1), (2, 4), (3, 9), (4, 16), and (5, 25).

Writing lists by hand can be inefficient, especially when we have many points. Rather than writing out each value, let’s use a loop to do the calculations for us.

Here’s how this would look with 1,000 points:

When the numbers on an axis get large enough, Matplotlib defaults to scientific notation for tick labels. This is usually a good thing, but you can tell Matplotlib to keep using plain notation if you prefer:

The ticklabel_format() method allows you to override the default tick label style for any plot.

To change the color of the points, pass the argument color to scatter() with the name of a color to use in quotation marks, as shown here:

You can also define custom colors using the RGB color model. To define a color, pass the color argument a tuple with three float values (one each for red, green, and blue, in that order), using values between 0 and 1. For example, the following line creates a plot with light-green dots:

Values closer to 0 produce darker colors, and values closer to 1 produce lighter colors.

A colormap is a sequence of colors in a gradient that moves from a starting to an ending color. In visualizations, colormaps are used to emphasize patterns in data. For example, you might make low values a light color and high values a darker color.

The pyplot module includes a set of built-in colormaps. To use one of these colormaps, you need to specify how pyplot should assign a color to each point in the dataset. Here’s how to assign a color to each point, based on its y-value:

The c argument is similar to color, but it’s used to associate a sequence of values with a color mapping. We pass the list of y-values to c, and then tell pyplot which colormap to use with the cmap argument. This code colors the points with lower y-values light blue and the points with higher y-values dark blue.

If you want to save the plot to a file instead of showing it in the Matplotlib viewer, you can use plt.savefig() instead of plt.show():

The first argument is a filename for the plot image, which will be saved in the same directory as scatter_squares.py. The second argument trims extra whitespace from the plot. You can also call savefig() with a Path object, and write the output file anywhere you want on your system.

15-1. Cubes: A number raised to the third power is a cube. Plot the first five cubic numbers, and then plot the first 5,000 cubic numbers.

15-2. Colored Cubes: Apply a colormap to your cubes plot.

To create a random walk, we’ll create a RandomWalk class, which will make random decisions about which direction the walk should take. The class needs three attributes: one variable to track the number of points in the walk, and two lists to store the x- and y-coordinates of each point in the walk.

We’ll only need two methods for the RandomWalk class: the init() method and fill_walk(), which will calculate the points in the walk. Let’s start with the init() method:

We’ll use the fill_walk() method to determine the full sequence of points in the walk. Add this method to random_walk.py:

Here’s the code to plot all the points in the walk:

Every random walk is different, and it’s fun to explore the various patterns that can be generated. One way to use the preceding code to make multiple walks without having to run the program several times is to wrap it in a while loop, like this:

This code generates a random walk, displays it in Matplotlib’s viewer, and pauses with the viewer open. When you close the viewer, you’ll be asked whether you want to generate another walk.

In this section, we’ll customize our plots to emphasize the important characteristics of each walk and deemphasize distracting elements.

15-3. Molecular Motion: Modify rw_visual.py by replacing ax.scatter() with ax.plot(). To simulate the path of a pollen grain on the surface of a drop of water, pass in the rw.x_values and rw.y_values, and include a linewidth argument. Use 5,000 instead of 50,000 points to keep the plot from being too busy.

15-4. Modified Random Walks: In the RandomWalk class, x_step and y_step are generated from the same set of conditions. Modify the values in these lists to see what happens to the overall shape of your walks. Try a longer list of choices for the distance, such as 0 through 8, or remove the −1 from the x- or y-direction list.

15-5. Refactoring: The fill_walk() method is lengthy. Create a new method called get_step() to determine the direction and distance for each step, and then calculate the step. You should end up with two calls to get_step() in fill_walk():

This refactoring should reduce the size of fill_walk() and make the method easier to read and understand.

Install Plotly using pip, just as you did for Matplotlib:

Plotly Express depends on pandas, which is a library for working efficiently with data, so we need to install that as well.

To see what kind of visualizations are possible with Plotly, visit the gallery of chart types at plotly.com/python.

We’ll create the following Die class to simulate the roll of one die:

Before creating a visualization based on the Die class, let’s roll a D6, print the results, and check that the results look reasonable:

Here’s a sample set of results:

A quick scan of these results shows that the Die class seems to be working. We see the values 1 and 6, so we know the smallest and largest possible values are being returned, and because we don’t see 0 or 7, we know all the results are in the appropriate range.

We’ll analyze the results of rolling one D6 by counting how many times we roll each number:

These results look reasonable: we see six frequencies, one for each possible number when you roll a D6, and no frequency is significantly higher than any other.

Now that we have the data we want, we can generate a visualization in just a couple lines of code using Plotly Express:

We first import the plotly.express module, using the conventional alias px. We then use the px.bar() function to create a bar graph. In the simplest use of this function, we only need to pass a set of x-values and a set of y-values. Here the x-values are the possible results from rolling a single die, and the y-values are the frequencies for each possible result.

The final line calls fig.show(), which tells Plotly to render the resulting chart as an HTML file and open that file in a new browser tab.

This chart is dynamic and interactive. If you change the size of your browser window, the chart will resize to match the available space. If you hover over any of the bars, you’ll see a pop-up highlighting the specific data related to that bar.

Feel free to try different chart types by changing px.bar() to something like px.scatter() or px.line(). You can find a full list of available chart types at plotly.com/python/plotly-express.

Now that we know we have the correct kind of plot and our data is being represented accurately, we can focus on adding the appropriate labels and styles for the chart.

The first way to customize a plot with Plotly is to use some optional parameters in the initial call that generates the plot, in this case, px.bar(). Here’s how to add an overall title and a label for each axis:

Rolling two dice results in larger numbers and a different distribution of results. Let’s modify our code to create two D6 dice to simulate the way we roll a pair of dice. Each time we roll the pair, we’ll add the two numbers (one from each die) and store the sum in results. Save a copy of die_visual.py as dice_visual.py and make the following changes:

This graph shows the approximate distribution of results you’re likely to get when you roll a pair of D6 dice. As you can see, you’re least likely to roll a 2 or a 12 and most likely to roll a 7. This happens because there are six ways to roll a 7: 1 and 6, 2 and 5, 3 and 4, 4 and 3, 5 and 2, and 6 and 1.

There’s one issue that we should address with the plot we just generated. Now that there are 11 bars, the default layout settings for the x-axis leave some of the bars unlabeled. Plotly has an update_layout() method that can be used to make a wide variety of updates to a figure after it’s been created. Here’s how to tell Plotly to give each bar its own label:

The update_layout() method acts on the fig object, which represents the overall chart. Here we use the xaxis_dtick argument, which specifies the distance between tick marks on the x-axis. We set that spacing to 1, so that every bar is labeled.

Let’s create a six-sided die and a ten-sided die, and see what happens when we roll them 50,000 times:

Instead of one most likely result, there are five such results. This happens because there’s still only one way to roll the smallest value (1 and 1) and the largest value (6 and 10), but the smaller die limits the number of ways you can generate the middle numbers. There are six ways to roll a 7, 8, 9, 10, or 11, these are the most common results, and you’re equally likely to roll any one of them.

When you have a figure you like, you can always save the chart as an HTML file through your browser. But you can also do so programmatically. To save your chart as an HTML file, replace the call to fig.show() with a call to fig.write_html():

The write_html() method requires one argument: the name of the file to write to. If you only provide a filename, the file will be saved in the same directory as the .py file. You can also call write_html() with a Path object, and write the output file anywhere you want on your system.

15-6. Two D8s: Create a simulation showing what happens when you roll two eight-sided dice 1,000 times. Try to picture what you think the visualization will look like before you run the simulation, then see if your intuition was correct. Gradually increase the number of rolls until you start to see the limits of your system’s capabilities.

15-7. Three Dice: When you roll three D6 dice, the smallest number you can roll is 3 and the largest number is 18. Create a visualization that shows what happens when you roll three D6 dice.

15-8. Multiplication: When you roll two dice, you usually add the two numbers together to get the result. Create a visualization that shows what happens if you multiply these numbers by each other instead.

15-9. Die Comprehensions: For clarity, the listings in this section use the long form of for loops. If you’re comfortable using list comprehensions, try writing a comprehension for one or both of the loops in each of these programs.

15-10. Practicing with Both Libraries: Try using Matplotlib to make a die-rolling visualization, and use Plotly to make the visualization for a random walk. (You’ll need to consult the documentation for each library to complete this exercise.)

D15-1. Service Uptime Squares: Generate a list of simulated uptime percentages for nodes 1 through 20, calculated as min(100, node ** 1.5) rounded to 2 decimal places. Print the list and use min(), max(), and sum() / len() to compute statistics. This mirrors the square number sequence generation pattern.

D15-2. Network Latency Walk: Implement a LatencyWalk class modeled on RandomWalk. Initialize with num_points=1000 and a starting latency of 20.0 ms. In fill_walk(), each step adds a random delta from choice([-2, -1, 0, 1, 2, 3]) but clamps the value between 1 and 200 (no negative latency). After generating the walk, print the min, max, and average latency.

D15-3. VLAN Traffic Frequency: Create a TrafficDie class analogous to Die that returns a random VLAN ID from a weighted list (INFRA=100 appears 3×, SECURITY=110 appears 2×, SERVICES=120, DATA=10, GUEST=30 each appear once). Roll it 1000 times. Count the frequency of each VLAN ID using .count(). Print the frequency table.

D15-4. BGP Event Histogram Data: Simulate 500 BGP session events where each event is one of ['established', 'idle', 'active', 'connect', 'opensent', 'openconfirm'], chosen randomly. Count the frequency of each state. Print a text-based bar chart where each state is followed by a bar of # characters proportional to its count (scale to max 40 chars).

D15-5. Stack Health Time Series: Generate a time series of 100 health check results for a 5-service stack. Each check, each service has an 80% chance of being 'up' and 20% chance of being 'degraded'. Store results as a list of dicts. Print the percentage of checks where all 5 services were up simultaneously.

C15-1. Auth Success Rate Curve: Generate auth success counts for ISE policy sets 1 through 10, calculated as int(1000 * (1 - 0.05 * i)) where i is the policy set index. This simulates decreasing success rate as policies get stricter. Print the list and compute min, max, and mean.

C15-2. Log Volume Walk: Implement a LogVolumeWalk class modeled on RandomWalk. Initialize with num_points=500 and a starting volume of 1000. Each step adds a random delta from choice([-50, -25, 0, 25, 50, 100]), clamped between 0 and 10000. After generating the walk, print the min, max, and average volume across all points.

C15-3. Syslog Severity Frequency: Create a SyslogDie class that returns a random severity level (0–7) with realistic weights (0=1, 1=2, 2=5, 3=10, 4=20, 5=15, 6=30, 7=17 occurrences in the pool). Roll it 1000 times. Count the frequency of each severity. Print the frequency table with severity labels.

C15-4. Endpoint Auth Distribution: Simulate 500 endpoint authentication attempts where each attempt randomly selects one of ['EAP-TLS', 'EAP-TEAP', 'MSCHAPv2', 'MAB', 'WebAuth']. Count the frequency of each protocol. Print a text-based bar chart proportional to frequency (scale to max 40 chars).

C15-5. Pipeline Throughput Time Series: Generate a 200-step time series of pipeline throughput values starting at 5000 events/sec. Each step, throughput changes by a random delta from randint(-200, 300), clamped between 0 and 20000. Count how many steps the throughput was above 8000 (high load) and below 2000 (near-stall). Print a summary.

L15-1. Disk Usage Squares: Generate a list of simulated disk usage percentages for filesystems 1 through 15, calculated as min(100, fs ** 1.8 / 10) rounded to 2 decimal places. Print the list and compute min, max, and mean. Flag any filesystem above 80%.

L15-2. CPU Load Walk: Implement a CPULoadWalk class modeled on RandomWalk. Initialize with num_points=300 and a starting load of 20.0. Each step adds a random delta from choice([-3, -2, -1, 0, 1, 2, 5]), clamped between 0 and 100. After generating the walk, print the min, max, and average CPU load. Count steps above 80% (critical).

L15-3. Package Install Frequency: Create an InstallDie class that returns a random exit code from [0, 0, 0, 0, 1, 2] (0=success, 1=partial, 2=failure). Roll it 500 times. Count the frequency of each exit code. Print the frequency table with labels and a success rate percentage.

L15-4. Service Restart Distribution: Simulate 300 service restart events where each restart selects a restart reason from ['OOM', 'segfault', 'config_change', 'manual', 'dependency_failure', 'watchdog']. Count the frequency of each reason. Print a text-based bar chart.

L15-5. Filesystem Check Time Series: Generate a 150-step time series tracking the number of filesystem errors detected per check, starting at 0. Each step adds randint(0, 3) errors and occasionally resolves randint(0, 1). Clamp to 0 minimum. Count the number of steps with more than 5 active errors. Print a summary including total errors accumulated.

E15-1. Vocabulary Growth Curve: Generate a list of cumulative vocabulary words learned across 20 study sessions, calculated as int(50 * session ** 0.8). Print the list and compute min, max, and growth rate from session 1 to 20. This mirrors the square sequence pattern.

E15-2. Study Intensity Walk: Implement a StudyWalk class modeled on RandomWalk. Initialize with num_points=200 and a starting intensity of 50 (0–100 scale). Each step adds a random delta from choice([-5, -3, 0, 3, 5, 8]), clamped between 0 and 100. After generating the walk, print the min, max, and average intensity across all steps.

E15-3. Chapter Difficulty Frequency: Create a ChapterDie class that returns a difficulty rating (1–5) for Don Quijote chapters, with weights approximating a realistic distribution (1=5, 2=15, 3=40, 4=30, 5=10 in the pool). Roll it 500 times. Count the frequency of each rating. Print the frequency table with labels.

E15-4. DELE Exercise Type Distribution: Simulate 400 DELE practice sessions where each session randomly selects one of ['comprensión lectora', 'comprensión auditiva', 'expresión escrita', 'expresión oral', 'léxico', 'gramática']. Count the frequency of each exercise type. Print a text-based bar chart proportional to frequency.

E15-5. Vocabulary Retention Time Series: Generate a 100-step time series tracking retained vocabulary words, starting at 500. Each step adds randint(0, 15) new words and subtracts randint(0, 5) (forgot), clamped to 0 minimum. After generating the series, print the min, max, final count, and net words gained over all 100 steps.