Pagination Strategies

Any API returning a collection will paginate. Without handling pagination, you get page 1 and silently lose the rest. Each strategy requires a different loop pattern.

Strategy comparison

Strategy Parameters Used by Tradeoff

Offset-based

page, size (or offset, limit)

Cisco ISE ERS, Django REST, many traditional REST APIs

Simple but slow on deep pages — the server skips rows to reach your offset

Cursor-based

cursor, next_token, after

AWS, Slack, Twitter/X, Elasticsearch

Consistent results, no skipping — but you cannot jump to page N

Link header

Link: <url>; rel="next"

GitHub, GitLab

Self-describing — the server tells you exactly where to go next

Keyset

WHERE id > :last_id LIMIT :size

Database-backed APIs, Stripe

Fast at any depth — uses an index seek instead of offset skip

Offset-based pagination

The server accepts a page number (or offset) and a page size. You increment until the response is empty or a total count is reached.

#!/usr/bin/env bash
# Collect all items from an offset-paginated API

page=1
size=100
total=0
all_items="[]"

while true; do
  response=$(curl -s "${API_URL}/items?page=${page}&size=${size}")

  items=$(echo "$response" | jq '.data')
  count=$(echo "$items" | jq 'length')

  if [[ "$count" -eq 0 ]]; then
    break
  fi

  all_items=$(echo "$all_items" "$items" | jq -s '.[0] + .[1]')
  total=$((total + count))
  page=$((page + 1))

  # Safety valve
  if [[ "$page" -gt 1000 ]]; then
    echo "Pagination safety limit reached at page ${page}" >&2
    break
  fi
done

echo "$all_items" | jq "length"  # Total collected

Some APIs use offset and limit instead of page and size. The logic is identical — increment offset by limit each iteration.

Cursor-based pagination

The server returns a cursor (opaque token) pointing to the next page. You pass it back until the cursor is null or absent.

#!/usr/bin/env bash
# Collect all items from a cursor-paginated API

cursor=""
all_items="[]"

while true; do
  if [[ -z "$cursor" ]]; then
    response=$(curl -s "${API_URL}/items?limit=100")
  else
    response=$(curl -s "${API_URL}/items?limit=100&cursor=${cursor}")
  fi

  items=$(echo "$response" | jq '.results')
  all_items=$(echo "$all_items" "$items" | jq -s '.[0] + .[1]')

  # Extract next cursor -- null means last page
  cursor=$(echo "$response" | jq -r '.next_cursor // empty')
  if [[ -z "$cursor" ]]; then
    break
  fi
done

echo "$all_items" | jq 'length'

AWS APIs use NextToken. Slack uses cursor inside a response_metadata object. Elasticsearch uses _scroll_id or search_after. The field name changes; the pattern does not.

Link header pagination

GitHub-style APIs embed pagination URLs in the HTTP Link header:

Link: <https://api.github.com/user/repos?page=2>; rel="next",
      <https://api.github.com/user/repos?page=5>; rel="last"

You follow rel="next" until it disappears.

#!/usr/bin/env bash
# Follow Link header pagination (GitHub-style)

url="https://api.github.com/users/octocat/repos?per_page=30"
all_items="[]"

while [[ -n "$url" ]]; do
  # Capture both headers and body
  response=$(curl -sD - "$url" -H "Accept: application/json")

  # Split headers from body (blank line separates them)
  headers=$(echo "$response" | sed '/^\r$/q')
  body=$(echo "$response" | sed '1,/^\r$/d')

  items=$(echo "$body" | jq '.')
  all_items=$(echo "$all_items" "$items" | jq -s '.[0] + .[1]')

  # Extract next URL from Link header
  url=$(echo "$headers" \
    | grep -oP '(?<=<)[^>]+(?=>; rel="next")' \
    || true)
done

echo "$all_items" | jq 'length'

The regex (?⇐<)[^>]+(?⇒; rel="next") uses a lookbehind to extract the URL between < and > that precedes ; rel="next".

Keyset pagination

Keyset (or seek) pagination uses the last seen ID to fetch the next batch. It requires a stable sort column, typically a primary key or timestamp.

#!/usr/bin/env bash
# Keyset pagination -- WHERE id > last_id

last_id=0
size=100
all_items="[]"

while true; do
  response=$(curl -s "${API_URL}/items?after_id=${last_id}&limit=${size}")

  items=$(echo "$response" | jq '.data')
  count=$(echo "$items" | jq 'length')

  if [[ "$count" -eq 0 ]]; then
    break
  fi

  all_items=$(echo "$all_items" "$items" | jq -s '.[0] + .[1]')

  # Last ID from the batch becomes the starting point for the next
  last_id=$(echo "$items" | jq -r '.[-1].id')
done

echo "$all_items" | jq 'length'

Keyset is the only strategy that stays fast at depth. Offset-based pagination at page 10,000 forces the database to skip 999,900 rows. Keyset uses an index seek regardless of position.

How netapi handles pagination

netapi abstracts pagination internally. When you call a list operation, the client detects the vendor’s pagination strategy and iterates automatically.

From the caller’s perspective, there is no loop:

# netapi handles ISE ERS offset pagination transparently
netapi ise endpoints list --all

# The client iterates pages internally:
# GET /ers/config/endpoint?page=1&size=100
# GET /ers/config/endpoint?page=2&size=100
# ...until empty

Each vendor definition declares its pagination strategy. The HTTP layer reads that definition and loops accordingly. This means adding a new vendor with cursor-based pagination requires only a configuration change, not new loop logic.

Choosing a pagination strategy

When building against a new API:

  1. Check the documentation for pagination parameters

  2. If undocumented, make a request and inspect:

    • Response body for next, cursor, nextToken, offset fields

    • Response headers for Link header

    • Total count fields (total, totalResults, count)

  3. Match the pattern to the correct loop from this page

  4. Set a safety limit on iterations — infinite loops against a production API are unpleasant

When building a new API (as a producer), prefer cursor or keyset pagination. Offset pagination is simpler to implement but scales poorly.