Navigation Mathematics
The mathematical foundations of navigation span from basic trigonometry to advanced spherical geometry. This module provides the rigorous treatment expected at the Naval Postgraduate School and Army Command and General Staff College.
Prerequisite: Angle Measurement
Degrees, Minutes, Seconds
Radians
Military Angular Measurement: Mils
Military forces worldwide use mils (milliradians) for artillery, mortar, and precision fire calculations. The mil system provides finer angular resolution than degrees for fire direction.
Definition
A mil (NATO standard) divides a circle into 6400 mils:
|
Different Mil Standards:
|
The Mil Relation Formula
The fundamental relationship for range estimation and fire adjustment:
Where:
-
\(W\) = width of target (meters)
-
\(R\) = range to target (meters)
-
\(m\) = angular measurement (mils)
Rearranged for range estimation:
Fire Adjustment
For lateral adjustments in artillery:
Conversion Factors
| Conversion | Formula |
|---|---|
Degrees → NATO mils |
\(\text{mils} = \text{degrees} \times 17.778\) |
NATO mils → degrees |
\(\text{degrees} = \text{mils} \times 0.05625\) |
Radians → NATO mils |
\(\text{mils} = \text{radians} \times 1018.59\) |
NATO mils → radians |
\(\text{radians} = \text{mils} \times 0.0009817\) |
Planar Trigonometry Review
Right Triangle
C
/|
/ |
c / | a
/ |
/ |
/_____|
A b B
Where:
- A, B, C are angles
- a, b, c are opposite sides
- C = 90° (right angle)
SOHCAHTOA
Pythagorean Theorem
Law of Cosines
Spherical Trigonometry
Earth is a sphere (approximately). Spherical trigonometry deals with triangles on the surface of a sphere.
Spherical Triangle
North Pole
/\
/ \
/ \
/ \
A /________\ B
\ /
\ C /
\ /
\ /
\ /
\/
Equator
-
Vertices are points on the sphere
-
Sides are arcs of great circles
-
Side lengths are measured as angles (central angles)
Spherical Law of Cosines (Sides)
Spherical Law of Cosines (Angles)
Spherical Law of Sines
Napier’s Rules for Right Spherical Triangles
For a spherical triangle with one right angle (C = 90°), Napier’s Rules provide elegant mnemonic formulas. Named after John Napier (1550-1617), inventor of logarithms, these rules are fundamental to celestial navigation.
The Napier Circle
Arrange the five parts (excluding the right angle) in a circle:
co-c
/ \
co-B co-A
| |
a -------- b
"co-" means complement (90° - x)
The five parts in order: \(a, b, \text{co-}A, \text{co-}c, \text{co-}B\)
The Two Rules
Rule 1: Sine of Middle = Product of Tangents of Adjacents
Rule 2: Sine of Middle = Product of Cosines of Opposites
The Ten Formulas
Applying Napier’s Rules generates all ten formulas for right spherical triangles:
| Middle Part | Formula |
|---|---|
\(a\) |
\(\sin a = \sin A \sin c = \tan b \cot B\) |
\(b\) |
\(\sin b = \sin B \sin c = \tan a \cot A\) |
\(\text{co-}A\) (i.e., \(\cos A\)) |
\(\cos A = \cos a \sin B = \tan b \cot c\) |
\(\text{co-}B\) (i.e., \(\cos B\)) |
\(\cos B = \cos b \sin A = \tan a \cot c\) |
\(\text{co-}c\) (i.e., \(\cos c\)) |
\(\cos c = \cos a \cos b = \cot A \cot B\) |
Quadrantal Triangles
When one side equals 90°, the triangle is quadrantal. Napier’s Rules apply by substituting the co-functions.
For a quadrantal triangle with \(c = 90°\):
Great Circle Distance
The shortest path between two points on a sphere follows a great circle.
Haversine Formula
The most numerically stable formula for calculating great circle distance:
Where:
-
\(\phi_1, \phi_2\) = latitudes in radians
-
\(\lambda_1, \lambda_2\) = longitudes in radians
-
\(\Delta\phi = \phi_2 - \phi_1\)
-
\(\Delta\lambda = \lambda_2 - \lambda_1\)
-
\(R\) = Earth’s radius (6,371 km mean)
-
\(d\) = distance
Why Haversine?
The spherical law of cosines has numerical issues for small distances:
When \(\Delta\lambda\) is small, \(\cos(\Delta\lambda) \approx 1\), causing precision loss.
Haversine uses \(\sin^2\) functions which are stable near zero.
Example Calculation
import math
# Convert to radians
lat1 = math.radians(33.9425)
lon1 = math.radians(-118.4081)
lat2 = math.radians(40.6413)
lon2 = math.radians(-73.7781)
dlat = lat2 - lat1
dlon = lon2 - lon1
# Haversine
a = math.sin(dlat/2)**2 + math.cos(lat1) * math.cos(lat2) * math.sin(dlon/2)**2
c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a))
R = 6371 # km
distance = R * c
print(f"Distance: {distance:.2f} km") # 3974.21 km
Bearing Calculations
Initial Bearing (Forward Azimuth)
The direction to follow from start to reach destination:
Result in radians, convert to degrees and normalize to 0-360°.
Final Bearing (Reverse Azimuth)
The bearing when arriving at destination:
Calculate initial bearing from destination to start, then add 180°.
Why Bearings Change
On a great circle route, the bearing constantly changes (except on meridians or equator).
N
│
LAX → → → → → → → → JFK
65° → 67° → 70° → ... → 90°
Initial bearing: ~66° (ENE)
Final bearing: ~90° (E)
Destination Point Calculation
Given start point, bearing, and distance, find destination:
Midpoint Calculation
Rhumb Line vs Great Circle
Rhumb Line (Loxodrome)
A line that crosses all meridians at the same angle. Constant bearing, but NOT the shortest path. The rhumb line spirals toward the poles, never reaching them (except for cardinal bearings).
Rhumb Line Bearing
The constant bearing (course) of a rhumb line:
Where \(\Delta\psi\) is the Mercator latitude difference:
Or equivalently (the inverse Gudermannian):
Rhumb Line Distance
When bearing is 90° or 270° (east-west):
Complete formula:
Where \(q\) is:
Comparison Table
| Great Circle | Rhumb Line | |
|---|---|---|
Path |
Shortest distance |
Constant bearing |
Bearing |
Changes continuously |
Constant |
Map projection |
Curved on Mercator |
Straight on Mercator |
Computation |
Spherical trig |
Logarithmic (Mercator) |
Use case |
Long-distance travel |
Short distances, simple navigation |
Fuel efficiency |
Optimal |
Suboptimal (longer route) |
Navigational ease |
Complex (continuous adjustment) |
Simple (hold one bearing) |
When to Use Each
| Scenario | Recommended |
|---|---|
Transoceanic flight |
Great circle (fuel savings outweigh complexity) |
Coastal sailing < 500 nm |
Rhumb line (simpler, minimal penalty) |
Artillery/surveying |
Depends on range and required precision |
Driving navigation |
Roads don’t follow either! |
Spherical Excess and Area
Spherical Excess
The sum of angles in a spherical triangle exceeds 180°. This excess is directly proportional to area.
Where \(E\) is the spherical excess in degrees.
The excess can also be computed via L’Huilier’s Theorem:
Where \(s = \frac{a+b+c}{2}\) is the semi-perimeter.
Area of Spherical Triangle
Converting from degrees:
Area of Spherical Polygon
For a polygon with \(n\) vertices:
Where \(\theta_i\) are the interior angles in radians.
Vector Operations
Position Vector (ECEF)
Convert lat/lon to unit vector on sphere:
Cross Product
For finding perpendicular vectors:
Dot Product
For finding angles between vectors:
Error Propagation in Navigation
Understanding position uncertainty is critical for military operations. A position is never known exactly—only to within some error bound.
Error Propagation Fundamentals
For a function \(f(x_1, x_2, \ldots, x_n)\) with independent variables having uncertainties \(\sigma_{x_1}, \sigma_{x_2}, \ldots\), the propagated uncertainty is:
Position Uncertainty Circle
For a 2D position with independent errors:
The Circular Error Probable (CEP) is the radius containing 50% of positions:
More precisely (Rayleigh distribution when \(\sigma_x = \sigma_y = \sigma\)):
Military Error Standards
| Standard | Probability | Definition |
|---|---|---|
CEP (Circular Error Probable) |
50% |
Radius containing half of positions |
SEP (Spherical Error Probable) |
50% |
3D sphere containing half of positions |
2DRMS (2× Distance RMS) |
95-98% |
\(2\sqrt{\sigma_x^2 + \sigma_y^2}\) |
R95 |
95% |
Radius containing 95% of positions |
Distance Error Propagation
For the Haversine formula, the error in computed distance depends on errors in input coordinates:
For small distances, a simplified rule of thumb:
Bearing Error Propagation
The bearing from point 1 to point 2:
Where: * \(\sigma_r\) = cross-track position uncertainty * \(d\) = distance to target
At 1 km with 10m position error: \(\sigma_\theta \approx 10/1000 = 0.01 \text{ rad} \approx 0.57°\)
Dead Reckoning Error Accumulation
Dead reckoning errors accumulate with time/distance. For a leg of length \(L\):
Where: * \(\sigma_v\) = speed error * \(\sigma_t\) = time error * \(\sigma_\theta\) = heading error (radians)
For multiple legs, errors combine:
Error Ellipse
Position uncertainty is often an ellipse, not a circle:
Where \(k\) determines probability: * \(k = 1.0\) → 39.4% (1-sigma ellipse) * \(k = 2.0\) → 86.5% (2-sigma ellipse) * \(k = 2.45\) → 95% confidence ellipse * \(k = 3.0\) → 98.9% (3-sigma ellipse)
Rotated Error Ellipse
When errors are correlated, the ellipse is rotated:
Where \(\rho\) is the correlation coefficient.
GPS Error Model
GPS position error follows a roughly Gaussian distribution:
| Source | Error (1σ) | Notes |
|---|---|---|
Ionospheric delay |
2-5 m |
Varies with solar activity |
Tropospheric delay |
0.5 m |
Weather dependent |
Satellite ephemeris |
2 m |
Broadcast vs precise |
Receiver noise |
0.3 m |
Hardware dependent |
Multipath |
0.5-2 m |
Environment dependent |
PDOP-weighted total |
3-10 m |
Civilian L1 C/A |
Position Dilution of Precision (PDOP):
Where UERE = User Equivalent Range Error (combined single-satellite error)
Coriolis Effect (Long-Range Ballistics)
For long-range artillery and sniper fire, the Coriolis effect deflects projectiles:
Horizontal Deflection
Simplified for horizontal fire:
Where: * \(\Omega\) = Earth’s angular velocity = 7.2921 × 10⁻⁵ rad/s * \(\phi\) = latitude * \(t\) = time of flight * \(v_h\) = horizontal velocity
Practical Rule (Sniper)
At 1000m, 45° latitude, typical deflection is 6-8 cm. At 2000m: 25-30 cm.
Northern Hemisphere: * Firing NORTH → bullet deflects RIGHT * Firing EAST → bullet deflects RIGHT * Firing SOUTH → bullet deflects RIGHT * Firing WEST → bullet deflects RIGHT
(All rightward due to Earth’s rotation; opposite in Southern Hemisphere)
Practical Calculations
CLI Implementation
haversine() {
# Calculate great-circle distance using Haversine formula
# Usage: haversine <lat1> <lon1> <lat2> <lon2> [unit]
# Units: km (default), m, nm, mi
# Example: haversine 33.9425 -118.4081 40.6413 -73.7781 km
local lat1=$1 lon1=$2 lat2=$3 lon2=$4 unit=${5:-km}
awk -v lat1="$lat1" -v lon1="$lon1" -v lat2="$lat2" -v lon2="$lon2" -v unit="$unit" '
BEGIN {
PI = 3.14159265358979323846
# Convert to radians
lat1_rad = lat1 * PI / 180
lon1_rad = lon1 * PI / 180
lat2_rad = lat2 * PI / 180
lon2_rad = lon2 * PI / 180
dlat = lat2_rad - lat1_rad
dlon = lon2_rad - lon1_rad
# Haversine formula
a = sin(dlat/2)^2 + cos(lat1_rad) * cos(lat2_rad) * sin(dlon/2)^2
c = 2 * atan2(sqrt(a), sqrt(1-a))
# Select radius based on unit
if (unit == "km") R = 6371.0088
else if (unit == "m") R = 6371008.8
else if (unit == "nm") R = 3440.065
else if (unit == "mi") R = 3958.761
else R = 6371.0088
distance = R * c
printf "%.4f\n", distance
}'
}
Python Implementation
"""
Convert geodetic coordinates to ECEF (Earth-Centered, Earth-Fixed)
Uses WGS84 ellipsoid parameters.
Formula:
N = a / sqrt(1 - e² * sin²(φ))
X = (N + h) * cos(φ) * cos(λ)
Y = (N + h) * cos(φ) * sin(λ)
Z = (N(1 - e²) + h) * sin(φ)
"""
lat_rad = math.radians(point.lat)
lon_rad = math.radians(point.lon)
sin_lat = math.sin(lat_rad)
cos_lat = math.cos(lat_rad)
sin_lon = math.sin(lon_rad)
cos_lon = math.cos(lon_rad)
# Radius of curvature in prime vertical
N = WGS84_A / math.sqrt(1 - WGS84_E2 * sin_lat ** 2)
x = (N + point.height) * cos_lat * cos_lon
y = (N + point.height) * cos_lat * sin_lon
z = (N * (1 - WGS84_E2) + point.height) * sin_lat
return ECEF(x, y, z)
def ecef_to_geographic(ecef: ECEF) -> GeoPoint:
"""
Convert ECEF to geodetic coordinates using Bowring's iterative method
"""
x, y, z = ecef.x, ecef.y, ecef.z
# Longitude is straightforward
lon = math.atan2(y, x)
# Iterative solution for latitude
p = math.sqrt(x**2 + y**2)
lat = math.atan2(z, p * (1 - WGS84_E2)) # Initial approximation
for _ in range(10): # Usually converges in 2-3 iterations
sin_lat = math.sin(lat)
N = WGS84_A / math.sqrt(1 - WGS84_E2 * sin_lat**2)
lat_new = math.atan2(z + WGS84_E2 * N * sin_lat, p)
Navigation Slide Rules
Before calculators, navigators used specialized slide rules.
Common Conversions
| Operation | Formula |
|---|---|
Nautical miles → km |
\(km = nm \times 1.852\) |
Statute miles → km |
\(km = mi \times 1.609\) |
Knots → km/h |
\(km/h = kts \times 1.852\) |
1° latitude |
≈ 111 km (60 nm) |
1° longitude at equator |
≈ 111 km |
1° longitude at lat φ |
\(\approx 111 \times \cos(\phi)\) km |
Exercises
Drill 1: Distance Calculations
Calculate the great circle distance between:
-
New York (40.7128°N, 74.0060°W) and London (51.5074°N, 0.1278°W)
-
Tokyo (35.6762°N, 139.6503°E) and Sydney (33.8688°S, 151.2093°E)
-
Los Angeles (34.0522°N, 118.2437°W) and Honolulu (21.3069°N, 157.8583°W)
Drill 2: Bearing Calculations
Calculate initial and final bearings:
-
From San Francisco to Tokyo
-
From London to New York
-
From Sydney to Santiago, Chile
Drill 3: Destination Point
Starting from your current location:
-
Travel 100 km on bearing 045° - where do you end up?
-
Travel 500 nm on bearing 270° - where do you end up?
-
Travel 1000 km on bearing 180° - where do you end up?
Quick Reference
| Formula | Use |
|---|---|
\(d = R \cdot \arccos(\sin\phi_1\sin\phi_2 + \cos\phi_1\cos\phi_2\cos\Delta\lambda)\) |
Spherical law of cosines (distance) |
Haversine (see above) |
Numerically stable distance |
\(\theta = \arctan2(\sin\Delta\lambda\cos\phi_2, \cos\phi_1\sin\phi_2 - \sin\phi_1\cos\phi_2\cos\Delta\lambda)\) |
Initial bearing |
\(1° lat \approx 111 km\) |
Quick latitude distance |
\(1° lon \approx 111 \cos(\phi) km\) |
Quick longitude distance |
Next
Continue to Time & Longitude to understand the relationship between time and position.