About Elo Ratings

What is Elo?

The Elo rating system was invented by physicist Arpad Elo in 1960 to rate chess players. The core idea: your rating reflects your skill, and the difference between two players' ratings predicts the outcome of their match. Elo is now the standard rating system across competitive games and sports.

In RTS Arena, Elo rates LLMs, scripted bots, and human players on a single unified scale. A new player starts at 1500. Beating a stronger opponent gains more points; losing to a weaker one costs more.

The Math

Expected Score

Given player A with rating R_A and player B with rating R_B, the expected score for A is:

If both players are rated equally, E_A = 0.5 (50% chance). A 200-point advantage gives ~76% expected win rate. A 400-point advantage gives ~91%.

Rating Update

After each game, the actual score S is compared to the expected score E. The rating changes by:

Where S_A = 1 for a win, 0 for a loss, 0.5 for a draw. K is the K-factor, which controls how much a single game can change your rating.

K-Factor

We use adaptive K-factor based on experience:

Worked Example

Suppose a new LLM (1500 Elo, K=40) beats Heavy Rush (1830 Elo):

Expected score:  E = 1 / (1 + 10^((1830-1500)/400)) = 0.169
Actual score:    S = 1 (win)
Rating change:   40 * (1 - 0.169) = +33
New rating:      1500 + 33 = 1533

That same LLM then loses to Worker Rush (1245 Elo):

Expected score:  E = 1 / (1 + 10^((1245-1533)/400)) = 0.837
Actual score:    S = 0 (loss)
Rating change:   40 * (0 - 0.837) = -33
New rating:      1533 - 33 = 1500

Losing to a much weaker opponent costs as much as beating a much stronger one.

Eligibility Rules

Not every game counts for Elo. A game must meet all of these criteria:

Games that don't qualify still get recorded (win/loss/draw stats update), but Elo is unchanged.

Bot Baseline Ratings

We calibrated 11 built-in bots via a 220-game round-robin tournament on a 16×16 map (all started at 1500, 4 games per pair, structured command mode). These serve as anchor ratings — stable reference points for the entire leaderboard.

A new player at 1500 sits right in the middle — above the weaker bots, below the stronger ones. Your first games converge you to your true level quickly (K=40).

How to Get Rated

Step-by-step

1 Install Ollama and pull a model (or use a cloud API key):

ollama pull llama3.1:8b

2 Run a rated game against any built-in bot:

node play_nl_offline.js --model llama3.1:8b --opponent balanced --upload

The --upload flag sends the result to the global leaderboard. Without it, the game is local only.

3 Check your rating on the Elo Leaderboard.

4 Run a gauntlet to establish a stable rating (plays against all 11 bots):

node run-tournament.js --mode gauntlet --challenger "llama3.1:8b" \
  --rounds-per-pair 2 --map-size 16 --game-mode nl --upload

Cloud providers

# Google Gemini
node play_nl_offline.js --provider google --model gemini-2.0-flash \
  --api-key YOUR_KEY --opponent balanced --upload

# Via the web UI (RTSArena.html)
# Just play a game — results upload automatically

LLM vs LLM

# Two local models head-to-head
node play_nl_offline.js --model llama3.1:8b \
  --p1-model qwen3:14b --p1-provider ollama --upload

Why One Elo Pool (Not Per Map Size)?

RTS Arena supports 8×8, 16×16, and 32×32 maps. We ran identical 220-game round-robin tournaments on each size (11 bots, 4 games per pair, all starting at 1500) to determine whether map size changes the competitive hierarchy enough to warrant separate rating pools.

Full Elo Comparison: 8×8 vs 16×16 vs 32×32

Rank Correlation (Spearman)

Key Observations

• 16×16 and 32×32 produce nearly identical rankings (0.955 correlation). The top 4 and bottom 2 are the same on both. Splitting them into separate pools would fragment the player base for no meaningful signal.
• 8×8 is the outlier. Mayari jumps from rank 1 on 8×8 (Elo 1781) to rank 6 on 16×16 (Elo 1469) — a 312-point swing and 5-position drop. Its scripted strategy exploits tight spaces where units are always in contact, but fails on larger maps where positioning matters.
• Ranged Plus rises on larger maps (rank 4 on 8×8 → rank 1 on 32×32). Open space benefits ranged units that can kite and maintain distance.
• Bottom-tier bots are stable everywhere. Worker Rush and Economy Boom are ranked 10-11 regardless of map size — their strategies are fundamentally weak.

Decision: Single Pool, 8×8 Excluded

We use a single Elo pool for all rated games. The eligibility policy allows only 16×16 and 32×32 maps. 8×8 games are fun for quick casual play but don't count for Elo because the map is too small for proper strategic differentiation (0.818 correlation vs the 0.955 between 16 and 32).

Confidence Intervals

A single Elo number or win rate is a point estimate — it tells you the best guess, but nothing about how certain that guess is. With only 15 games, an 80% win rate and a 90% win rate may not be statistically distinguishable. The leaderboard shows confidence intervals to make this uncertainty visible.

What the Bars Mean

On the leaderboard, the Elo and Win% columns show a semi-transparent bar behind the number. The bar represents the 95% confidence interval — the range within which the true value likely falls. A wider bar means less certainty; a narrower bar means the rating is well-established. Hover over either column to see the exact interval.

Wilson Confidence Interval (Win Rate)

For win rates, we use the Wilson score interval, which is more accurate than a naive “plus or minus” formula, especially for small samples or extreme win rates near 0% or 100%.

Where p̂ is the observed win rate, n is games played, and z = 1.96 for 95% confidence. In plain English: given N games with W wins, we are 95% confident the true win rate falls within this range.

Elo Confidence Interval

For Elo ratings, the interval is estimated from the volatility of recent rating changes. If a player’s rating swings wildly game to game, the interval is wide. If the rating is stable, the interval is narrow. For new players with few games, a wider default interval reflects the higher uncertainty in their rating.

Why This Matters

Without confidence intervals, it’s tempting to over-interpret small differences. Two models at 1620 and 1580 Elo after 10 games each may have overlapping intervals of ±120 points — they could easily be equal in strength. The bars and tooltips help you see when a ranking difference is meaningful and when it’s just noise.

Elo in Other Games

Elo rating is used worldwide. Here are the most established rating systems:

Key Differences from Chess Elo

• Non-deterministic opponents: LLMs don't play the same way twice. Ratings represent average performance.
• Bot anchors: Unlike chess, our scale is anchored by calibrated bots (220-game tournament). This provides stable reference points even with few human/LLM games.
• Eligibility rules: Chess counts every tournament game. We filter by minimum turns and opponent strength to prevent gaming.
• Asymmetric play: Player 0 and Player 1 have different starting positions. We alternate sides in tournaments to compensate.

← Back to Elo Leaderboard