Co-Evolution in the Successful Learning of Backgammon Strategy

Analysis

Why This Paper Matters

This paper is a landmark in the history of AI and machine learning, particularly for game-playing and evolutionary computation. At a time when Tesauro's TD-Gammon had set the standard for reinforcement learning in backgammon, Pollack and Blair demonstrated that a much simpler method—hillclimbing with relative fitness—could achieve comparable results. This was surprising because hillclimbing had been largely dismissed as a weak optimization method, especially for high-dimensional neural network weights. The paper's significance lies in its challenge to the prevailing orthodoxy that sophisticated gradient-based learning is necessary for complex tasks. It opened the door to re-evaluating simple evolutionary methods in domains with favorable dynamics.

The paper also provides a deep analysis of why the method works, focusing on the concept of a "meta-game" of self-learning. The stochastic nature of backgammon (dice rolls) prevents the champion from becoming trapped in suboptimal equilibria, as mutations can occasionally produce challengers that exploit weaknesses. This insight has broader implications for co-evolutionary algorithms and the study of learning dynamics in noisy environments.

Technical Contributions

• Hillclimbing with relative fitness: Instead of using TD learning or backpropagation, the authors use a simple tournament between a champion and a mutated challenger. The challenger replaces the champion only if it wins a match, ensuring monotonic improvement in relative performance.
• Mutation-based exploration: The challenger is created by adding small random perturbations to the champion's weights. This simple exploration mechanism is sufficient due to the stochasticity of the game.
• Domain analysis: The authors identify that the success of hillclimbing depends on the "meta-game" dynamics—specifically, that the evaluation function's performance is measured relative to opponents, and that the stochastic nature of backgammon prevents the champion from becoming a fixed point that cannot be improved.
• Feedforward architecture: A 4,000 parameter network is used, which is modest by modern standards but was competitive at the time. The network takes board features as input and outputs a scalar evaluation.

Results

The paper reports that the hillclimbing approach produces a backgammon player that is competitive with TD-Gammon, which used temporal difference learning. The authors do not provide exact win rates against TD-Gammon in the abstract, but they state the method "worked rather well" and produced a strong evaluation function. The key result is that the method avoids suboptimal equilibria, a common failure mode for self-play learning. The paper's analysis shows that the relative fitness evaluation, combined with the stochasticity of dice rolls, allows the hillclimbing process to escape local optima that would trap a purely self-play TD learner.

Significance

This paper has had a lasting impact on the field of evolutionary computation and reinforcement learning. It demonstrated that simple methods can be surprisingly effective when the domain has the right properties, and it provided a theoretical framework for understanding when co-evolutionary learning succeeds. The work influenced later research on co-evolution in games, neuroevolution (e.g., NEAT), and the use of relative fitness in training neural networks. It also serves as a cautionary tale against assuming that more complex algorithms are always necessary. The paper remains a classic example of how domain-specific dynamics can enable simple algorithms to achieve state-of-the-art performance.