When combining the different elements of AI in Games we can create robust simulations and real games. In the Total War series of video games, the diplomacy AI is used to simulate the behavior of non-player factions in the game world, allowing the player to negotiate with and form alliances or go to war with other factions. The diplomacy AI in Total War games uses a combination of rule-based and decision tree approaches to simulate the behavior of the other factions. The AI evaluates its relationship with the player and other factions based on a variety of factors, such as current military strength, shared enemies or allies, and past actions. The diplomacy AI also takes into account the faction’s long-term strategic goals and will sometimes make decisions that prioritize those goals over short-term gains. For example, a faction may be more willing to form an alliance with the player if they are facing a common enemy, but may be less willing to do so if they believe the player’s long-term goals are in conflict with their own.
In addition to the rule-based and decision tree approaches, the diplomacy AI in Total War games also uses machine learning techniques to improve its performance over time. Specifically, the AI is trained using reinforcement learning to learn which decisions lead to successful outcomes in different situations. The use of diplomacy AI in Total War games helps to create a more immersive and engaging game experience for players, as they can negotiate with other factions and form alliances or go to war in a way that feels realistic and challenging. The AI also adds an element of unpredictability to the game, as the player must constantly adapt to the changing behavior of the other factions.
There are several machine learning techniques used for diplomacy in the Total War games, including reinforcement learning and decision trees.
Reinforcement learning is a type of machine learning that involves training an agent to take actions that lead to a specific goal, based on a reward signal. In the case of diplomacy in Total War games, the goal is often to maximize the faction’s overall power and influence, while avoiding unnecessary wars or conflicts. The AI is trained using reinforcement learning by simulating many different scenarios and evaluating which decisions lead to the best outcomes. Over time, the AI learns which decisions are most likely to result in positive outcomes and adjusts its behavior accordingly.
Decision trees are another machine learning technique commonly used for diplomacy in Total War games. A decision tree is a graphical representation of a series of decisions that must be made in order to achieve a specific outcome. In the context of diplomacy in Total War games, decision trees can be used to model the behavior of different factions and to determine how they are likely to respond to different player actions. The AI can then use this information to make more informed decisions about whether to form alliances or go to war with other factions.
Other machine learning techniques that may be used for diplomacy in Total War games include Bayesian networks and neural networks. Bayesian networks are probabilistic models that use probability distributions to model relationships between variables, while neural networks are computational models that simulate the behavior of biological neurons. Both of these techniques can be used to analyze complex relationships between different variables and make predictions about how the other factions are likely to behave in different situations. In the game world, there is one game that is known for its crafting of a Diplomacy Artificial Intelligence. In the following case study we will look at the Diplomacy AI of ‘Total War’.
Diplomacy and AI go further back then game development. An active area of research is optimizing diplomacy bots in the serious game ‘Diplomacy’:
The main community on the internet that focuses on writing AI bots that play Diplomacy is the Diplomacy AI development Environment (DAIDE)…. Also they have developed a language for communication between the bots. (De Jonge, 6) [http://www.daide.org.uk]
It is simpler to create a Diplomacy AI or bot in a game than the real world. The following section is based on the case study of Tommy Thompson (Thompson 2018b). Total War came out in 2000 with the release of Shogun Total War. It had three main areas of AI: Diplomacy, which handles troop and resources, establishes unit troop formations, and it’s state management uses Genetic Algorithms; Combat which was campaign management, strategy for attack and defense; Unit Manager that collects unit types, manages combat, uses an Artificial Neural Network for managing individual units (i.e. Calvary). The AI had integrated into it a knowledge base from Sun Tsu’s Art of War. For an example rule base or maxims:
-when you surround an army, leave an outlet free
-on open ground, do not try to block the enemy’s way
-if you outnumber the enemy 10:1 surround them
-if you outnumber the enemy 5:1, attack them directly
-if you outnumber the enemy 2:1 divide them
The use of Genetic Algorithms created tactics for each State of Feudal Japan in war with each other. With some states being more aggressive and others more diplomatic. This created variance in the different feudal states and complex negotiations in Diplomacy.
Total War had subsequent releases: Medieval Total War 2002, Foundations of Shogun Total War, Total War Rome 2004. Also, in 2004 the franchise opened the ability for fans to modify the game, by using an API to write their own scripts for game play. With the release of Total War Empire 2009 an expanded game was presented that added to warfare the need to tax or raise finances to fight and naval battles. In 2009 the core AI was redeveloped, and it adopted GOAP for state management. As one game designer that worked on the AI notes:
This AI is not like any other we have written… by far the most complex code edifice I’ve ever seen in a game. I wrote much of the campaign AI for Shogun and Medieval… And I know that even quite simple ‘static’ evaluate-act AI’s with no plans or memory can be complex enough to exhibit chaotic behavior (we’re talking about mathematical ‘butterfly effect’ style chaos here). It does what it does, and it’s not quite what you intended. This can be a good thing- you cull out the bad behaviors and are left with just what is good, and with a simple system that’s not too predictable.
…The empire AI is way more complicated than any of our previous product… The net result is an AI that plans furiously and brilliantly and long term, but disagrees with itself chronically and often ends up paralyzed by indecision“
(Mike Simpson, Blog the Second, Total War Blog, Oct 9, 2009) [emphasis added]
As we can see from an algorithmic complexity perspective even a self-contained universe of a game managed by AI can lead to disagreement and paralyzed by indecision, not to mention bad decisions.
The campaign AI (Artificial Intelligence) in the Total War series of video games is responsible for controlling the non-player factions and armies in the game’s campaign mode. It is designed to simulate the decision-making processes of human players, and to provide a challenging and dynamic opponent for the player to face. The campaign AI in Total War games takes into account a variety of factors when making decisions, such as the strength and position of enemy armies, the economic situation of each faction, and the diplomatic relationships between factions. It uses this information to make strategic decisions about where to move armies, which territories to conquer, and which factions to ally with or declare war on. In addition, the campaign AI in Total War games is designed to adapt to the player’s actions and strategies, creating a dynamic and challenging experience. For example, if the player is winning too easily, the campaign AI may increase the difficulty by forming alliances with other factions or by launching surprise attacks on the player’s territories.
The Campaign AI allowed for interactions with players, in negotiations they asked 3 key questions:
-how well am I doing right now?
-what can I do next?
-what resources can I allocate to that?
It also used the previously noted Belief, Desire, Intention (BDI) system. Which was used for evaluating offers, etc. there are some game mechanics that can be used to limit the desires of the player, such as resource scarcity or game rules that restrict certain actions. For example, in a game where players are competing for resources, an AI opponent might actively try to control certain resource-rich areas to limit the player’s access to those resources. Another approach that game developers might use to limit player desires is by providing moral or ethical considerations that impact the game world. For example, in a game where the player is leading a nation or empire, the AI might limit certain actions that would be considered unethical or immoral in the context of the game world. This can be achieved by incorporating moral or ethical systems into the AI’s decision-making process, influencing a leader to be overly concerned in a game world with being nice, etc.
The Campaign AI was divided into sub-AIs:
-Financial System
-Construction System
-Diplomacy System
-Task Management System
-Technology Management
-Character Management
The sub-AIs, each responsible for managing a specific aspect of the game. Here’s a brief overview of what each of these sub-AIs is responsible for:
Financial System: This sub-AI manages the economy of the faction controlled by the AI. It is responsible for making decisions about what buildings to construct, what units to recruit, and how to allocate resources in order to maximize the faction’s income and overall financial stability.
Construction System: This sub-AI is responsible for managing the construction and development of buildings and infrastructure within the faction’s territories. It makes decisions about what buildings to construct, where to build them, and how to optimize the faction’s infrastructure to support its military and economic goals.
Diplomacy System: This sub-AI manages the faction’s diplomatic relationships with other factions in the game. It makes decisions about when to form alliances, when to declare war, and how to negotiate treaties and trade agreements in order to advance the faction’s strategic goals.
Task Management System: This sub-AI is responsible for managing the various tasks and actions that the faction’s armies and agents can undertake within the game. It makes decisions about where to move armies, which territories to attack or defend, and how to best utilize the faction’s agents (such as spies and diplomats) in order to achieve the faction’s goals.
Technology Management: This sub-AI manages the faction’s research and development of new technologies and advancements. It makes decisions about what technologies to research, how to allocate resources to research efforts, and how to utilize technological advancements to gain an advantage over other factions.
Character Management: This sub-AI is responsible for managing the faction’s characters, such as generals, governors, and other notable figures. It makes decisions about how to allocate skill points, how to equip characters with weapons and armor, and how to utilize characters in battle and diplomacy.
By dividing the Campaign AI into these sub-AIs, the Total War series is able to create a sophisticated and dynamic system that simulates the decision-making processes of a human player, while also providing a challenging and engaging opponent for the player to face
With the release of Total War Rome2 a new decision manager was introduced which used Monte Carlo Tree Search techniques (MCTS). Along with introducing diplomacy ‘personalities’, AI Agents negotiating with other AI Agents and human players. These personalities were collections of probabilities that influence decisions for budgeting, diplomacy evaluations, negotiations between factions and technology research.
As Tommy Thompson notes regarding some of their tactics of these personalities can lead to extreme and binary results, where burn it all down strategies dominate toward the end of the game, when the Campaign AI is in an unwinnable position. A mathematical analysis of why it develops a ‘scorched earth’ approach, also exhibited by Nazis in World War 2, would be an insightful endeavor.
In the context of video game AI, Monte Carlo Tree Search (MCTS) is a technique used for decision-making that is particularly well-suited for games with complex, branching decision trees. Total War Rome 2 is one such game, as it involves a large number of possible decisions that the AI must make in order to manage its faction and compete against the player.
MCTS works by simulating many possible outcomes of a given decision, using a process called “rollout”. For each possible decision, the AI simulates a number of future game states, each representing a possible outcome of the decision. It then evaluates the quality of each outcome, based on some predefined criteria, such as the likelihood of winning a battle or achieving a strategic objective. Finally, it selects the decision that has the highest expected value, based on the quality of the simulated outcomes. In the context of Total War Rome 2, the new decision manager using MCTS is able to evaluate many possible outcomes of a given decision in a relatively short amount of time. This allows the AI to make more sophisticated and strategic decisions, based on a deeper understanding of the potential consequences of each option.
Implementation of Monte Carlo Tree Search (MCTS) in Python, for a simplified game with two possible actions:
It is simpler to create a Diplomacy AI or bot in a game than the real world. The following section is based on the case study of Tommy Thompson (Thompson 2018b). Total War came out in 2000 with the release of Shogun Total War. It had three main areas of AI: Diplomacy, which handles troop and resources, establishes unit troop formations, and it’s state management uses Genetic Algorithms; Combat which was campaign management, strategy for attack and defense; Unit Manager that collects unit types, manages combat, uses an Artificial Neural Network for managing individual units (i.e. Calvary). The AI had integrated into it a knowledge base from Sun Tsu’s Art of War. For an example rule base or maxims:
-when you surround an army, leave an outlet free
-on open ground, do not try to block the enemy's way
-if you outnumber the enemy 10:1 surround them
-if you outnumber the enemy 5:1, attack them directly
-if you outnumber the enemy 2:1 divide them
The use of Genetic Algorithms created tactics for each State of Feudal Japan in war with each other. With some states being more aggressive and others more diplomatic. This created variance in the different feudal states and complex negotiations in Diplomacy.
Total War had subsequent releases: Medieval Total War 2002, Foundations of Shogun Total War, Total War Rome 2004. Also, in 2004 the franchise opened the ability for fans to modify the game, by using an API to write their own scripts for game play. With the release of Total War Empire 2009 an expanded game was presented that added to warfare the need to tax or raise finances to fight and naval battles. In 2009 the core AI was redeveloped, and it adopted GOAP for state management. As one game designer that worked on the AI notes:
This AI is not like any other we have written... by far the most complex code edifice I’ve ever seen in a game. I wrote much of the campaign AI for Shogun and Medieval... And I know that even quite simple 'static' evaluate-act AI's with no plans or memory can be complex enough to exhibit chaotic behavior (we're talking about mathematical 'butterfly effect' style chaos here). It does what it does, and it's not quite what you intended. This can be a good thing- you cull out the bad behaviors and are left with just what is good, and with a simple system that's not too predictable.
...The empire AI is way more complicated than any of our previous product... The net result is an AI that plans furiously and brilliantly and long term, but disagrees with itself chronically and often ends up paralyzed by indecision"
(Mike Simpson, Blog the Second, Total War Blog, Oct 9, 2009) [emphasis added]
As we can see from an algorithmic complexity perspective even a self-contained universe of a game managed by AI can lead to disagreement and paralyzed by indecision, not to mention bad decisions.
The campaign AI (Artificial Intelligence) in the Total War series of video games is responsible for controlling the non-player factions and armies in the game's campaign mode. It is designed to simulate the decision-making processes of human players, and to provide a challenging and dynamic opponent for the player to face. The campaign AI in Total War games takes into account a variety of factors when making decisions, such as the strength and position of enemy armies, the economic situation of each faction, and the diplomatic relationships between factions. It uses this information to make strategic decisions about where to move armies, which territories to conquer, and which factions to ally with or declare war on. In addition, the campaign AI in Total War games is designed to adapt to the player's actions and strategies, creating a dynamic and challenging experience. For example, if the player is winning too easily, the campaign AI may increase the difficulty by forming alliances with other factions or by launching surprise attacks on the player's territories.
The Campaign AI allowed for interactions with players, in negotiations they asked 3 key questions:
-how well am I doing right now?
-what can I do next?
-what resources can I allocate to that?
It also used the previously noted Belief, Desire, Intention (BDI) system. Which was used for evaluating offers, etc. there are some game mechanics that can be used to limit the desires of the player, such as resource scarcity or game rules that restrict certain actions. For example, in a game where players are competing for resources, an AI opponent might actively try to control certain resource-rich areas to limit the player's access to those resources. Another approach that game developers might use to limit player desires is by providing moral or ethical considerations that impact the game world. For example, in a game where the player is leading a nation or empire, the AI might limit certain actions that would be considered unethical or immoral in the context of the game world. This can be achieved by incorporating moral or ethical systems into the AI's decision-making process, influencing a leader to be overly concerned in a game world with being nice, etc.
The Campaign AI was divided into sub-AIs:
-Financial System
-Construction System
-Diplomacy System
-Task Management System
-Technology Management
-Character Management
The sub-AIs, each responsible for managing a specific aspect of the game. Here's a brief overview of what each of these sub-AIs is responsible for:
Financial System: This sub-AI manages the economy of the faction controlled by the AI. It is responsible for making decisions about what buildings to construct, what units to recruit, and how to allocate resources in order to maximize the faction's income and overall financial stability.
Construction System: This sub-AI is responsible for managing the construction and development of buildings and infrastructure within the faction's territories. It makes decisions about what buildings to construct, where to build them, and how to optimize the faction's infrastructure to support its military and economic goals.
Diplomacy System: This sub-AI manages the faction's diplomatic relationships with other factions in the game. It makes decisions about when to form alliances, when to declare war, and how to negotiate treaties and trade agreements in order to advance the faction's strategic goals.
Task Management System: This sub-AI is responsible for managing the various tasks and actions that the faction's armies and agents can undertake within the game. It makes decisions about where to move armies, which territories to attack or defend, and how to best utilize the faction's agents (such as spies and diplomats) in order to achieve the faction's goals.
Technology Management: This sub-AI manages the faction's research and development of new technologies and advancements. It makes decisions about what technologies to research, how to allocate resources to research efforts, and how to utilize technological advancements to gain an advantage over other factions.
Character Management: This sub-AI is responsible for managing the faction's characters, such as generals, governors, and other notable figures. It makes decisions about how to allocate skill points, how to equip characters with weapons and armor, and how to utilize characters in battle and diplomacy.
By dividing the Campaign AI into these sub-AIs, the Total War series is able to create a sophisticated and dynamic system that simulates the decision-making processes of a human player, while also providing a challenging and engaging opponent for the player to face
With the release of Total War Rome2 a new decision manager was introduced which used Monte Carlo Tree Search techniques (MCTS). Along with introducing diplomacy ‘personalities’, AI Agents negotiating with other AI Agents and human players. These personalities were collections of probabilities that influence decisions for budgeting, diplomacy evaluations, negotiations between factions and technology research.
As Tommy Thompson notes regarding some of their tactics of these personalities can lead to extreme and binary results, where burn it all down strategies dominate toward the end of the game, when the Campaign AI is in an unwinnable position. A mathematical analysis of why it develops a ‘scorched earth’ approach, also exhibited by Nazis in World War 2, would be an insightful endeavor.
In the context of video game AI, Monte Carlo Tree Search (MCTS) is a technique used for decision-making that is particularly well-suited for games with complex, branching decision trees. Total War Rome 2 is one such game, as it involves a large number of possible decisions that the AI must make in order to manage its faction and compete against the player.
MCTS works by simulating many possible outcomes of a given decision, using a process called "rollout". For each possible decision, the AI simulates a number of future game states, each representing a possible outcome of the decision. It then evaluates the quality of each outcome, based on some predefined criteria, such as the likelihood of winning a battle or achieving a strategic objective. Finally, it selects the decision that has the highest expected value, based on the quality of the simulated outcomes. In the context of Total War Rome 2, the new decision manager using MCTS is able to evaluate many possible outcomes of a given decision in a relatively short amount of time. This allows the AI to make more sophisticated and strategic decisions, based on a deeper understanding of the potential consequences of each option.
Implementation of Monte Carlo Tree Search (MCTS) in Python, for a simplified game with two possible actions:
import random
class Node:
def __init__(self, action=None, parent=None):
self.action = action
self.parent = parent
self.visits = 0
self.score = 0
self.children = []
def add_child(self, action):
child = Node(action=action, parent=self)
self.children.append(child)
return child
class MCTS:
def __init__(self, max_iterations=1000):
self.max_iterations = max_iterations
def search(self, root_node):
for i in range(self.max_iterations):
node = self.select(root_node)
reward = self.rollout(node)
self.backpropagate(node, reward)
return self.best_child(root_node)
def select(self, node):
while node.children:
node = self.best_uct(node)
return node
def best_uct(self, node):
UCT_CONSTANT = 1.0
best_score = float("-inf")
best_child = None
for child in node.children:
score = child.score / child.visits + UCT_CONSTANT * math.sqrt(
math.log(node.visits) / child.visits
)
if score > best_score:
best_score = score
best_child = child
return best_child
def rollout(self, node):
if random.random() > 0.5:
return 1
else:
return 0
def backpropagate(self, node, reward):
while node is not None:
node.visits += 1
node.score += reward
node = node.parent
def best_child(self, node):
best_score = float("-inf")
best_child = None
for child in node.children:
score = child.score / child.visits
if score > best_score:
best_score = score
best_child = child
return best_child.action
# Create the root node of the game tree
root_node = Node()
# Create the MCTS object and perform a search
mcts = MCTS()
best_action = mcts.search(root_node)
print("Best action:", best_action)
In this example, the Node class represents a single node in the game tree, and the MCTS class performs the search. The select, rollout, and backpropagate methods are the core components of the MCTS algorithm, and are used to select a node to expand, simulate a possible outcome, and update the scores of the nodes along the path from the selected node to the root, respectively.
This is just a very simple example, and in practice, the implementation of MCTS can become much more complex, depending on the nature of the game being played and the specific goals of the AI system. But I hope this helps to give you a sense of how MCTS can be used in game AI.
Leave a Reply