In this article, we propose a new algorithm called COMA (Committee of Maximum Agents) for multi-agent reinforcement learning. COMA is designed to handle complex situations where multiple agents work together to achieve a common goal. Our main idea is to have each agent focus on maximizing its own contribution to the overall reward, rather than trying to maximize the global reward directly.
How COMA Works
COMA trains a critic for each agent to estimate the value of its actions. The critic uses a counterfactual baseline to compare the value of the agent’s action with the value of taking a default action (which is calculated as the policy-weighted average of all possible actions). The advantage function of COMA is then defined as the difference between the value of the agent’s action and the value of the default action.
COMA also uses an actor to learn the policy for each agent. The actor takes the advantage function of the critic as input and outputs a probability distribution over the possible actions. This distribution represents the agent’s policy for selecting actions based on the current state of the environment.
Learning Rates
In COMA, we use two different learning rates for the actor and the critic. The actor has higher learning rates to adapt quickly to changes in the environment, while the critic has lower learning rates to avoid overfitting to the agent’s policy.
Entropy Coefficient
The entropy coefficient is a hyperparameter that controls how exploratory the agent should be. We tune this parameter for each instance of COMA to find the optimal balance between exploitation and exploration.
Conclusion
In summary, COMA is a simple and efficient algorithm for multi-agent reinforcement learning. By focusing on maximizing individual contributions rather than directly maximizing global rewards, COMA can handle complex situations where multiple agents work together to achieve a common goal. Our experimental results show that COMA outperforms other state-of-the-art algorithms in simulated environments.
