Multi-Armed Bandits: Algorithms, Applications, and Future Directions

Abstract

The Multi-Armed Bandit (MAB) problem is a fundamental framework in sequential decision-making. The main goal is to balance exploration of uncertain options with exploitation of known rewards. Originating from classical probability theory, MAB has evolved into a significant modern reinforcement learning. This problem can be applied to all walks of life in reality, from online advertising and recommendation systems to healthcare and finance. Over the years, algorithms such as Explore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS) have emerged as leading strategies, each offering unique trade-offs between exploration efficiency and theoretical regret bounds. This paper gives an overview of the history, main algorithms, applications, and challenges of MAB, and discusses possible future research directions. Furthermore, it highlights the practical implications of these algorithms for real-world decision-making scenarios.