We explore an extension of nonatomic routing games that we call Markov decision process routing games where each agent chooses a transition policy between nodes in a network rather than a path from an origin node to a destination node, i.e. each agent in the population solves a Markov decision process rather than a shortest path problem. This type of game was first introduced in [1] in the finite-horizon total-cost case. Here we present the infinite-horizon average-cost case. We present the appropriate definition of a Wardrop equilibrium as well as a potential function program for finding the equilibrium. This work can be thought of as a routing-game-based formulation of continuous population stochastic games (mean-field games or anonymous sequential games). We apply our model to ridesharing drivers competing for fares in an urban area.
