We explore an extension of nonatomic routing games that we call Markov decision process routing games where each agent chooses a transition policy between nodes in a network rather than a path from an origin node to a destination node, i.e. each agent in the population solves a Markov decision process rather than a shortest path problem. We define the appropriate version of a Wardrop equilibrium as well as a potential function for this game in the finite horizon (total reward) case. This work can be thought of as a routing- game-based formulation of continuous population stochastic games (mean-field games or anonymous sequential games). We apply our model to the problem of ridesharing drivers competing for customers.
