As the use of cancer immunotherapy expands, a personalized benefit-risk profile of these agents needs to be considered in special populations at higher risk of experiencing adverse events with these agents. Cancer patients with preexisting autoimmune conditions or those who experience rheumatic adverse events with immunotherapy are faced with difficult decisions regarding their treatment. Often, therapeutic gains in survival need to be balanced with a deleterious impact on quality of life. Risk consideration should include types of potential benefits and harms, magnitude of effects, alternative therapeutic options, and, most importantly, patient values and preferences. Future research should aim to quantify risks in these special populations to ensure evidence-based decision-making that considers patient trade-offs between quality of life and survival.