Persistent activity, the maintenance of neural activation over short periods of time in cortical networks, is widely thought to underlie the cognitive function of working memory. A large body of modeling studies has reproduced this kind of activity using cell assemblies with strengthened synaptic connections. However, almost all of these studies have considered persistent activity within networks with homogeneous neurons and synapses, making it difficult to judge the validity of such model results for cortical dynamics, which is based on highly heterogeneous neurons. Here, we consider persistent activity in a detailed, strongly data-driven network model of the prefrontal cortex with heterogeneous neuron and synapse parameters. Surprisingly, persistent activity could not be reproduced in this model without incorporating further constraints. We identified three factors that prevent successful persistent activity: heterogeneity in the cell parameters of interneurons, heterogeneity in the parameters of short-term synaptic plasticity and heterogeneity in the synaptic weights. We also discovered a general dynamic mechanism that prevents persistent activity in the presence of heterogeneities, namely a gradual drop-out of cell assembly neurons out of a bistable regime as input variability increases. Based on this mechanism, we found that persistent activity is recovered if heterogeneity is compensated, e.g., by a homeostatic plasticity mechanism. Cell assemblies shaped in this way may be potentially targeted by distinct inputs or become more responsive to specific tuning or spectral properties. Finally, we show that persistent activity in the model is robust against external noise, but the compensation of heterogeneities may prevent the dynamic generation of intrinsic in vivo-like irregular activity. These results may help informing the ongoing debate about the neural basis of working memory.