💡 The paper introduces MatrAIx, a population-scale simulated-user evaluation infrastructure for testing AI systems and digital products with heterogeneous users. The problem addressed is that human evaluation of AI systems and digital products is costly, slow, and difficult to scale, while offline evaluations often abstract away human diversity and interactive behavior.

The MatrAIx infrastructure has three core components: Persona8B, which contains 8.3 billion persona records represented by 1290 categorical dimensions, with a quality-filtered core set of approximately 1 million personas comprising 599847 human-grounded and 400000 synthetic records. The MatrAIx Playground provides four environments in which diverse users evaluate and interact with digital products: Survey, AI Chatbot, Web, and App. MatrAIx also provides 1010 application tasks spanning more than 25 domains, including Commerce, Software, Finance, and Healthcare.

The persona agents were powered by three large language models: Claude Opus 4.8, GPT 5.5, and Claude Haiku 4.5. The results of 18189 evaluation trials across eight representative tasks showed that decisions and preferences vary across persona backgrounds, including hesitation after a price increase, willingness to continue after an AI assistant fails, and latency tolerance.

The paper also conducted two main validation studies. The first study, a 400-trial controlled study, evaluated persona adherence across ten behavioral attributes and all four environments, with declared behavior expressed or correctly suppressed in 366 trials, which is 91.5 percent. The second study had human and LLM judges evaluate the extraction quality of human-grounded personas.

Overall, MatrAIx provides an end-to-end infrastructure for evaluating AI systems and digital products with diverse simulated human users, addressing the need for scalable and realistic evaluation of AI systems and digital products.