We’re excited to share the positive results of a recent evaluation of the mobile phone game, 9 Minutes, that was produced as part of the “Half the Sky Movement”.
The game was developed for feature phones commonly used in India and East Africa. Learn more here. 9 Minutes plays out the adventure of pregnancy and rewards pregnant women and their spouses for keeping both mother-to-be and the baby inside her healthy and happy.
The evaluation shows measurable positive shifts in knowledge, attitudes, and behavioral intentions toward promoted safe pregnancy and delivery actions following exposure to the game.
Full evaluation is available here.
The report, published with support from USAID and PEPFAR, details the results of a C-Change project evaluation conducted in India. The study utilized a mixed-methods design with quantitative pre-/post-tests combined with qualitative focus group discussions. 608 women and 308 men participated in the study.
Study objectives included measuring how familiar safe pregnancy practices were prior to game exposure and whether or not knowledge, shifted perceptions of acceptability or intention to act was increased through game exposure.
For example, when participants were asked to name beneficial pregnancy activities, significant increases were made from pre- to post-test intervention. See chart of results below for a range of such activities:
Beneficial Activities for Pregnant Women
Games for Change worked with evaluation experts to define a theory of change around the game early on in development. Research on games’ impact is frequently unattainable, costly, or an afterthought. Yet, demonstrating effectiveness is an essential step for scaled adoption. As we recognize the importance of evaluation, we highly encourage G4C projects to consider this early in the development process.
Below are some “tips” on how we approached the evaluation for 9 Minutes and lessons we want to share.
1. Define Your Theory of Change Early
The impact goals of a game (e.g. learning, raising awareness, behavior change, real world action, fundraising, etc) can be defined at concept and early design stages. These goals could heavily impact the features included/excluded from a game, the game content/presentation format and the actions and choices you allow your player to make in the game. We recommend articulating the theory of change in a concise 2-3 page document, and considering support from an evaluation expert in that task. It is a living document that will be tweaked throughout the development process.
2. Consider an Evaluation Format that Compares Game to Other Media
The “9 Minutes” study randomly assigned participants to one of two intervention groups:
- •Group A participants were exposed only to the 9 Minutes game
- •Group B participants played the game, watched a related video, and participated in a brief facilitated small group discussion (the full intervention package).
Comparing the two groups, we were able to evaluate how effective the games are as a unique modality for impact and demonstrate positive outcomes from the game itself. We recommend evaluating games against other forms of media to better isolate and demonstrate game effectiveness.
3. Employ an External, Impartial Entity to Conduct Evaluations
As stakeholders intimately involved with the development and content creation for the game, we could not conduct the evaluation without skewing the results. We recommend working with an unbiased Monitoring and Evaluation company which can conduct the study.
4. Don’t be Afraid of Results
Despite some trepidation, we embraced the notion that the evaluation could have come back with negative results. During prior on-the-ground testing, and in early stages of development, we received mixed reactions to the game. While we are excited by the final results, we were prepared to take any evaluation feedback as lessons we can take into the next game project. Fearlessly pursue and embrace any opportunity for testing and evaluation – negative feedback can offer insight into areas to improve or features to reconsider.
Through funding made possible by the United States Agency for International Development (USAID), the FHI 360-managed C-Change project supported the development of the 9 Minutes mobile game and video on the topics of safe pregnancy and delivery in India.
The game (and two others in the series) was executive produced by Games for Change and Show of Force, developed by Mudlark Studios and published by e-Line Media. These products were part of the Half the Sky Movement, a multi-donor, multimedia initiative inspired by and in collaboration with Nicholas Kristof and Sheryl WuDunn, authors of the best-selling book, Half the Sky: Turning Oppression into Opportunity for Women Worldwide.
Learn more about the Half the Sky Movement here.