Harmony Institute researches the influence of entertainment. Our organization is founded on the premise that stories are powerful tools for creating social change. We’re interested in finding a meaningful understanding of how films, novels, television shows, and games affect their audiences and the metrics that matter for each medium. More than anecdote or assumption, impact assessment can provide clear, accurate answers to the hard, but fundamental question of what we hope to achieve when creating entertainment.
Movies, books, and television shows were, until recently, how most stories were told to mass audiences. Those mediums all move in one direction—they deliver a complete story to an audience. Gaming, on the other hand, is interactive; it’s a two-way street in which players have agency, or control, in the story. This participation can contribute to audiences’ increased learning or retention of information. The participatory nature of gaming and its ramifications for interactive data analysis promise exciting opportunities for measuring and understanding impact.
In the social sciences, there can be a tendency to treat games like traditional media and focus on the game as a whole unit by assigning it to corpus or genre, such as “violent” or “educational” based on a single dimension of play.
However, the interactive nature of games can offer insights into granular learning processes. Every level completed or executable player action can contribute to the measurement of that game’s effect on audiences. This approach is similar to recent economic studies on in-game behavior or the gaming industry’s method of playability testing, where game testers report on whether individual levels were an enjoyable experience or how well the mechanics of gameplay worked.
To explain this type of evaluation in more concrete terms, we’ve chosen three game examples that are each engaging in different ways. We’ll walk through the particular ways in which they can create change in an audience, as well as give examples of how we might measure those changes.
Did the player learn something?
We can learn something in almost any context. Our question is what makes games unique as learning vehicles? In the gaming world, this question is the focus of the “serious games” genre, which applies interactive design principles to express educational or social issues, although almost any game can engage these questions. For example, Dys4ia, an 8-bit style indie game that explores the difficult details of undergoing hormone-replacement therapy and transitioning from one gender to another. Players are exposed to multiple mini-games along a narrative filled with emotional cues and medical facts.
The game creator, Anna Anthropy, perfectly describes the game play: “This was a story about frustration – in what other form do people complain as much about being frustrated? A video game lets you set up goals for the player and make her fail to achieve them. A reader can’t fail a book. It’s an entirely different level of empathy.”
Context is one of the great challenges of social science research. It’s hard to measure social behavior in the confines of a lab or through a simple survey. However, games like Dys4ia provide a natural setting for understanding how players respond to the social messages embedded in games. How might we imagine adapting gameplay so that we could measure an audience’s understanding of the issues that transgender people face? A developer might be able to further the experience with an additional mini-game at the end of the formal game. This mini-game could present choices to the user that reveal his or her knowledge of past lessons. Another option might be to pinpoint a current section of the game and examine how users respond to the tasks by measuring click counts and cursor movement. By staying within the context of the game and using easily measurable data, we can embed evaluation directly into play.
Did the player feel something?
An alternative to the abstract, unfamiliar experience of Dys4ia are games with immersive storylines, strong simulation qualities, a visual aesthetic that mirrors the desired emotions, and a focus on character development. The 2012 third-person shooter Spec Ops: The Line embodies these qualities. The game is a triple-A title with high-quality action and an immersive gameplay environment. The storyline initially leads players through a seemingly standard war game, based in Dubai, and progressively introduces emotionally complex morality choices. Imagine experiencing firsthand the plot of Heart of Darkness or Apocalypse Now. While not explicitly about a social issue, the game has started a dialogue. Many reviewers agree that the game offers a commentary on war, or more specifically, the way audiences approach war video games.
From an evaluation perspective, Spec Ops has a highly affective quality and offers researchers the potential to measure player changes in attitudes about shooting, violence, and war. For example, an evaluation could include monitoring comments in game forums, or even player-created Spec Ops YouTube videos to determine which scenes players were discussing and whether these scenes drew on players’ skills in the game mechanics or their decisions in how to advance the narrative.
Another study might track individual participants as they progress through the game and note their actions at pivotal game benchmarks, such as whether the number of casualties decreases substantially over time, or whether highly-skilled players begin to waver from ‘good’ gameplay as the story becomes increasingly more. For example, do the players stand still in the game for longer stretches or does their movement become more erratic? Are they more likely to forget game commands or press buttons accidentally as the story line changes? All of these metrics could be used to assess the effects of the game’s morality on a player’s cognitive load, or the number of interactions his or her brain and working memory can process at one time.
Did the player do something?
Casual games have exploded in popularity and show that simple, addictive gameplay is a powerful mechanic for enticing players to take action in the real world. For traditional forms of media, we generally assess audience responses after they’ve interacted with the media. For games, we can also look at the actions they take during gameplay, such as spending money or points on character upgrades, or the number of specific types of clicks. Take, for example, the game Half the Sky Movement: The Game.
Half the Sky was released in March 2013 as the game component in a transmedia campaign of the same name seeking to address the issue of oppression of women around the world. It now has more than 950,000 players, who have triggered more than $126,000 in donations toward life-changing surgeries and the purchase of 205,000 books. Players are tasked with moving the main character, Radhika, through a difficult storyline. The game utilizes mini-game mechanics reminiscent of casual games like Bejeweled.
As a highly interactive game, Half the Sky lends itself to quantitative analytics, and the designers measured over 85 different response activities for players. These opportunities to respond are intentionally paired with actions in the game. For example, as seen in the image above, players can donate money for vaccinations in the real world after successfully helping Radhika get her child to a clinic.
These real-world player responses and donations are important outcomes and can also be used as a measure of player engagement. The pairing of response opportunities and plot points can be used to test the efficacy of the game’s story line, such as whether there are significant differences in players’ levels of response after controlling for items like time spent in game or the level reached.
This data can then be used to drive changes to the storyline in further releases of the game by illustrating to game designers which plots are engaging players. Designers can repeat this process multiple times. This method of iterative design is one of the core concepts behind data-driven storytelling.
However, data requires context. This type of evaluation incorporates the theory of causality in narrative, which states that moments of high drama in a narrative are also the best places to place informational or educational content. Accordingly, by experimenting with plot pairings and measuring players’ responses, game designers can work with writers to create new story lines or shift dialogue or plot structures based on what their audience finds most compelling.
And just who is that audience?
According to Games for Change (G4C), Half the Sky had “…much more success with people who already play Facebook games, regardless of their support of the cause.” Despite this element of self-selection among players, it is still possible to further evaluate player responses within the game. Facebook Graph Search allows researchers to slice up user psychographic data in useful ways.
For example, even though they may be a minority for Half the Sky, are players interested in social issues more or less likely to respond to prompts for real world donations? Facebook Graph also shows us that users who like The Fistula Foundation, a Half the Sky partner, also like the game pages of Tamagotchi Life and Furby, demonstrating a general interest in caretaking games. Are these players donating to the game, and if so, is it during a thematic point of caretaking?
The primary research question here is who is responding or donating? Are they already interested in the issue? This approach could be used to think creatively about the success of partners featured by the game.
Designing with evaluation principles in mind
Like any form of entertainment, a game cannot be all things to all people, which is why it’s important for game developers to design for their audiences, understand their audience goals, and be able to iterate with the help of evaluation metrics. However, even if a game developer decides not to undertake a formal assessment, it is useful to keep evaluation goals in mind, as a guiding development tool. For example, when designing a game to have a message, think about whether the factual information you’re trying to convey adds to the narrative or distracts from the player experience, or whether you can simplify that information to create a more immersive story. Ultimately, the identification of these goals may even help to clarify the purpose of a game and what it can evoke in its audience.
At their best, games are playful and interactive; they can realistically or fantastically simulate situations and events; they encourage strategic thinking and problem solving. Games have the potential to become an important force in our society. Accordingly, they demand rigorous, research-based analysis that reflects their unique properties and widespread appeal.