Using Data to Measure Impact

By Eric Martin
Posted on February 27, 2018
Docscale india

How do you know if documentary film makes a difference in the world?  

If you’re a social scientist, you evaluate it.

That’s what the Aspen Planning and Evaluation Program did in one of the most extensive studies ever to look at the impact of documentary film in a global development setting (173 pages with attachments, for those counting).  The recently released study presents data and findings for Women and Girls Lead Global, a partnership between ITVS, USAID, and the Ford Foundation, with additional evaluation support from the Bill and Melinda Gates Foundation: 5 years, 5 countries, 34 films, 60 partners, 21 major objectives.  (You can skim the highlights in the executive summary.)

The full report is aimed at global development professionals, with its tables of data and references to behavior change and social norms.  But there are some big takeaways for filmmakers, too—especially the many documentary producers who work tirelessly on important social issues, lead ambitious engagement campaigns, and seek new ways to prove impact.

  1. Facilitated screenings change the way people see the world.  The facilitated screenings at the heart of the Women and Girls Lead Global program helped move the dial on dozens of measures related to how people see the world, themselves and others.  A consistent range of 15-30 percentage points (which is a lot!) was found across multiple issue and countries.  One standout example: the number of young men who said they would intervene in sexual harassment more than doubled after screening and discussing three ITVS documentaries about female empowerment.

  1. Real change depends on the partner—but can be very powerful.  In the 280 schools studied in Bangladesh where the project worked closely with the Ministry of Education, both child marriage and dropout rates dropped from roughly one in twenty to one in a hundred—or an estimated 1500 fewer child brides and/or teenage dropouts.  Big institutional partners may be the hardest to bring on board, but they proved the most impactful to work with.

  1. New tools are needed to measure the impact of “art.”  Major studies like the report from Aspen are not feasible for most documentary filmmakers or organizations. With that in mind,  ITVS developed and tested a prototype of DocSCALE, a new platform that could one day help filmmakers collect quick, cheaper, and better data—and do it in a way that’s more participatory with audiences and gathers insights in their own words.  You can read more about the DocSCALE platform in this white paper or a more layman’s view in this recent article at Stanford Social Innovation Review

The study is certainly not the final word on the impact of documentary film, and more studies are hopefully coming.  But the next time a funder asks a filmmaker how we know that documentaries work, this report should be shared with them as a powerful piece of an age-old puzzle: proving the real-world impact of film.

From our blog

  1. Let's Talk Industry Sustainability at ITVS Independents Summit

    July 11, 2019

    Congratulations to the ITVS Independent Summit participants. We are thrilled to announce the visionary filmmakers who will be attending this summer's ITVS Independents Summit.Ryah Aqel Phil Bertelsen Margaret ByrneChrista CassanoLauren Cioffi Chico ColvardReilly Dowd Alejandro FloresRobie FloresReed HarknessSu KimDaresha KyiMarianela

  2. Sugar Shakers, Handbags, and Resistance: Screaming Queens: The Riot at Compton’s Cafeteria

    June 21, 2019

    Placards are pieces of history. They show us verve. They show us wit. They show us pain and personality. They are floating, gripped, vital statements of resistance, rendering the individual voices of protest visible. The protests and riots of our times are documented and extended through social media, creating long-lasting visual histories that live

  3. Visualizing the Modern Digital Age

    June 12, 2019

    In the land of memes, baby photos, and selfies, social media can serve many purposes for users. Social media not only connects friends and families who are wide distances apart but also allows users to author their own stories. The Instagram grid can be an archive of personal growth, while Facebook videos can narrate lived experiences.  For many