To measure meaningful impact, beyond ratings and “likes”, requires a flexible, multi-faceted approach. Many projects use a blend of quantitative and qualitative assessment tools. Qualitative methodologies are gaining greater notice in recent years, as they can provide opportunities for more diverse and inclusive levels of assessment that might countermand the inherent biases of quantitative data. Impact can be assessed at many levels of direct distribution, such as individual, societal, organizational, as well as the analysis of discourse (online or offline) before and after the film is presented. Analyzing and understanding what the status quo thinking is around a film’s subject matter informs how a film can influence the ecosystem of advocacy efforts in conversations, social media, news, even speeches. Dr. Karen Hughes explains in her article Measuring the Impact of Viewing Wildlife, ” the best indicator of a program’s effectiveness is whether or not it increases (viewers’) commitment to, and involvement in, conservation actions.” More investment is not necessarily better. The key is a well-defined impact assessment strategy, implemented at the outset of the project, that has clear goals.
In addition to some better-known tools, we list some approaches that while are not typically used in film assessment, might be helpful in evaluating long-term media impact.
Tools and Methodologies
To evaluate the full impact of a film, it is necessary to look at all of the elements from the early development of the film idea, to contextual conversations in potential communities where the film might be shown, to the final product and how it performs during distribution and outreach. Assessment is not just in terms of numbers, but in reactions and actions. Impact is measurable against a current snapshot of the state of the conversation and the level of awareness of the topic subject of the film. Measuring impact is a fluid and flexible process, rather than a procedure, and evolves over time according to the goals set and the constant validation and reconsideration of them.
In this first phase of our research, we identified several measurement tools that could be useful for impact evaluation. Several of them are included in both Assessment of Social Issue Documentaries: Research Methods & Future Considerations , and The Impact Field Guide and Toolkit . Others used in educational arenas for training evaluation and instructional design. It can be useful to try different frameworks and combinations of tools.