Thinking Globally to Evaluate Social Programs

Permission to use image granted by Michael Quinn Patton (2023). Illustration is from Australian Simon Kneebone, Australian Evaluation Society 2017 conference on Evaluating Transformation and from the book Blue Marble Evaluation.

GBLD 538: Evaluation in a Global Context is an innovative elective course in the MA in Global Leadership Program, Royal Roads University. The course builds on work that the graduate level students do to become global systems leaders to support initiatives such as those described by the UN Sustainable Development Goals (SDGs).

Global Thinking

Osland, Bird and Mendenhall (2012) emphasize four processes of global thinking: 

1. Collaborating: working with others in relationships characterized by community, flexibility, respect, trust and mutual accountability.
2. Discovering: transformational processes leading to new ways of seeing and acting which, in turn, will lead to the creation of new knowledge, actions and things.
3. Architecting: the mindful design of processes that will align, balance and synchronize organizational behaviour.
4. Systems thinking: seeing and/or discovering the interrelationship among components and levels in a complex system and anticipating the consequences of changes in and to the system. (p.220)

When applied to the world of intervention programs addressing social problems, global thinking extends beyond localized systems thinking to macro level thinking that puts a program and its evaluation as to its efficacy into the context of the multiple economic, political, and social drivers that influence the program and all its players. Global thinking involves understanding the intersectionality of client needs, divergent interests of institutional agents, the range of actual and potential interventions, and potential outcomes relevant to improved lives.  Global thinking includes implementing evaluation using action learning cycles so that change can happen within programs and most importantly externally among the policy makers and funding organizations that support the existence of social service programs. Many of these principles are articulate by Michael Quinn Patton in his writings on Blue Marble Evaluation (Patton, 2019)

Overview of GBLD 538

The GBLD 538 course engages students to think globally by first exploring readings on leadership which they then apply to themselves.  Next students are introduced to four different paradigms (Mertins & Wilson, 2019) for viewing social programs and the role of evaluation a) post-positivism, b) pragmatism, c) social constructivism and d) transformative thinking, which they apply to the case example of a program addressing homelessness/unhoused. Using a favorite paradigm (or a combination of two or more paradigms), students choose an intervention strategy for dealing with homelessness/the unhoused in their community. In the following eight weeks of the course, students learn how to design an evaluation plan that gathers information on the process and effectiveness of the program, as well as facilitates learning and feedback loops to improve the program. In final weeks, students explore how evaluation contributes to learning and change – how they can facilitate this process. The course concludes with students reflecting on their own world view paradigm and other learnings.

Four Paradigms of Global Thinking Applied to Program Evaluation

1. Postpositivist Paradigm  
   • An objective reality that can be measured and quantified
   • Focus is on finding evidence-based results on program efficacy
   • Seeks good outcomes and mitigating risk and harm
   • Treat everyone equally and fairly with dignity and respect
   • The evaluator is neutral and often external to the program 
2. Pragmatism Paradigm 
   • Focuses on the problem and finding a practical solution
   • Accepts that there may be different problems and thus different solutions
   • Not concerned with the truth but with usefulness and what works.
   • Tends to use mixed methods of data collection 
   • The evaluator is neutral but could be external and/or internal to the organization. 
3. Constructivism Paradigm 
   • Acknowledges that there are multiple realities from the perspective of multiple stakeholders
   • Puts an emphasis on values and feelings in contrast to hard facts 
   • Primarily uses qualitative inquiry methods 
   • Encourages meaningful dialogue and reflection 
   • Evaluator is a member of the system and must take into account their own positionality/values 
4. Transformative 
   • Collaborates with multiple stakeholders including the program intended recipients; builds trust and relationship 
   • Expresses respect for cultural values and norms in the hopes of promoting human rights for social justice
   • Assumes various versions of reality that people navigate and understand privilege. 
   • Puts greater emphasis on the perspective of the marginalized and non-privileged
   • Emphasizes both qualitative and quantitative mixed methods to contextualize historical factors, explore process and discover outcomes at multiple levels and across systems
   • Using systems analysis to understand complexity and intersectionality of factors

Student Learning

Student learning was very illuminating.  By using the four paradigms, they were able to identify the complexity of homelessness and the needs of the unhoused.  They noticed the different demographics of people affected and the extensiveness of other social and health problems common to marginalized people who are experiencing housing challenges. Many of them identified with the transformative paradigm because it addressed issues of privilege and power, economic disparity, racism and discrimination and mental health and addiction, declaring that all people had the right to have adequate food, water, sanitation, security, shelter, respect and dignity. (The Universal Declaration of Human Rights adopted in1948 and subsequently in the Declaration of the Rights of the Child (adopted in 1959). Students carried their thinking and principles forward to the role of evaluation in examining the social programs implemented to address the issue of homelessness and the needs of the unhoused.  Students were also attracted to the use of the Pragmatism Paradigm because of the focus on solutions and ensuring utility of the evaluation effort. Combining both the Transformative Paradigm and the Pragmatic Paradigm was seen as a way to implement evaluative approaches that addressed the complexity of issues experienced by the unhoused while also considering the problem of homelessness for the whole community and facilitating transformational solutions that improve quality of life of the unhoused and whole communities.  

For information on how to participate in this course, go to the General Studies courses offered by Royal Roads University. A new course is being offered November 20, 2023. 

References 

Mertens, D. M., & Wilson, A.T. (2019). Program evaluation theory and practice. (2nd ed.). Guilford Press

Osland, J. S., Bird, A., Mendenhall, M. (2012). Developing global mindset and global leadership

capabilities. In G. K. Stahl, I. Björkman, & S. Morris (Eds.), Handbook of research in international human resource management (2nd ed., pp. 220-252). Edward Elgar. 

Patton, M. Q. (2019). Transformation to global sustainability: Implications for evaluation and evaluators. New Directions for Evaluation, 162, 103–117

Patton, M. Q. (2019). Blue marble evaluation: Premises and principles. Guilford Publications.

Patton, M. Q. (2023). Evaluation implications for the Coronavirus Global Health Pandemic Emergency, retrieved July 5, 2023 from https://bluemarbleeval.org/latest/evaluation-implications-coronavirus-global-health-pandemic-emergency