Challenges of integrating evidence into health policy and planning: linking multiple disciplinary approaches April 2017, Volume 27, Issue 2

Introduction

Scholarly work on evidence utilisation aims to understand and improve policy making and policy outcomes in terms of better mobilisation of evidence. Science based policy, evidence based policy and evidence informed policy making (EIPM) are terms used in both academia and policy to describe a form of policy making that draws on the key principle of evidence based medicine – that is, that strong, well-designed and well-conducted research should influence how policies and programs are developed and delivered. It is not surprising that the EIPM concept has been readily embraced in health policy, public health, health promotion, health services and health systems, considering its close ties to clinical and population medicine.

Although the call for more, and better, EIPM is not new, there has been a substantial increase in the amount of scholarly work on this subject in recent years. Systematic reviews have uncovered a large number of studies recounting the limited use of evidence in policy making, and descriptive studies attempting to unpick the problem of low uptake of evidence, and the specific barriers and facilitators for it.^1-4 Recent critiques identify a dominant thread in this literature that takes the problem of EIPM and narrows it to a nexus between evidence production (figuratively positioned outside the policy world) and specific policy targets (usually within government), leading to a dearth of studies that seek a broader understanding of policy processes and how they actually work.^1,5-7

In this article, we seek to highlight some of the research and actions from multiple disciplines that contribute to a comprehensive understanding of EIPM, with the intention of creating an accessible starting point for a more interdisciplinary approach to the subject. Public health, social science, political science and other disciplines have engaged with the challenges of increasing the use of research evidence in decision making processes, creating a diverse landscape of scholarship and action. The ability to access, understand and make use of approaches, concepts and theories from all of these disciplines can support evidence utilisation in health policy and planning; however, there are challenges in doing so.

We seek to address these challenges by bringing together key insights from several disciplines to explore how an interdisciplinary approach may facilitate more comprehensive theory-driven approaches to EIPM.⁸ We define interdisciplinarity along the lines of Choi and Pak as coordinating and linking different disciplines into a coherent whole to address a particular issue or problem.⁹

Challenges of interdisciplinary research and action

Research into evidence utilisation in policy has spanned multiple disciplines, including public health, public policy, behavioural sciences, political philosophy and administration science. However, the different disciplinary approaches often use very different research methods and begin with very different basic assumptions.⁶ Even conceptualisations of policy making and policy makers are approached differently. Here, we use a broad definition of policy making as the process of making decisions in a particular policy area that includes defining, and finding workable solutions to, societal problems. Unless stated otherwise, we use the term policy maker to mean government officials who engage in this process. However, we acknowledge that part of the challenge of interdisciplinarity in this field is that some disciplines are more likely to refer to policy makers as elected officials while others define policy makers as individuals working across sectors, including government and nongovernment agencies. Different disciplines may tend to focus on a particular stage of the policy making process when considering EIPM, while others take a macro view. Disciplines will also differ in terms of how they consider evidence: what constitutes evidence, which evidence is important and what we need evidence about. Research into EIPM is therefore faced with a fundamental challenge of multiple disciplinary discourses.

There are both practical and philosophical obstacles to bridging disciplinary divides.¹⁰ In a practical sense, research published in one discipline tends to accumulate in a select set of academic journals, and articles tend to cross-reference these journals. Journals from other disciplines may be indexed in different databases, further limiting the literature captured even in systematic reviews. Certain terms and keywords used to describe and index articles also differ across disciplines, and scholars educated in one discipline tend to become familiar with that literature and its language.¹¹ Researchers tend to have to compete for funding among disciplinary peers, so set parameters around research that align with common starting points, research paradigms, priorities and methodologies. It is also unusual for researchers from multiple disciplines to collaborate, despite moves from funding bodies to foster interdisciplinary research.

These practical obstacles create and compound philosophical obstacles to interdisciplinary work. Once a researcher is embedded in a research paradigm from one discipline, the application and interpretation of alternative approaches prove more difficult. This leads to a proliferation of research that adopts common assumptions, and tends not to challenge the underlying epistemological, conceptual and methodological premises of knowledge. There is a reluctance to delve into unfamiliar disciplinary fields, partly as a result of conflicts of understanding, and partly as a result of not knowing where or how to ‘enter’ a new discipline and do justice to its many facets, directions, theories, frameworks and disciplinary history. Key concepts and terms used in one discipline may be used very differently in another, and even have negative connotations outside the disciplinary context. Terms such as ‘rational’, ‘nonrational’, ‘power’, ‘influence’, ‘hierarchy’, ‘rule’ and ‘compliance’ are just some that may be difficult to apply across contexts. These are common terms in some of the approaches we explore.

However, there are three particularly important reasons why it is worth challenging disciplinary divides in EIPM. First, different disciplines bring important insights for understanding the complexity of policy making and where evidence fits into its processes. We know that health policy and planning require multiple complex considerations, and research and action in this area must recognise this complexity.¹²

Second, policy makers themselves come from multiple disciplinary backgrounds. They interact with evidence in different ways and will be convinced by, and willing to work with, different types of evidence from different sources.⁶ Being able to acknowledge different strengths, weaknesses and applicability of research from different backgrounds is important for working with and within policy.

Third, and perhaps most importantly, EIPM is itself an attempt to bring key principles from medicine and natural sciences research into a social world of decision making. Fundamental to increasing the use of evidence in policy are the underlying assumptions based on the philosophy of science about what we know and how we know it.⁶ Acknowledging and understanding this fundamental debate about knowledge and how it is constructed will help us to advance the use of evidence in policy.

Insights from four disciplines

We assume that research into evidence utilisation that describes the evidence needs of policy makers in terms of succinct, understandable, timely evidence is well known^2,3,13, as is the literature that describes financial, organisational, temporal and political constraints on individual decision making in the policy environment.⁴ We therefore aim to highlight key theoretical insights from four different disciplinary approaches as exemplars that could strengthen research and action around these assumptions. We have chosen these four broad approaches on the basis of their contrasting starting points, theoretical constructs and epistemological assumptions about knowledge creation. We cannot and do not suggest that each of the disciplines we discuss here is addressed comprehensively. Each has a long history, and encompasses a scientific community that has its own breadth of methodologies, approaches and debates about the nature of its discipline. We highlight some important contributions of each with respect to EIPM, and define a select few concepts and frameworks in a way that should be accessible for those who are research literate but unfamiliar with the other disciplines. In doing so, we refer to seminal authors, systematic reviews and easily accessible summaries within the literature. We do this to offer an entry point to each approach as a means to encourage the exploration and adoption of interdisciplinary principles in EIPM work.

Information processing and behavioural sciences

Policy makers at all levels are occupied with making decisions. These might be what to include in a ministerial briefing, which policy instrument to adopt to induce a behaviour change, which program to fund or what issue to pay attention to.

Understanding how individuals make decisions is therefore an important facet of grasping how evidence is used in policy. This approach has human cognition and relationships as its central focus. It seeks primarily to understand why and how people make the decisions they do. Key concepts include rationality, cognitive processing, human relationships and human behaviour.

In behavioural approaches, the concept of bounded rationality – whereby decision making ability is limited by environments as well as personal cognitive factors – challenges assumptions about the extent to which decisions makers can, and are willing to, act on evidence, even if it is strong, relevant, timely and well communicated. This has parallels with clinical decision making in evidence based medicine, where a particular health problem of a patient is presented and the clinician has to make a decision about the best course of action. In clinical decision making, evidence is used but tailored to the individual circumstances, and the ‘importance’ of evidence is relative to other factors that may predominate. For example, because of patient preferences, comorbidities or access factors, the ‘best’ evidence based option may not be chosen.

Behavioural approaches distinguish between different cognitive processes in decision making. Policy makers often have multiple and potentially competing objectives, such as completing specific instructed tasks, maximising benefits to society and working towards personal or group goals.¹⁴ In some instances, competing goals will be prioritised and available information will be considered to determine a preferred course of action.¹⁵ In other instances, decisions are made based on other factors. For example, in situations where people have to achieve multiple objectives within a set timeframe, they may ‘satisfice’ (i.e. choose the first alternative that meets or satisfies the minimum criteria for meeting their objectives)¹⁴, be unsure which objectives to prioritise, or make mental shortcuts such as “drawing on emotions, gut feelings, deeply held beliefs, and habits”.¹⁵ In other words, the ability to make rational decisions is bounded.

Decision making is complicated even further in situations where decisions are made in groups, as is frequently the case in policy making environments.¹⁴ Behavioural approaches to EIPM are therefore also useful to understand potential phenomena of group decision making, such as groupthink, where the dynamics of a group lead to suboptimal decisions¹⁶, and nonrational escalation of commitment, where an individual becomes more committed to a particular course of action because of the resources and group commitment already invested in it.¹⁷

Each of these elements can challenge how to think about decision making in policy, but also presents opportunities for research and action. To encourage further incorporation of insights from behavioural approaches, we suggest posing the following question as a gateway: “In what ways might policy makers use evidence in their cognitive and group decision making processes?”

Theories of policy making and the political sciences

Like behavioural approaches, dominant theories of policy making found in the policy and political sciences do not necessarily take evidence as a starting point, or even as a core component, in policy making. The main purpose of theories of policy making is to determine how and why certain policies come into being, and elucidate the role of institutions, individuals, strategy, coalitions and networks in setting agendas and arriving at solutions. Key concepts include systems of government, networks, issue framing, institutions and ideology. Evidence may feed in to the policy making process, but the discipline is also concerned with evidence about the policy process.

Research in these disciplines seeks to make sense of a highly complex policy environment made up of actors, relationships, ideas and sets of core drivers. The strength of the literature lies in the conceptual frameworks that try to make sense of this complexity. Two of the most widely known and adopted are the multiple streams framework¹⁸ and the advocacy coalition framework.¹⁹ The multiple streams framework puts forward conditions for policy change – the opening of a policy window – that occur when focusing events (problem stream), acceptable solutions (policy stream) and exogenous forces (political stream) temporally align.¹⁸ The advocacy coalition framework lays out how multiple actors in policy processes, in different locations, work together to influence policy systems and subsystems in sometimes stable and sometimes unpredictable contexts.¹⁹

Some of the most frequently cited works in the EIPM literature stem from policy sciences, but their insights are frequently misinterpreted or underused. Weiss’s seminal article illustrating a typology of six methods of evidence utilisation described instrumental models (knowledge driven, problem driven and interactional), as well as political, tactical and enlightenment models.²⁰ The enlightenment model was, and remains, the least well understood and least applied in research. However, it is the one that most embraces a complex view of policy making, describing situations in which ideas from evidence are “percolating through informed publics and coming to shape the way in which people think about social issues”.²¹ An example is the change over time in the acceptability of smoking cigarettes.

Reference to policy science theory in EIPM research and action has increased in recent years²², but its true application still proves challenging. Again, we suggest that this is because of the existence of both practical and philosophical disciplinary divides. As an avenue to overcome them, we suggest posing the question “How do we understand the way in which policy is being made?”

Critical theory and political philosophy

The critical research paradigm begins with an ontological and epistemological position that knowledge is framed by those with power, and the purpose of research is to uncover and challenge the underpinnings of those frames.²³ Critical research into EIPM starts with questions about the value of evidence utilisation, both in conjunction with, and juxtaposed against, the role of the policy maker, the elected official and government. Key concepts are governance, democracy, representation, ethics and power. In terms of EIPM, the discipline is concerned with who produces evidence, and how evidence is interpreted and used.

Political philosophy has long occupied itself with the function of the modern state, and the role of elected officials and bureaucracies within it, including the primary function of working through moral and ethical issues to make choices and act in the best interests of constituent publics. The process of making these choices can partly be based on technical knowledge, but is also largely occupied with the moral challenge of fulfilling obligations under the so-called social contract – whereby the public agrees to surrender part of its autonomy in exchange for the benefits of living in a stable society that protects the right to pursue happiness.²⁴

Critical research therefore explores the underlying tensions between EIPM and democracy. Many scholars conclude that the two are fundamentally incompatible because of several factors.^6,25,26 First, the nature of the production of research and knowledge means that they are created with a particular frame of ‘the good’ and what is important. It is the task of public representatives to scrutinise knowledge production, and when and how knowledge should be used.^26,27 Second, legitimate government requires working through vexing moral choices; this can only be done through debate.¹² Third, there are risks with equating the right to power with the possession of knowledge – akin to ‘epistocracy’, a term coined by Estland to describe the type of society in which experts rule over democratic policy.^26,28

We argue that each of these streams of argument found in the critical theory literature is worth exploring and understanding because they provide insights into the limits of EIPM from practical and ethical standpoints, and can provide understanding about opposing views. As a gateway, we suggest asking “What assumptions are being made about the value of EIPM, and at what point does that value reach its limit?”

Intervention research and implementation science

Finally, we return to an area of research that, at first glance, seems most compatible with the principles of EIPM, but is surprisingly still limited in the literature.²⁹ Intervention research attempts to determine the impact of strategies applied in real-world settings to induce some form of change (e.g. health-promoting behaviour, use of evidence in policy). There is a strong focus on finding the right study design to gain plausible explanations of what works in what contexts. Key concepts include impact, effectiveness, theory of change, causation and context. EIPM in this discipline relates to the generation and use of reliable knowledge that can inform what policy approach to take, what programs to implement and how to execute them.

Implementation science is the study of methods to promote the uptake of what we know ‘works’ from such research into real-world practice.³⁰ Its foundations are in medicine and clinical practice, as are those of evidence based medicine. Despite this, in public health broadly, the amount of literature dedicated to intervention research is a small fraction of overall research.²⁹ In EIPM specifically, Moore and colleagues found just 5 of more than 100 papers reported on studies that attempted to evaluate the impact of strategies using intervention research.¹ Similarly, Williamson and colleagues found just nine intervention studies in a review of strategies to increase evidence utilisation in mental health policy.³¹

Some of the strategies that have been tested in intervention research include providing timely research summaries, training policy makers in critical appraisal skills, deploying knowledge brokers, facilitating proximity and dialogue between policy makers and researchers, and developing packages of interventions that address the culture of policy making settings to make them more evidence attuned.^31,32

Intervention research maintains a focus on study design and methodological rigour, but also embraces challenges of ‘natural’ context and increasing complexity. This can make it difficult to gain entry into such research, but we argue that it should not deter scholars and advocates for EIPM from embracing key insights from this approach. As a gateway, we propose asking “How can we know what is making a difference?”

Lessons learnt

The four approaches outlined in this article have very different starting points and use very different methodologies. Whereas intervention research frequently adopts experimental and quasi-experimental methods, critical scholarship relies on historical methodology and reasoning. Policy sciences tend to use case studies and case comparisons to build and test models and theories of large-scale systems and processes, whereas behavioural approaches are concerned with individual- and group-level units of study and use mixed methods designs to understand decision making. These fundamentally different epistemologies and methodologies can make it difficult to move from one approach to another, and combine their insights logically and usefully.

One approach to making sense of multiple disciplinary approaches is in frameworks of EIPM that combine insights from more than one of the approaches above.³³ For example, in documenting the “science of muddling through”³⁴, Lindblom drew on concepts of bounded rationality to describe the persistence of the status quo in decision making and spurred a policy science theory of incrementalism. The SPIRIT Action Framework³⁵ from Redman and colleagues pivots around catalysts, capacity and engagements, while incorporating different modes of research utilisation and multiple policy influences. The challenge with these and similar frameworks is how to make best use of their disciplinary breadth. Factors such as conscious effort, time and resources to develop interpersonal relationships and effective communication between experts from different disciplines may be key³⁶, and development of these relationships can be incentivised at the institutional level.³⁷ However, we suggest that this requires further exploration in EIPM research.

We therefore propose four questions that we suggest will, if pursued, act as a stimulus to an interdisciplinary approach to scholarship and action on EIPM, exploring multiple theories, models and methods: In what ways might policy makers use evidence in their cognitive and group decision making processes? How do we understand the way in which policy is being made? What assumptions are being made about the value of EIPM, and at what point does that value reach its limit? How can we know what is making a difference?

Copyright:

© 2017 Huckel Schneider and Blyth. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.