Researchers often struggle to impact policy and practice. In recent years, greater use has been made of management tools, such as logframes, to aid this process. However, many researchers are wary of such devices, and the underlying assumption that impact is about following generalised templates. There are also concerns that these logframes curb ‘blue skies’ or curiosity-driven enquiries. Such concerns certainly resonate with my own experience, though I believe, when applied flexibly, management tools are effective in holding research to its policy objectives. In this blog, I will discuss the Young Lives’ logframe and Theory of Change, and why these management tools can be extremely valuable.
Young Lives’ research is largely applied,
In the early days of Young Lives, there was little guidance on how to ensure research uptake in policy and practice and few metrics for gauging policy impact. There were also limited channels for communicating research evidence outside academia, beyond personal networks and stakeholder workshops. However, we did have a logframe—this tool having a long history in international development research—and later developed our Theory of Change, a device adopted by researchers relatively recently. The main advantage of these management tools is that they make explicit the causal link between the key components of a research project—i.e., its assumptions and objectives, its resources, activities and outputs, and ultimately, its outcomes and impact. Ideally, the tools should be complementary and work together iteratively. Thus, the logframe provides the indicators for monitoring a study’s performance against its chief objective and the Theory of Change articulates the methodology, or strategy, for meeting this objective. These two devices allowed Young Lives to untangle the knot, and see where we needed to invest our energies so that we would achieve the impacts we wanted.
Young Lives' logframe
The Young Lives’ logframe focuses on ensuring high-quality research and policy impact in our four study countries, as well as globally. The logframe also reflects the study’s commitment to maximising the power of its research by making our data accessible to all, and promoting its use. Thus, the study’s logframe comprises three broad outputs:
- robust research and policy products which highlight the determinants, pathways and consequences of poverty during childhood and adolescence;
- effective channels for disseminating research evidence to stakeholders and engaging with them around the policy implications;
- high-quality longitudinal mixed methods data, with survey data archived with the UK Data Service to magnify access.
Young Lives tracks the logframe outputs through a series of complementary quantitative and qualitative indicators. The Theory of Change specifies the channels of activity or pathways used to accomplish its poverty-reduction objective, with each pathway also feeding into and supporting one or more of the logframe’s outputs. We have always found it easier to identify quantitative indicators of impact, while qualitative indicators often appear elusive, but to capture a true picture of a study’s impact it is essential to use both.
For instance, one indicator of research uptake is the number of stakeholder events, policy briefs, blogs and other engagement activities and products that use Young Lives’ findings. However, input information of this sort is meaningless unless backed by evidence of actual outcomes or impacts. Often these latter are best assessed through qualitative indicators, such as reporting by policy stakeholders on policy or programmatic decisions and changes they have made in line with Young Lives’ messages. That said, even with regular monitoring of indicators, corroboration of uptake and impact can be extremely difficult to secure and remains a major challenge for Young Lives to this day.
Frequently, researchers’ resistance to logframes, and other tools, centres on the misperception that such tools are set in stone and therefore act as a straightjacket to research. But, in practice, as working documents, impact-focused management tools must be applied flexibly. At the very least, if they are to be relevant and useful they need to be adapted to the different political environments and communications contexts in which studies operate. This is quite complicated for Young Lives, both because of the multi-sectorial and multi-level nature of policies affecting children and young people and because the study covers five very different political-economic, socio-cultural and institutional contexts—i.e. not just four study countries but also the global policy arena. Thus, while close engagement and building trust with key policy stakeholders is central to policy uptake of research everywhere, whether these stakeholders come from central or local government, international or national non-governmental organisations, donors, or the private sector, varies widely depending on the context. What’s more, in longitudinal cohort research the stakeholders inevitably change as the study sample ages. As the Young Lives sample transitioned from childhood to adolescence the policies affecting them
Thus, the logframe is subject to regular appraisal and adjustment. In Young Lives, each indicator has a milestone and achievement in relation to this milestone, which we determine annually, this permits scrutiny of resource allocation as it evolves over time. These reviews also facilitate identification of policy-engagement opportunities, as well as obstacles and potential risks to accomplishment of logframe outputs, together with risk mitigation strategies.
In longitudinal research, government change is one of the most common risks to policy uptake; it can trigger major shifts in both policy priorities and important personnel, forcing researchers to start building trust and knowledge among officials from scratch. Political flux is one thing, and can be mitigated by forging strong partnerships with prominent local institutions, but some events are unforeseen and cannot be reasonably planned for. During my time as Director, the single most significant threat to Young Lives was the 2016 Referendum on the UK’s future relationship with the EU. The value of the Sterling went into freefall just as the fifth household survey round and final wave of school-effectiveness research were about to commence, hugely increasing our fieldwork costs. There was a substantial adverse knock-on effect; collection and public archiving of data had to take precedence over all other activities, reducing the volume of research outputs and policy engagement for the following 18 months.
Young Lives' Theory of Change
Another source of disquiet about logframes, and other management tools, is that they instrumentalise research unnecessarily, blocking ‘blue skies’ research in which real-world application to policy or practice is not immediately apparent. This is a genuine risk, and can be especially incapacitating for longitudinal studies given that it isn’t always possible to know what the most important policy-related questions are at the study’s outset. Young Lives has tried to address this concern through its Theory of Change by employing the ESRC’s conceptualisation of impact, which distinguishes three different kinds:
- Instrumental–contributing to the development of childhood poverty reduction policy, practice and/or services; shaping legislation and advocacy campaigns; and
- Conceptual– increasing awareness of childhood poverty and inequality, informing debates, directions in thinking and culture that lead to developments in policy and practice.
- Capacity—creating an enabling environment for research uptake in policy and practice by developing the skills of researchers, policy stakeholders, practitioners and advocates.
The Young Lives’ Theory of Change identifies the various pathways that we follow in order to achieve these three impact types. The advantage of this conceptualisation is its recognition that research can have an indirect as well as a direct influence on policy and practice, inasmuch as shifting thinking or stimulating debate can create new understandings, new priorities and new fields of policy and practice. However, there is some difficulty with this approach; while researchers tend to be very comfortable working towards conceptual change, donors are often compelled to demonstrate that their funds have direct instrumental impact.
This highlights the importance of discussing with donors the potential and limits of longitudinal observational research in relation to influencing policy. Crucially, for Young Lives this has meant highlighting the difference between observational longitudinal research and randomised controlled trials (RCTs). While RCTs provide valuable but somewhat isolated evidence on whether a certain intervention works, observational cohort data offer vital guidance on broader questions, including ‘under what conditions’ an intervention works and ‘in what systems’, as well as on options for systemic reform that cannot easily be addressed with experimental approaches. Thus, while acknowledging that as an observational longitudinal study it is challenging for Young Lives to achieve instrumental impact, the study can address questions essential to embed RCT evidence in a wider theory of change, helping the selection and design of effective programmes. This allowed us to agree with our donors on the importance of providing for both conceptual and instrumental impact in our work, as well as undertaking capacity development whenever possible. Only time can tell, but I suspect that our conceptual work on topics, such as the potential for children to recover cognitively from nutritional deficits in early childhood, and the value added by certain education systems in terms of children’s learning, may well turn out to have greater influence on children’s wellbeing, and development in LMICs, than our impact on specific policies or interventions.
The Young Lives’ Theory of Change recognises that achieving policy impact is about much more than simply building a body of rigorous evidence and analysis and tailoring messages to, and engaging with, key stakeholders. It is also about establishing an environment in which there is high acceptance of longitudinal research as a credible source of evidence for policy planning. This is where capacity development and innovation come in:
- Capacity-development - aware that stakeholders need to be able to understand the value, possibilities and limits of using evidence from longitudinal research to shape policy, Young Lives engages in developing the capacity, skills and knowledge of researchers in mixed-methods, longitudinal research and of external stakeholders in the utility of evidence-based approaches for planning, policy and practice.
- Innovation – using its unique research design to deliver age-appropriate and context relevant comparative, longitudinal mixed-methods evidence, Young Lives aims to be a globally recognised laboratory for innovation in research uptake and impact. This isn’t just down to the significance, quality, volume and/or applicability of Young Lives’ work, but being a ‘go-to’ source of information and learning for researchers, advocates, policy-makers and practitioners.
It is extremely hard for longitudinal observational research, such as Young Lives’, to get traction in policy, especially given the complex policy environments in which much research of this kind operates. It can be harder still to get confirmation from stakeholders of impact. Given the many challenges involved, it is understandable that researchers may be reticent about impact-focused management tools, particularly when they’re applied by donors solely to evaluate the cost effectiveness of a study in relation to narrow, instrumental policy objectives.
Yet despite these misgivings, the Young Lives’ Theory of Change and logframe have proved valuable as planning aids, helping establish what actions are most effective and where our energies and resources are best placed. They have enabled us to align our work with the study’s research, data and policy priorities and ensure that these support our overall objectives. Because the logframe is reviewed and revised annually it has also been helpful for tracking important trends in our impact efforts—for example, in data access and use by external researchers, and research dissemination and policy engagement and uptake.
I would argue that the effective use of such tools implies the need for openness and flexibility on the part of both donors and researchers as well as close negotiation around their function and limitations. These tools should not be considered as hoops researchers have to jump through to gain or retain funding, or a mechanism used by donors to police instrumental impact, but rather as useful exercises in and of themselves.
Researchers often struggle to impact policy and practice. In recent years, greater use has been made of management tools, such as logframes, to aid this process. However, many researchers are wary of such devices, and the underlying assumption that impact is about following generalised templates. There are also concerns that these logframes curb ‘blue skies’ or curiosity-driven enquiries. Such concerns certainly resonate with my own experience, though I believe, when applied flexibly, management tools are effective in holding research to its policy objectives. In this blog, I will discuss the Young Lives’ logframe and Theory of Change, and why these management tools can be extremely valuable.
Young Lives’ research is largely applied,
In the early days of Young Lives, there was little guidance on how to ensure research uptake in policy and practice and few metrics for gauging policy impact. There were also limited channels for communicating research evidence outside academia, beyond personal networks and stakeholder workshops. However, we did have a logframe—this tool having a long history in international development research—and later developed our Theory of Change, a device adopted by researchers relatively recently. The main advantage of these management tools is that they make explicit the causal link between the key components of a research project—i.e., its assumptions and objectives, its resources, activities and outputs, and ultimately, its outcomes and impact. Ideally, the tools should be complementary and work together iteratively. Thus, the logframe provides the indicators for monitoring a study’s performance against its chief objective and the Theory of Change articulates the methodology, or strategy, for meeting this objective. These two devices allowed Young Lives to untangle the knot, and see where we needed to invest our energies so that we would achieve the impacts we wanted.
Young Lives' logframe
The Young Lives’ logframe focuses on ensuring high-quality research and policy impact in our four study countries, as well as globally. The logframe also reflects the study’s commitment to maximising the power of its research by making our data accessible to all, and promoting its use. Thus, the study’s logframe comprises three broad outputs:
- robust research and policy products which highlight the determinants, pathways and consequences of poverty during childhood and adolescence;
- effective channels for disseminating research evidence to stakeholders and engaging with them around the policy implications;
- high-quality longitudinal mixed methods data, with survey data archived with the UK Data Service to magnify access.
Young Lives tracks the logframe outputs through a series of complementary quantitative and qualitative indicators. The Theory of Change specifies the channels of activity or pathways used to accomplish its poverty-reduction objective, with each pathway also feeding into and supporting one or more of the logframe’s outputs. We have always found it easier to identify quantitative indicators of impact, while qualitative indicators often appear elusive, but to capture a true picture of a study’s impact it is essential to use both.
For instance, one indicator of research uptake is the number of stakeholder events, policy briefs, blogs and other engagement activities and products that use Young Lives’ findings. However, input information of this sort is meaningless unless backed by evidence of actual outcomes or impacts. Often these latter are best assessed through qualitative indicators, such as reporting by policy stakeholders on policy or programmatic decisions and changes they have made in line with Young Lives’ messages. That said, even with regular monitoring of indicators, corroboration of uptake and impact can be extremely difficult to secure and remains a major challenge for Young Lives to this day.
Frequently, researchers’ resistance to logframes, and other tools, centres on the misperception that such tools are set in stone and therefore act as a straightjacket to research. But, in practice, as working documents, impact-focused management tools must be applied flexibly. At the very least, if they are to be relevant and useful they need to be adapted to the different political environments and communications contexts in which studies operate. This is quite complicated for Young Lives, both because of the multi-sectorial and multi-level nature of policies affecting children and young people and because the study covers five very different political-economic, socio-cultural and institutional contexts—i.e. not just four study countries but also the global policy arena. Thus, while close engagement and building trust with key policy stakeholders is central to policy uptake of research everywhere, whether these stakeholders come from central or local government, international or national non-governmental organisations, donors, or the private sector, varies widely depending on the context. What’s more, in longitudinal cohort research the stakeholders inevitably change as the study sample ages. As the Young Lives sample transitioned from childhood to adolescence the policies affecting them
Thus, the logframe is subject to regular appraisal and adjustment. In Young Lives, each indicator has a milestone and achievement in relation to this milestone, which we determine annually, this permits scrutiny of resource allocation as it evolves over time. These reviews also facilitate identification of policy-engagement opportunities, as well as obstacles and potential risks to accomplishment of logframe outputs, together with risk mitigation strategies.
In longitudinal research, government change is one of the most common risks to policy uptake; it can trigger major shifts in both policy priorities and important personnel, forcing researchers to start building trust and knowledge among officials from scratch. Political flux is one thing, and can be mitigated by forging strong partnerships with prominent local institutions, but some events are unforeseen and cannot be reasonably planned for. During my time as Director, the single most significant threat to Young Lives was the 2016 Referendum on the UK’s future relationship with the EU. The value of the Sterling went into freefall just as the fifth household survey round and final wave of school-effectiveness research were about to commence, hugely increasing our fieldwork costs. There was a substantial adverse knock-on effect; collection and public archiving of data had to take precedence over all other activities, reducing the volume of research outputs and policy engagement for the following 18 months.
Young Lives' Theory of Change
Another source of disquiet about logframes, and other management tools, is that they instrumentalise research unnecessarily, blocking ‘blue skies’ research in which real-world application to policy or practice is not immediately apparent. This is a genuine risk, and can be especially incapacitating for longitudinal studies given that it isn’t always possible to know what the most important policy-related questions are at the study’s outset. Young Lives has tried to address this concern through its Theory of Change by employing the ESRC’s conceptualisation of impact, which distinguishes three different kinds:
- Instrumental–contributing to the development of childhood poverty reduction policy, practice and/or services; shaping legislation and advocacy campaigns; and
- Conceptual– increasing awareness of childhood poverty and inequality, informing debates, directions in thinking and culture that lead to developments in policy and practice.
- Capacity—creating an enabling environment for research uptake in policy and practice by developing the skills of researchers, policy stakeholders, practitioners and advocates.
The Young Lives’ Theory of Change identifies the various pathways that we follow in order to achieve these three impact types. The advantage of this conceptualisation is its recognition that research can have an indirect as well as a direct influence on policy and practice, inasmuch as shifting thinking or stimulating debate can create new understandings, new priorities and new fields of policy and practice. However, there is some difficulty with this approach; while researchers tend to be very comfortable working towards conceptual change, donors are often compelled to demonstrate that their funds have direct instrumental impact.
This highlights the importance of discussing with donors the potential and limits of longitudinal observational research in relation to influencing policy. Crucially, for Young Lives this has meant highlighting the difference between observational longitudinal research and randomised controlled trials (RCTs). While RCTs provide valuable but somewhat isolated evidence on whether a certain intervention works, observational cohort data offer vital guidance on broader questions, including ‘under what conditions’ an intervention works and ‘in what systems’, as well as on options for systemic reform that cannot easily be addressed with experimental approaches. Thus, while acknowledging that as an observational longitudinal study it is challenging for Young Lives to achieve instrumental impact, the study can address questions essential to embed RCT evidence in a wider theory of change, helping the selection and design of effective programmes. This allowed us to agree with our donors on the importance of providing for both conceptual and instrumental impact in our work, as well as undertaking capacity development whenever possible. Only time can tell, but I suspect that our conceptual work on topics, such as the potential for children to recover cognitively from nutritional deficits in early childhood, and the value added by certain education systems in terms of children’s learning, may well turn out to have greater influence on children’s wellbeing, and development in LMICs, than our impact on specific policies or interventions.
The Young Lives’ Theory of Change recognises that achieving policy impact is about much more than simply building a body of rigorous evidence and analysis and tailoring messages to, and engaging with, key stakeholders. It is also about establishing an environment in which there is high acceptance of longitudinal research as a credible source of evidence for policy planning. This is where capacity development and innovation come in:
- Capacity-development - aware that stakeholders need to be able to understand the value, possibilities and limits of using evidence from longitudinal research to shape policy, Young Lives engages in developing the capacity, skills and knowledge of researchers in mixed-methods, longitudinal research and of external stakeholders in the utility of evidence-based approaches for planning, policy and practice.
- Innovation – using its unique research design to deliver age-appropriate and context relevant comparative, longitudinal mixed-methods evidence, Young Lives aims to be a globally recognised laboratory for innovation in research uptake and impact. This isn’t just down to the significance, quality, volume and/or applicability of Young Lives’ work, but being a ‘go-to’ source of information and learning for researchers, advocates, policy-makers and practitioners.
It is extremely hard for longitudinal observational research, such as Young Lives’, to get traction in policy, especially given the complex policy environments in which much research of this kind operates. It can be harder still to get confirmation from stakeholders of impact. Given the many challenges involved, it is understandable that researchers may be reticent about impact-focused management tools, particularly when they’re applied by donors solely to evaluate the cost effectiveness of a study in relation to narrow, instrumental policy objectives.
Yet despite these misgivings, the Young Lives’ Theory of Change and logframe have proved valuable as planning aids, helping establish what actions are most effective and where our energies and resources are best placed. They have enabled us to align our work with the study’s research, data and policy priorities and ensure that these support our overall objectives. Because the logframe is reviewed and revised annually it has also been helpful for tracking important trends in our impact efforts—for example, in data access and use by external researchers, and research dissemination and policy engagement and uptake.
I would argue that the effective use of such tools implies the need for openness and flexibility on the part of both donors and researchers as well as close negotiation around their function and limitations. These tools should not be considered as hoops researchers have to jump through to gain or retain funding, or a mechanism used by donors to police instrumental impact, but rather as useful exercises in and of themselves.