Assessing Assessors – How to Evaluate the Scientific Performance of a Pesticide Regulatory Authority, by Sari Autio

A scientific performance evaluation is a valuable means for organizational learning and for strengthening the accountability of regulatory authorities. At the request of the Dutch Competent Authority for authorisation of pesticides, an evaluation of its scientific processes, scientific output, and decision-making process was undertaken by an International Visitation Committee in 2013 and repeated in 2018. Drawing on this experience, a general guidance for conducting such evaluations is suggested.

Scientific Performance of a Competent Authority

Citizens’ concerns about the risks of hazardous chemicals are increasing. Accountability and reliability of regulatory authorities involved in the multidisciplinary risk assessment of hazardous chemicals is a prerequisite for public trust. When citizens receive transparent, reliable, and up-to-date information from regulatory authorities, it creates confidence and trust in their institutional performance.

Pesticides, including plant protection products and biocides, are intentionally designed to be toxic to living target organisms and have the potential to kill or control harmful organisms and pests. At the same time, they can cause adverse effects to other organisms, human health, and the environment. Therefore, in the European Union, the release onto the market of these hazardous chemicals is strictly regulated. Based on European Regulations (EC) 1107/2009 and (EU) 528/2012, active substances are approved at the Community level, and products containing those active substances are authorised nationally. The Competent Authorities of the Member States are involved in scientific risk assessments for European-wide approvals of active substances and are responsible for product authorisations. 

While harmonized risk assessment methodologies for pesticides exist in the EU, there are differences between countries in regulating environmental health risks. The European regulatory framework provides common criteria for the risk assessment, whereas risk management is a Member State issue with variable practices, for instance when introducing risk mitigation measures to prevent neonicotinoid-induced bee mortalities. Joint procedures are developed and recommended, for instance, by the Organisation for Economic Co-operation and Development.

A strong EU-oriented integration requires a high level of competence and expertise among national authorities to ensure that their performance is efficient, objective, and adequate. However, there is no legal requirement to evaluate the scientific performance of national regulatory authorities. Such undertakings are based on voluntary interest, as scientific performance evaluations enable the institutional learning and innovation of authorities.

In the Netherlands, the Board for the Authorisation of Plant Protection Products and Biocides (Ctgb) is the Competent Authority for authorisation of pesticides. At its request, an evaluation of its scientific processes, scientific output, and decision-making process was undertaken by an external panel in 2013 and repeated in 2018 to follow the progress made during the intermediate five years. Both evaluations were carried out by an International Visitation Committee of experienced senior experts on pesticide risk assessment and risk management in the public sector.

There is lots of literature concerning project, programme, and practice evaluation of different fields in society. Still, there is little practical guidance and few suggestions available on how to conduct an evaluation of a pesticide regulatory authority. Therefore, our paper provides practical insights about methodologies and approaches that could be applied in such visitations (Autio et al. 2021). Details of the Ctgb visitations are available in the original reports (Koëter et al. 2013, Koëter et al. 2018), while some general recommendations arose from our experience in conducting the evaluations. 

Approach

An important element of any visitation is the stature of the members of the visitation committee. The Chair is appointed by the management of the Competent Authority; potential committee members are identified by the Chair for endorsement by the Competent Authority, based on their expertise, independence, and geographical coverage over the zones in the EU.

A Terms of Reference document describes the objective(s) of the evaluation, composition of the visitation committee, evaluation methodologies, timetable, confidentiality issues, and remuneration questions. Moreover, a roadmap for an evaluation is a tool to keep track of the project’s progress and to avoid omissions. In the case of the Netherlands, the objectives of the evaluation of the Ctgb were to assess its independence, as well as the scientific quality and the legal compliance of its formal decision-making process and decisions taken for the authorisation of plant protection products and biocides. The roadmap for Ctgb provided information on data sources, planning for interviews, an assessment plan of information gathered, a draft compilation of observations and conclusions, and timetables for providing the draft report for scrutiny and presenting the final report. Documents of interest for the visitation committee included general management and decision-making processes, scientific staff policies, dossiers of authorisation requests, and general informative documents. Neither the authorisation fees nor comparisons to other Competent Authorities were within the scope of this evaluation.

The evaluators should organise an appropriate number of meetings with the management and scientific staff to promote a balanced approach of learning and accountability. During the visits, individual interviews with staff members can be held. In addition, the evaluators should communicate regularly between themselves via video calls and e-mails.

The quality of the scientific staff determines the reliability of the scientific output of the institution, and therefore an evaluation should focus on the multitude of different disciplines necessary for conducting scientific risk assessments. Access is required for a range documents – such as the annually updated accurate Curricula Vitae and Declarations of Interest of all scientific staff. As the visitation deals with personal data, the evaluators should always adhere to Evaluator’s Ethical Guiding Principles and keep the information confidential.

Interviewing of scientific staff members – individually and/or as teams – is an essential part of evaluation methodology. Open-ended interviews should cover workload, teamwork, professionally challenging level, level of appreciation, responsibilities, career options and awareness of the relevance for the health and safety of society and the environment. The senior management and human resources staff should also be interviewed separately from the staff members since their respective views affect the scientific output of the organization. As a general recommendation, the Human Resources staff could regularly stimulate and support the scientists to broaden and improve their knowledge and scientific competency.

While the EU regulations define the tasks of the Competent Authorities of the Member States, it is not stated how they should be organized in each country. National authorities deal with risk assessment and risk management according to their national regulatory systems. In some countries, risk assessment and risk management are carried out by separate organizations. In others, they are carried out as distinct functions within one Competent Authority. In either case, EU principles require that risk assessment activities be science-based, transparent, and independent from risk management. This is a critical issue to be evaluated. Risk assessment is based on the dossier of scientific studies according to internationally agreed study guidelines, on the one hand, and on guidance documents grounded in the Uniform Principles laid down in Commission Regulation (EU) 546/2011 , on the other hand. Risk management considers the outcome of the risk assessment as the starting point of its reasoned decision-making process, and considers aspects like local economic, agricultural, ethical, and political consequences.

When the same organization is responsible for both functions, its management should ensure that scientific risk assessment conclusions will not be blurred by risk management arguments. Ideally, the organization would have two physically separated units. Therefore, the evaluation should consider the organizational structure in detail to scrutinize whether, in its outputs, there is a clear and understandable separation between the scientific risk assessment and the risk management decision – taken with full supporting arguments – and whether this is presented in a transparent and consistent format.

Dossier evaluation is the cornerstone of a Competent Authority’s scientific activity, and therefore a major effort is expected from evaluators in scrutinizing dossier evaluations. The purpose is not to review the validity of specific risk assessments’ conclusions, but rather to get an overview of the scientific process producing such documents in general.

Because the risk assessment and risk management processes take time and involve several experts in the workflow, it is important to keep track of all actions during the scientific process. This internal traceability is usually enabled by a Document Management System, and is a requisite for the transparency and trustworthiness of the whole process.

Proactive contribution to the harmonization of the regulatory framework and to the development of international guidance documents and risk assessment methodologies, is an important opportunity for the staff to develop their scientific expertise and knowledge, and should be covered in any evaluation of a Competent Authority. The forum of authorities within an EU zone, for instance, is an important platform to discuss and solve long-running zonal issues in the field of pesticide legislations.

Openness, Transparency, and Communication

Competent Authorities are leading regulatory forces whose activities are able to maintain public trust. The development of public trust requires clear, effective, and open communication with the outside world – including all stakeholders and the wider international regulatory community. This is also the overarching principle of the European General Food Law (EU) 2019/1381.

Therefore, it is recommendable to ensure that the conceptual framework of the Competent Authority distinguishing risk assessment from risk management shows an appropriate level of openness and transparency throughout its work processes. Such a framework would further provide insights into the interactions between the scientific risk assessment conclusions and risk management arguments. Such insight would be necessary in order to finalize regulatory decisions.  

Confidentiality and data protection issues, openness, and transparency are key values underpinning risk assessment and risk management. Equally important is the balance to be found amongst these components. Industry tends to prioritize confidentiality and protection of data, backed up by fairness in competition and the right of enterprises to defend their own invention. Non-governmental organizations highlight increased transparency and free access to dossiers, with the aim of minimizing human and environmental exposure and risks. Legislative documents such as  the Aarhus Convention and judgments of the Court of Justice of the European Union (Tweedale and Hautala) also appear to prioritise the right to demand access to safety-relevant information. The Dutch governance tradition (originating from the collaborate damming of polder areas) similarly calls for open knowledge-sharing in environmental policy, as well as stakeholders’ engagement. Overall, developing a balanced approach between confidentiality and transparency is an ongoing process involving the whole EU regulatory framework, where scientific authorities can apply reflective practices in their risk communication.  

Conclusions

In conclusion, and having considered the evaluation policy of the United Nations Economic Commission for Europe, a scientific performance evaluation is a valuable means for organizational learning, strengthening the accountability of public bodies, and providing societal benefits. To our knowledge, an evaluation as performed in the Netherlands is unique among the Competent Authorities authorizing hazardous chemicals. Drawing on our experience, we brought insights and advice, and suggested a general guidance for conducting such evaluations. Visitation committees carrying out voluntary evaluations should be perceived – and behave – as advisors rather than examiners. The progress made by the Ctgb in five years indicates that periodic external evaluation could assist the Competent Authorities in many countries, especially regarding the following: developing their scientific capacities; streamlining their processes for gaining high trust amongst society; and achieving a high level of protection of human health and the environment.

Posted by Sari Autio (Senior Officer, PhD, Finnish Safety and Chemicals Agency Tukes)

Suggested citation: S. Autio, “Assessing Assessors – How to Evaluate the Scientific Performance of a Pesticide Regulatory Authority”, published on REALaw.blog and available at https://realaw.blog/?p=610