Quality assurance della formazione medica specialistica: applicazione della norma IS0 9001 e gestione della valutazione nella scuola di specializzazione in pediatria di Padova

Introduction - The issues of quality assurance and quality improvement are betweeen the leading points in guiding the European political process for the development of common strategies in higher education, known like Bologna Process, from the end of years '90. In post-graduate medical education there is an increasing interest in such issues, reflecting the imminent scene of a common European sanitary system, characterized from the circulation of the sanitary professionals in the European Countries, and the growing demand for excellence in the medical performances. Quality assurance in medical education has meant to develop standards and guidelines for the university and post-graduate education, and to promote the mutual acknowledgment of the curricula and degrees. Moreover, promotion of cooperation in quality assurance is also a “culture of quality”, that it should permeate each individual post-graduate medical education programs. The ISO 9001:2000 Standard outlines requirements for quality management systems, offering a support for quality assurance strategies. The objective of these standards is to provide an effective system reflecting a company’s practices for producing goods and services that conform to requirements and enhance customer satisfaction. The peculiarity of a quality system based on the ISO 9001:2000 standards is the fact that it focuses on process rather than on product quality or on content, and that it can be applied to any type of organization. The Resident Affair Commettee of the postgraduate pediatric training program of the University of Padova has decided to apply ISO 9001 Standards of Quality, not only for the organizational and administrative aspects, but above all for the management of the educational processes (learning and practice activities). Between ISO 9001 principles, the continuous improvement (based on the monitoring of processes) and the customer focused organisation are of paramount importance in the quality management for the postgraduate medical education. In this context, “continuous monitoring” can be translated in “evaluation of the residents in terms of medical competency, professional attitudes, level of autonomy, in relationship to the learning outcomes“, and the “customer satisfaction” concept can be translated in “the appraisal on the programs of formation supplied by the residents (the main customers of the School)”. Purpose of the thesis is to describe the experience of application of the ISO 9001:2000 Standard for the management of internal quality assurance and quality improvement within the pediatric training program of the Univeristy of Padova, and to evaluate some results, limits and perspectives that this experience can suggest for the improvement of the quality in the higher medical education. Another objective of the thesis is to discuss the implementation of a comprehensive evaluation system, in order to satisfy ISO 9001:2000 requirements, and the strategic meaning of the evaluation like source for the development of indicators for the continuous quality improvement. Methods - The postgraduate pediatric training program of the University of Padua is a Ministerial accredited five-year program that provides a Diploma of Specialization in Pediatrics. Approximately 80% of learning activities take place in the setting of clinical practice, under supervision, with increasing levels of responsibilities throughout the five-year period. The remainder of the learning activities include formal lectures, seminars, workshops and personal studies. 92 residents were in training in 2007/2008 academic year. In order to build a quality management system based on the ISO 9001:2000 standard, the Direction of the School has established a Quality Policy, defining Mission and quality goals of the School; therefore it has identified the interlinked processes on which the training program is based, using a a “process management approach”. The Quality Management System has been documented by a Quality Manual, organizational procedures, work instructions and records. A system to record, monitor and produce objective data regarding some of the activities of the postgraduate pediatric training program was introduced. The entire process of creation and implementation of this system took almost 18 months. During this period three internal audits of the ISO 9001:2000 standards were conducted by an expert group. After the three internal audits, an accredited third-party registrar audited the quality system and certified that the postgraduate pediatric training program of the Padua University met the ISO 9001:2000 requirements. Between the main quality system processes, the one relative to the evaluation of the residents and the activities has been submitted to a substantial review. First of all, specific formative objectives, for every year of school and every period of training, have been defined, to which the appraisal of the single residents should be referred. Then, questionnaries of evaluation have been developed. The inner coherence of questionnaries has been validated by measurement of ? of Cronbach (mean value > 0,9). The appraisal process has thus become source of results for monitoring customer satisfaction and the impact of the certification. Results - By adoption of a quality management system through ISO 9001:2000 Standard, for the first time the pediatric postgraduate training program formalized its Policy for Quality Assurance, declaring Mission and values, and defined the main processes the School, that are overall summarized in one macro-process, which is to make a specialist in Pediatrics out of a graduate in Medicine. Then, a wide corpus of documents has been produced, constituted by a Handbook of the Quality, a Document of Process, 10 organizational procedures and 23 operating instructions. The overall process was fragmented in a series of interlinked processes, that were clustered into three categories (the “main processes”, meaning the ones intimately related to the training activities; the “supportive processes”, which, even if not contributing directly to the training itself, are essential for the appropriate realization of the main processes; the “general processes”, the ones offering general support to all the others in order to meet all the ISO 9001:2000 requirements). All this material was written and made publicly available. Furthermore a system for governing and periodically reviewing the teaching activities as well as for assessing the residents and the teaching staff was implemented. All this permitted to develop objective indicators to continuously monitor the functioning of the training program and to set improvement actions. The results obtained by the evaluation of the activities of the School allowed the comparison between the period before the certification of the School and the period after (March-October 2008). Residents answered to 43 questionnaires on the activity of the School and 213 questionnaires on the training periods lead in the previous academic year (71% responses); moreover, 98 questionnaires on the training periods, post-certification (65.3% responses). The results of the retrospective evaluation on the training show a total level of satisfaction (defined like good/optimal judgments) to 63%, but less than 30% for the effectiveness of the frontal lessons, the planning of the activities of clinical training, the systems of evaluation, the communication of the outcomes of the evaluation and the availability of the teachers. From this analysis, the Residents Affair Committee has left for the definition of the objectives of improvement for the next three years (planning and organization of the theoretical didactics, improvement of the clinical training and practice, introduction of a new system of residents’ evaluation and feed back of the result of the judgment). After the certification of the School, the evaluation on the activities of the School (“customer satisfaction”) is not changed, even if there is a positive trend for two items: better definition of the formative objectives and better organization of the job (from 38% to 50%), and improvement on respect of the times of job (from 51% to 57%). In both cases the result goes toward the goal of 60% satisfaction. Both the retrospective judgments and the post-certification ones show that the worst aspects of the clinical training are the paucity of didactic meeting during clinial training, and the insufficient possibility to learn invasive procedures (levels of satisfaction are inferior to 25% both in the pre than in the post-certification period). After the introduction of the quality management system, the Responsibles of the Clinical Wards expressed 73 questionnaires of evaluation on the activity of the residents (cover to 48%); similarly, using the same questionnaires, the residents evaluated themselves (84 questionnaires on 150, equal cover to 56%). The agreement between external appraisals and self-made evaluation (calculated by means of k of Cohen on 33 joined questionnaires) is equal to 0,067: the residents critically judge their level of autonomy and their medical knowledges and give themselves worst evaluation than theirs tutors. Conclusions - The process of application of ISO 9001:2000 Standard to the management of the Pediatric post-graduate program of University of Padova represents an innovative approach toward the quality improvement, to met the action lines derived from the Bologna Process. The ISO 9001 requirements allowed to identify, to make explicit and to declare the intimate processes and their interactions underlying professional post-graduate pediatric training in Padua and provided a system to control, monitor and improve them. All this, in brief, meant a better governance. In our experience, this approach has contributed to a diffuse culture for recognising the importace of the quality in everybody’s work, supported from an analogous quality management system, adopted by the clinical Services of the Children’s Hospitals where residents work. By adopting the concepts of monitoring and measurement of prefixed objectives, this quality management system lead the School to identify points of force and areas of weakness, and to set up concrete actions of improvement. The planning of a new system of evaluation through computer science support has been crucial, because it has allowed to obtain results of customer satisfaction. They have identified a good degree of satisfaction on regard of the formative activities (already before the introduction of the quality management), and put in light criticities (low effectiveness of the frontal lessons, of the planning of the trainings anf of the modality of evaluation); these items became new objective of improvement for the next three years. This experience has some important limits: to plan a system of management in compliance with ISO 9001:200 requirements has been really expensive in terms of human resources and of hours of job, so as the implementation of this system and its successive maintenance, that needs continuous reiforces for its manteinance. Moreover, the first results of the monitoring of the indicators demonstrate that changes are slow and still little perceived, for which these first elaboration of customer satisfaction must be judged like a preliminary matter, and indeed it is premature for being able to express a realistic judgment on the impact of the certification ISO on the quality of the School. In conclusion, for the first time in the post graduate medical education, ISO 9001 requirements have been introduced like tool of planning, control and improvement of the processes of formation, adapting them to the peculiarities and the requirements of the formative medical process. This experience of quality assurance and quality improvement is not proposed as a gold standard to apply to similar programs of formation, but as a possible referring example for developing the action lines of the Bologna Process. At last, this project of the pediatric training program of Padova remembers that the responsibility for quality assurance in higher medical education lies with each single institution, whatever are the strategies of quality adopted.