Mendive. Revista de Educación, january-may 2019; 17(1): 109-121

Translated from the original in Spanish

The observer of the National Accreditation Board in the external evaluation processes of the universities

El observador de la Junta de Acreditación Nacional en los procesos de evaluación externa de las universidades

Maricela Maria González Pérez¹, Fatima Addine Hernández², Edith González Palmira³

¹University of Pinar del Río " Hermanos Saíz Montes de Oca". Office of the Vice President for Computerization, Research and Postgraduate. Cuba. Email: maricela@upr.edu.cu
²National Accreditation Board. Cuba. 
³University of Matanzas. Cuba.

Received: September 9^th, 2018. 
Approved: January 1^st, 2019.

ABSTRACT

In the analysis carried out in the plenary session of the National Accreditation Board on the processes of external evaluation of the Institutions of Higher Education in Cuba, a group of difficulties was identified that recommended the incorporation of a member of the same as an observer of these . From the analysis of the reports presented by each evaluator and the assessed in the plenary session of the National Accreditation Board, this work was proposed to systematize the experiences reached from the year 2016 to date and to propose a procedure that guides the work of future observers . The practice demonstrated the utility of the observer in terms of ensuring compliance with the regulations and procedures established by all the evaluators, while also leaving important recommendations for future processes.

Keywords: accreditation of Institutions of Higher Education (IHE); external evaluation of IHE; evaluation.

RESUMEN

En los análisis realizados en el pleno de la Junta de Acreditación Nacional sobre los procesos de evaluación externa a las Instituciones de la Educación Superior en Cuba, utilizando el método de nivel teórico histórico lógico con el auxilio de la técnica del análisis documental permitió identificar un grupo de dificultades que recomendaron la incorporación de un miembro de la misma como observador de estos. A partir del análisis de los informes presentados por cada evaluador y los análisis en el pleno de la Junta de Acreditación Nacional, este trabajo se propuso sistematizar las experiencias alcanzadas desde el año 2016 a la fecha, con el objetivo de proponer los fundamentos teóricos del trabajo del observador, así como, una metodología que oriente la labor futura de los mismos. La práctica demostró la utilidad del observador en función de velar por el cumplimiento de los reglamentos y procedimientos establecidos por parte de todos los evaluadores, dejando además importantes recomendaciones para la realización de futuros procesos.

Palabras clave: acreditación de Instituciones de la Educación Superior (IES); evaluación externa de IES; evaluación.

INTRODUCTION

The regulations that are in force approved by the National Accreditation Board (JAN) do not include the participation of a board member as an observer in the external evaluation processes of the Institutions of Higher Education (IES). However, the practice in the last two years has evidenced the need for its incorporation.

This paper aims to analyze the first results of the follow-up of the external evaluation processes of HEIs based on the JAN observer's function, so that there is empirical evidence that makes it possible to verify compliance with the procedures used and their transparency, as well as knowing the repercussions of the analysis carried out after the external evaluation.

The activity of the JAN observer, for the fulfillment of its function of verifying compliance with the norms established for conducting external evaluations to higher education institutions, becomes an exploratory-descriptive research exercise, from a predominantly qualitative perspective, ethnographic. More than quantifying it is important to describe, interpret and explain the process to improve your understanding and especially the work to be done. Under this perspective, the methods of compilation, observational registration, questionnaires and interviews were selected, although more quantitative instruments can be used in future studies since to date there has been no structured procedure to carry out this work.

In the same way, we investigated the relationships between the participating subjects and their objects. The practical and intellectual actions with which they constructed their representations and meanings were analyzed, as well as the sense that for them assumes the process of being a member of the JAN in general and of observer in particular, with the purpose of contributing to their understanding; so that those who intervene in it, especially those in charge of the direction and orientation of the process, identify the tasks and the moments in which their help is essential.

From the result of a process of recording and critical interpretation of the evaluation practice developed as JAN observers, some foundations are laid out to build a theoretical model that will guide the development of the methodology to be used for the empirical inquiry that it develops during the evaluation process and that they can contribute to the preparation and better performance of these observers.

The following question arose from the exploratory analysis explained above:

Which and how are the actions that constitute the training activity carried out through which a member of the JAN makes sense of the experiences, knowledge and modes of action with which he builds his identity?

MATERIALS AND METHODS

In the development of the research the theoretical level methods were used: logical, systemic and modeling history with the support of the techniques of document analysis and the procedures of analysis and synthesis and induction and deduction.

First, the research needs associated with the tools and procedures used by the evaluators to carry out their work were determined. We worked with the reports of one hundred percent of the observers who acted between 2016 and 2017 in external evaluation processes to higher education centers in the country, as well as with current regulations.

RESULTS AND DISCUSSION

The practice is based on knowledge, but also produces new knowledge. When you intervene in reality, it is possible to obtain - if you reflect on it - a rich and deep knowledge about it, the subjects with which you interact, the intervention strategies, and about yourself. However, the meaning of the knowledge produced in and for practice and, therefore, the way it is produced, are different from those of other types of knowledge.

Main findings and lessons learned by observers in their reports

The learning was carried out jointly with other participants, whose experiences, visions and interests are diverse, as well as their forms of intervention and obviously, their interpretations of the practice and its effects. In terms of knowledge about reality, their ways of generating it also differ, resulting in different products.

First, the observer gives a name - identifies - the problematic situation that is presented to him, then explores and decides what makes it problematic (what gives it the character of it) then postulates the causes of Problems that he has defined, this allows him to specify what he wants - and can - change in the situation and finally, he postulates what actions could modify it upon his return to the HEI and in the JAN Plenary.

In the reports of: Cadenas , F. (2015), Crespo , L (2016), García , JL (2017), González , E (2016), González , M (2016) , González , M (2017) , Mestre , U (2016), Romero , RE (2017), in general the most repeated approaches were the following:

• It is necessary the previous analysis of the self-assessment report, the program to be developed in the HEI in order to develop a plan for participation in the activities. For what is important to maintain exchanges with the direction of the commission and with the HEI to be evaluated.
• The plan must provide for interacting with the greatest number of strata: institutions, students, managers, teachers, researchers, workers, organizations, among others. This plan must be operative and flexible, which is systematically updated according to the evaluative practice that serves as a guide for action during the external evaluation process and as a guide for the rapporteurship of the activities developed and their learning.
• Starting the observation by the variables, in particular by the variable I, this facilitates quickly to obtain an overview of the projection of the institutional management.
• Develop the ability to anticipate, be proactive to alert the Evaluation Commission of possible assignments or negative trends in the evaluation process (number of classes observed, main indications, quantity and content of the suggested methodological activities, among others).
• Strengths and weaknesses not identified by the HEI were identified by the evaluating pairs, and it was found that weaknesses that were not identified were identified, based on the evidence analyzed. Sufficient evidence was collected that allowed for the preparation of a report that synthesized and highlighted the essential strengths and weaknesses of the institution. However, the work of reconciling the strengths and weaknesses between the evaluating pairs and their counterparts is a process that demands the expertise of the evaluators and the knowledge of normative documents in the work of higher education institutions.
• Need for verification, based on permanent feedback, the results of the participant observation, exchanges and the application of instruments, the correspondence of the strengths and weaknesses identified by the Evaluation Commission with the evaluation practice lived.
• It was recognized that there was rigor, professionalism and skills in the selected experts, as well as the negotiating capacity of the evaluators with those evaluated.
• Deepen the theoretical-methodological reference that allows the identification of impacts: when there are results, effects, transformations and sustainability.
• The core nature of variable III is ratified in the Institution Evaluation and Accreditation System (SEA IES) and as a nucleus for the exchange and triangulation of information.
• Appreciate different approaches or points of view to focus work (variables, faculties, and study centers), strengths and weaknesses.
• Professionalism of the management team in the conduct of the process and of the evaluators, proactivity, tolerance, perception of their role and the responsibility that this process entails and their future involvement in the improvement of the institution where they work.
• Include testimonies from the evaluators or those evaluated that provoke reflection actions aimed at improving the evaluation practice in the JAN observer report.
• The observer role contributes to the preparation as a member of the JAN plenary and to the best performance of the advisory functions in the IES of origin. Therefore, it allows experiencing a situation of change and rapprochement, re-significance, construction of the representation of external evaluation processes, as well as the creation of a new professional relationship where learning and socialization are identified and interpreted.
• Expose significance of the JAN and the external evaluation and accreditation processes in all collective activities briefly, by evaluators and members of the JAN.
• Give differentiated attention to experts with less preparation, before and during the process.
• It is agreed that, for the daily meeting, the objective of the activity should be made more explicit from the beginning; deliver in writing, every day, the information that must be given, even if it is shared with everyone orally. This would allow collecting the information for the final report, systematically and complementing it or specify it in the final day.
• Suggestion to recruit as JAN experts to directors of Municipal University Centers (CUM) with experience and successful results to act as external evaluators in the processes of institutional evaluation, so as to guarantee a true peer evaluation. At least one in each evaluation planning the visits in a staggered way and not as it has been done so far, all CUMs the same day.
• Incorporate into the regulation that each variable, faculty and CUM visited must prepare their own plan of activities based on what is established; as well as verify (surveys and group interviews to be carried out, areas to visit, among other aspects).
• Explicit in the documents that the presentations, both that of the rector and the variables and faculties, are not to read the strengths and weaknesses that are in the report, but to explain the work systems that guarantee their obtaining and sustainability.
• Expand the methodological indications for the development of improvement plans by HEIs.
• Need for the verification of all aspects raised in the self-assessment reports to be easily verifiable by the evaluators.
• It became clear that when an IES has identified a weakness to be accepted the proposal by the evaluators, it is necessary to have quantitative indicators, eg number of doctors in the cloister, number of projects I + D + i Total and by categories, number of publications in groups I and II, etc; since analyzes are difficult when speaking in terms of low or insufficient.
• Need to refine the terms used in the annexed models for the collection of quantitative information, particularly those that are related to the research and extension processes.
• Although it is an achievement to have a compendium of instruments for the verification of evidence from interviews and surveys by variable, the need to make an integration process between them to avoid duplicating their application by evaluators was highlighted. Nor do meetings with the same participants with different objectives, as well as verify the degree of application of the same by the evaluators.
• The models that are annexed in the SEA-IES provide necessary information, but in the SEA-IES Pattern the time of reference to them must be made explicit, on the one hand, and on the other complement them with the models that in the IES are auditable.
• Record from the evaluative practice the reflections that emerge, which may become modifications to the SEA-I and its link with the rest of the SEA.
• Maintain, from a strategic vision, the conformation of the commission and the evaluating pairs in a balanced way by experienced evaluators and institutions that must face evaluative processes soon, including incorporating rectors or other personnel from the strategic apices of the institutions or reserves of these.
• Do not confuse strengths with functions. They should not be written descriptively, very generally, on absolute occasions and do not focus on impact. As for the weaknesses, they are aspects that limit or reduce the capacity for effective development of the organization's strategy. Strengths are written mechanically.
• Maintain the delivery to the evaluators of the views of the Directorates of the Ministry of Higher Education (MES) on the self-evaluation report for their study, never as indications. (See Article 19 of the SEA-IES Regulation).
• Disseminate and include methodological procedures for the preparation of improvement plans for verification in expert preparation workshops.
• Project systematization and socialization actions from all observer participation records.
• Develop procedures to ensure organizational, control and reliability measures in the application and evaluation of the exams applied. Paying special attention to the percentages of disapproved by instruction No. 1, on the spelling discount.
• Need for a negotiating capacity as the competence of the evaluator to influence the evaluated one from a position and equality, as a couple, in the debate and exchanges in the evaluation practice. This allows the evaluated ones to discover the essences of strengths and weaknesses, only achieved if experts have management experience at similar levels, especially if they are in the current exercise of it.
• The leadership survey, due to the need for its completion, it is suggested that it not be included in the evaluation program, but that it be mounted independently, so as not to confuse the evaluated themselves, since it does not pay information for the process directly and that may affect, be affected or be perceived as affected by the institutional evaluation process.

As previously stated to date, there is no procedure for carrying out the work of the observer. The previous collection was due to the considerations issued in this regard in the JAN plenary session, to a first proposal of guidelines developed by González , E (2016 ) and above all to the initiative of each evaluator.

Taking into account the accumulated experience, as well as the work of the Management Quality Group (GCG) of the Center for Education Studies of the Central University of Las Villas (2016), a procedure proposal is presented below, from the perspective of the result of a process of recording and critical interpretation of the evaluation practice developed by the observers in evaluations external to HEI.

As a first result, this action facilitated the identification of the following starting points for a JAN observer:

a) The starting points for a JAN observer are:     

• Self-preparation of the evaluator before, during and after the institutional evaluation.
• Flexibility
• Proactivity
• Transparency
• Empathy with the evaluated and evaluators.
• Objectivity
• Observation of the negotiating capacity of evaluators.
• Understanding of what a strength and weakness means and the improvements related to the quality of the management of HEI.
• Avoid confusion between management control (function of the MONTH) and evaluation of management quality (objective of the Institutional Evaluation).
• Differentiated orientation to pairs according to their performance.
• Use of dialogue for exchange and possible conflict resolution.
• Recognize that evaluation practice has intrinsic learning based on improvement.

b) Methodological design for the work of the JAN observer

Corresponding to the question asked the results and contributions of the observers in the processes carried out and the recognized starting points, it has been possible to identify a group of elements to structure a methodology that guides its work.

The general objective of this methodological proposal is to diagnose compliance with the established norms for the realization of the external evaluations to the institutions of higher education, to validate the process and propose actions for its improvement.

For such research, the following are defined as categories, subcategory of analysis and indicators (Table 1 and Table 2):

Table 1 - Category: Subjects participating in the external evaluation  

Table 2 - Analysis category: External evaluation processes 

The fundamental methods and techniques to be used can be:

• The participant observation of the JAN observer. The participant observation is considered that which the observer performs in activities in which he must be part of the evaluation team and in the performance of his duties. Fundamentally, it refers to the initial meeting, the report presentation meeting and the conclusions of the external evaluation, as well as those that it carries out as part of its individual plan to obtain information outside the processes carried out by the evaluators.
• The non-participant observation of the JAN observer is considered as the one performed by the observer in activities that he attends to see how the processes and the performance of the members of the commission are developed and in which it is not included as part of the subjects that They participate in it.
• Questionnaire for evaluators according to agreed formats.
• The in-depth interview with the direction of the evaluation commission and executive secretary JAN, the evaluators and students, workers and representatives of the institution responsible for the attention and contribution of information necessary to evaluate the institution by variable areas.

In each case, it has been possible to design each of the techniques and tools in order to gain homogeneity in the future work of the observers.

The experience of linking to the team of evaluators in the processes of external evaluation of HEIs a member of the JAN Plenary as an observer was enriching and is of extraordinary value, since it contributed to his better preparation as a member of the JAN, as well as, to the job advisory function for continuous improvement and during the institutional evaluation process of the HEI in which he works.

The JAN observer's responsibility lies in the fact that the evaluation process proceeds according to what is established in the rules and procedures of the SEA-IES, issuing value judgments about the quality of the evaluation practice, as well as recording and socializing its results.

It is necessary to take care of the figure of the expert or evaluator as a leading entity of the processes with the implementation of a strategy of selection, training and certification of their suitability and where the JAN observer can, in the process of external evaluation, accompany the member of the Technical Committee and to the Executive Secretary.

The proposed guidelines presented include the content of the regulations established for conducting external evaluations of Higher Education Institutions in Cuba, as well as systematizing some of the practical experiences in the evaluations carried out.

The proposed methodological design can be enriched with the contributions of JAN observers, at the time of carrying out their task and from the specific conditions in which they carry out their work, as well as from previous experiences. There is a previous conception or anticipation of the role to be played in the external evaluation space. Thus, the subject is prepared to adapt to the demands of the process.

The projection of the observation process must be based on transformation, constant movement, independent search and the use of information and communications technologies.

The result of these successive approaches to the definition of the functions and instruments for the fulfillment of the JAN observer's task, with their contribution, may contribute to the elaboration of a definitive document that guides the activity, which must be legitimized for the JAN.

Organizational learning requires orderly and systematic procedures, as well as mechanisms for the prevention of errors and the willingness to learn, which is equivalent to the willingness to improve.

BIBLIOGRAPHIC REFERENCES

Cadenas; F. (2015). Informe de resultado del trabajo del observador JAN en la evaluación externa a la Universidad de La Habana.

Crespo; L. (2016). Informe de resultado del trabajo del observador JAN en la evaluación externa a la Universidad Tecnológica de La Habana. CUJAE

García; J.L. (2017). Informe de resultado del trabajo del observador JAN en la evaluación externa a Universidad de Matanzas.

González; E. (2016). Propuesta de orientaciones generales para el trabajo del observador JAN en la ejecución de la evaluación externa.

González; E. (2016). Informe de resultado del trabajo del observador JAN en la evaluación externa a la Universidad de Pinar del Rio.

González; M. (2016). Informe de resultado del trabajo del observador JAN en evaluación externa a la Universidad de Cienfuegos.

González; M. (2017) Informe de resultado del trabajo del observador JAN en la evaluación externa a Universidad de Guantánamo.

Grupo de calidad de la gestión (GCG) del Centro de Estudios de Educación de la UCLV (2016). Procedimiento para el perfeccionamiento del observador JAN.

Mestre; U. (2016). Cuestionario elaborado para valorar el resultado del trabajo del observador JAN en la evaluación externa a Universidad de Ciego de Ávila.

Romero; R, E. (2017). Informe de resultado del trabajo del observador JAN en la evaluación externa a Universidad de Guantánamo.

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
Copyright (c) Yanaisys Caro Martínez, Reinaldo Cueto Marín, Sulany Sánchez Curbelo