The Evaluation Process

The evaluation procedure is detailed in the regulation, available here in English translation. Sections which are not relevant for the expert evaluators have been greyed out of readability. A summary description of the main points is also available below. 

The self-evaluation reports and multiannual development plans of all institutes in the winter 2011-2012 evaluation wave are available here.

Preparation

The institutions upload the summary table, the other quantitative characteristics and the self-evaluation report and the development plan via the online platform. Experts will have access to the documents through this page, for a period of at least two weeks before the visit, in order to become familiarized with the institution. They may also interact via email, or via the online platform in order to exchange first impressions and to organize and plan the visit.

Before the visit the experts may request additional information from the institution, by electronic means, mediated by the personnel of the National Authority for Scientific Research. The visit cannot be longer than 3 workdays. After the visit, the evaluators have 3 weeks to finalize a preliminary report. After feedback from the institution management, from the coordinating authority of the institution and from the Advisory Council, the experts submit the final report. The final report may be approved or rejected, by vote, by the Advisory Council.

Grading

The team will give informal grades and short comments for each of the 26 subcriteria detailed in the Evaluator’s Sheet.

The team will give formal integer grades between 1 and 5 for each of the following main criteria:

       a)      The quality of the results of the research activity.

       b)      The quality of the human resources.

       c)      The quality of the infrastructure and its level of exploitation.

       d)     Managerial efficiency and the quality of the research environment.

       e)      The quality of the institutional development plan.

For each institution the final grade will be calculated as an average of the 5 grades for the criteria. The classification is accorded as follows:

a)      For a grade greater than or equal to 4.5: A+

b)      For a grade greater than or equal to 3.5 but less than 4.5: A

c)      For a grade greater than or equal to 2.5 but less than3.5: A-

d)     For a grade greater than or equal to 1.5 but less than 2.5: B

e)      For a grade below 1.5: C

f)       Also, for institutions that have obtained a grade of 1 for at least 2 of the criteria, the classification will be C, by exception from a)-e).

The visit

The visit is preceded by a meeting of the experts in order to lay out the details of the visit and the tasks assigned to each member. The representative of the Advisory Council and the representative of the coordinating public authority under which the institution functions will provide the evaluators with general information regarding the intensity of the research activity in Romania in the relevant fields, recent public funding history, sectoral policy in the field of the institution and other relevant information regarding the legal, economic and labor market context in the field of the institution.

The visit is composed of the following steps:

a)   A session open to the public with the director of the institution and his or her management team; the presentation will include general aspects concerning the structure, resources and results of the institution, but with a special focus on a single, major, representative research project undertaken and finalized within the institution.

b)   A discussion with the members of the scientific council.

c)   Discussions with each of the research teams presented in the personnel list submitted by the institution.

d)   If deemed necessary, discussions with the auxiliary, technical or administrative personnel.

e)   A discussion with representatives of the doctoral students, if any.

f)   In exceptional cases the expert team can interview individually any person whose contribution is deemed useful. Such interviews will be limited to not more than 20 minutes.

g)   A final closed meeting of the team of evaluators, within which the members decide on the informal and formal grades,  as well as the classification which results. The formal grades and the classification are recorded in writing at the end of the meeting, within a declaration signed by all the expert evaluators;

The two representatives may participate in the discussion at f) or g) only with explicit approval of the team members. If they participate they may contribute only in order to clarify administrative, legal or procedural aspects relevant for the functioning of the institution, of the national research system, of the Advisory Council or of the evaluation procedure, and only with the approval of the leader of the team of expert evaluators.

The final report

 A schematic model of the report is available here. For the detailed explanation please see below.

The evaluation report is elaborated under the coordination of the leader of the team of experts and is approved by the vote of the team of experts. The report is written in English, with the exception of institutions in the field of Romanian language and literature, for which the report may be written in Romanian.

For institutions for which the proposed classification is A- the length of the report cannot exceed a number of pages 5 larger than the number of research teams in the institution. In other cases, the length of the report will be limited to a number of pages 3 larger than the number of research teams in the institution.

One page of the report will present an executive summary of the evaluation and the main conclusions, including the grades for the main criteria and the final overall grade.

At least 2 pages of the report will argue and detail the given grades and classification, with particular attention to the strong and weak points on each of the 5 major criteria, and also comments and suggestions for the 26 subcriteria.

For the case where the proposed classification is A- the report will present a set of objectives and actions aimed at improving the weak points identified, in view of implementation for the following 2 or 3 years.  This section cannot occupy more than 2 pages and must specify the timeframe foreseen for the implementation of the objectives and actions: either 2 years or 3 years.

The report will avoid a descriptive presentation without a real analysis of the strong and weak points, ambiguous or general formulations which would lead to a confused picture of the situation in the institution, or an exceedingly diluted or cosmetic presentation which would be inconsistent with the grades given.