A report by the European Science Foundation (ESF) offers numerous and excellent advice which can be applied when assessing research project proposals. The European Peer Review Guide is primarily intended to help European financing bodies to improve and reconcile their assessment procedures, however, it can also serve as a guide to facilitate young scientists when preparing research projects. The guide offers an overview of the various types of financing programs underway within Europe. The most popular options are listed: ranging from management of disputes to confidentiality of documents right up to how to organize the work of referees. The European Peer Review Guide was created with the contribution of 30 national research foundations across 23 counties, amongst which, the European Research Council (ERC). The document can be divided into two parts: the first consists of a general overview on the peer review system, the second part provides specific instructions on the guidelines to follow. Upon closer inspection of the guidelines, seven fundamental principles are provided to ensure a correct revision and assessment process:
Excellence: the projects selected for financing must demonstrate high quality, applied to the themes and criteria satisfying the competition requisites.
Impartiality: all proposals presented must be examined in the same way. Assessors have the task of assessing only merit, disregarding the identity of applicants. Financing organizations and assessors must not discriminate in any way in light of sex, age, ethnicity, country of origin or social standing, based upon religion or personal beliefs.
Openness: decisions must be made based upon clearly described rules. All candidates must receive suitable feedback on the assessment results pertaining to their proposal. To avoid discretionary decisions, effective and open communication represents a crucial element to safeguard the integrity of the system.
Suitability to the objective: the assessment process must be suited to the bid call type and the target research area.
Effectiveness and promptness: ensuring the presence of both promptness and quality during the process is extremely important.
Confidentiality: all proposals and relative data must be treated as confidential.
Fig.1 - Peer review assessment stages (European Peer Review Guide)
SELECTION OF ASSESSORS
A fundamental stage in the peer review process is the selection of experts who will examine the projects. Dependent on the model adopted for the assessment, specific decision-makers have to be selected. Many research establishments have a database which is used to classify, dependent on skills, the candidates. A survey was carried out, demonstrating that 90% of the organizations uses an IT system during various stages of the classification process. There is a major tendency, all the same, to entrust internal sources to define the classification criteria. The guide highlights how these classification methods are not comparable; it is therefore suggested that common criteria are adopted when assigning expert scientific profiles so that they are used clearly and without confusion amongst the different organizations. The simplest and most used way when making the selection, relates to "correspondence in terms of the subject"; experts who are most familiar with the subject in question are selected. Other organizations rely on classification which sees the association between a keyword which features in the research proposals and the assessor profiles. The first method may not be effective in identifying qualified persons where there are multiple or inter-subject projects to be examined, whilst the second method will only identify researchers with research experience in specific categories. The best choice would be to unite and integrate both methods. The minimum number of referees depends on the peer review format, therefore, the number of proposals to be examined and the scientific range of the program. The objective should be to ensure the availability of different view points; at least three expert opinions should be provided to ensure correct assessment.
Conflicts of interest may be divided into two categories: intangible, in the case of academic relationships, or tangible, where the two parties have financial-type ties. The assessors must avoid assessing a proposal which presents them a conflict of interest; they must not be present when the proposal in question is being discussed. The rules relative to conflict of interest may vary depending on the assessment stage, or the role of the expert. There may be circumstances where these situations may be resolved or attenuated without completely excluding the user with the declared conflict of interest. For every proposal assessed, every expert must sign a statement of impartiality. It is also strongly recommended that international assessors and groups of experts in developing countries are used. The assessor must also sign a code of conduct which ensures confidentiality and prevents dissemination of information. Before the assessment process begins, it is important that information sessions are provided for referees, so that they are provided information and a description of the assessment criteria which must be employed. The research organization must assign some of their staff members as the points of contact for the experts. The communication channel should therefore always be open in order to respond to any type of issue that may arise during the assessment process.
Main parameters that referees should adopt when carrying out the assessment:
- relevance and expected impact;
- relevance of the research proposed to the competition field of application;
- perspectives with larger impact (scientific, knowledge building, socio-economic);
- associated risks;
- budget;
- personnel involved;
- equipment and consumable materials;
- travel;
- ethical issues: compliance with regulations and ethical standards relative to safety, the use of animals and human subjects;
- general opinion: some organizations promote equality between the sexes.
PROJECT ASSESSMENT
The scientific proposal must be clear and convincing. All projects displaying originality, both in terms of approach and new technologies, will be viewed positively. All the same, a survey carried out on peer review practices in the majority of European countries demonstrates that 70% of the financing establishments do not have specific tools to suitably assess innovative proposals, hence why, in approximately 50% of cases, it is rare to see research projects proposed which promise to change paradigms governing knowledge. The survey also presents the reluctance of investors to finance these types of project, owing to the fear that they will loose their investment; the guide recommends using experts able to judge the proposal and reduce the risk to a minimum. Examinations of CVs is essential in order to find out if researchers already have experience within the field of research in question and to find out whether the group of applicants has the necessary skills and connections to successfully complete the project. In order to summarize and compare assessments as best as possible, it may be advantageous to assign a scoring system for each of the criteria adopted. Many research establishments use multiple choice when compiling the assessment. Usually these assume a numerical or alphabetic format.
We recommend keeping this system as simple as possible, in order to easily calculate the average score from all scores, to achieve a final score. A threshold may be determined to separate the proposals that can be financed and those which are to be refused. Before sending applicants the referee results, organizations must be provided with the opportunity to study the referee assessments, after which, they must be sent back to the research groups. Candidates must always be informed of all process stages and relative time frames. Some research establishments include, as part of the assessment stage, the right to reply. This enables researchers to comment on errors or misunderstandings. The financing agencies are encouraged to simplify procedures, trying, when possible, to reduce the time taken for assessment and decision-making to a minimum. Dependent on the bid weighting, the assessment system can be divided into two stages. During the first stage, the candidates are invited to submit a letter of intention in which they briefly describe the project and the general objectives of the research. Those who pass this stage, must send the complete proposal during the subsequent stage, which will definitively be assessed. Some organizations include the possibility of candidate interviews or project presentations. In the case of national programs, 6 months is a good reference time period, and with international competitions, 12 months is considered an acceptable time period. It is very important to establish clear rules about the methods and the procedures used to make the final decision. In the majority of cases, the final decision lies with the internal committee of the organization responsible for the program. Other establishments organize a committee of scientists that, dependent on the instructions of the assessors, decides upon the destination of the financing. Very few research agencies see that the final decision is made only by the "assessor association". In order to guarantee the quality of the process, it is necessary to monitor the quality via specific indicators; personnel with explicit roles must monitor the entire process. A survey carried out demonstrated that only 6.7% of establishments have a dedicated office, with the explicit task of guaranteeing the quality of their organization.
