Another step towards a fair assessment of research projects comes again from the private sector: we are talking about the Cariplo Foundation. On the wake of Telethon, which opened the road by importing the peer-review system inspired by the U.S. National Institutes of Health and considered the best in the world, the merit-based and impartial model is taking place among private "funding agencies".
The Cariplo Foundation, officially founded in 1991, represents the historical continuation of the Cassa di Risparmio delle Provincie Lombarde established in Milan in 1823 with philanthropic aims and active in four areas: Environment, Arts and Culture, Scientific research and Services to Citizens. Its activity is based on income arising from the management of its assets, valued at approximately 6.36 billion euro, out of which the Foundation allocates about 190 million euro per year to non-profit entities.
The selection of projects to be financed has an extraordinary strategic importance for all charities and the Cariplo Foundation, at least in the field of Scientific Research, in recent years has set up an ex-ante evaluation system aimed at ensuring that the money invested produce value and scientific results. The recent publication "Project selection and relevance of scientific results" analyzes the results obtained in order to verify, ex-post, that the investments have been scientifically productive.
The selection process in Cariplo Foundation is carried out according to merit-based methods which have been refined over the years also taking inspiration from other national and international situations and includes a call for proposals, an initial assessment by the Scientific Office which assesses eligibility and compliance with the call specifications and a second step carried out by external reviewers, the actual peer-review phase.
In the 2010 call, for example, about 200 foreign and Italian reviewers were involved, among those holding permanent working positions for over 10 years abroad, in order to prevent any possible conflicts of interest. Each project was assigned to three independent reviewers who worked through remote connection and who provided their written evaluations on the basis of specific criteria that have different weights in the composition of the final score. Indeed, the Cariplo Foundation has set up a special system to develop a final score, which is the weighted average of the votes of the three reviewers based on their self-declared specific knowledge of the area of the project. Each project is ranked on the basis of the final score and the Board of Directors of the Cariplo Foundation disburses the funds allocated for the call until fully used up. The system to arrive at the final score is complex, and is probably meant to compensate for the absence of the plenary meetings or "study sections". These meetings are normally the crucial phase of "peer review" selections, because they allow a limited number of international, competent, impartial and independent auditors to receive the external reviewers' evaluations, to have an overview of all the competing projects and to avoid a fragmented evaluation and the application of different voting scales. The Cariplo Foundation does not use them for various reasons, including cost.
The ex-post analysis of funded projects is complex and detailed, and the part relating to scientific publications refers to 120 projects approved from 2001 to 2007, amounting to approximately € 19.5 million in grants. Out of these 120 projects, 545 works were published and indexed in the Web of Science (self-declared by the project managers, therefore potentially subject to an opportunistic increase) and, according to Cariplo Foundation, these documents have had an average of 23 citations per article in the analyzed period, and especially the ratio of actual citations to expected citations has always been greater than 1, whether data are analyzed by magazine or by sector, which proves the good quality of these scientific publications.The most interesting data, however, come from the national and international scientific community. The publications resulting from the funded projects in 10 of the most important areas of biomedical research have received a number of citations more than double than the Italian publications, exactly twice the publications of the OECD countries and equal to 1.8 times the publications of the 8 most industrialized countries.
The figures of both the projects and the analyzed publications are small but the trend is positive. The peer review merit-based system designed to exclude conflicts of interest, biases, and self-referencing, is slowly catching on here too, at least in the private sector.
To download the Cariplo Foundation Notebook 6 dedicated to scientific research click here.
