Again this year, the Telethon marathon ensured 30 million euro (a little more is is expected in June with the closing of the balance sheet) most of which will be used to fund cutting edge research on genetic diseases. A good economic result, therefore, with some concern due to the growth in applications for grants by almost one third, many of which are polycentric. "It's a pity because we will have to be more selective, but this also shows how much potential research there is in Italy", said Francesca Pasinelli, general manager of Telethon, interviewed by Scienzainrete on recent controversies over calls for applications in research and evaluation methods.
As regards polycentric research, the minister Mr. Profumo was criticized for having focused the recent calls for applications, PRIN, precisely on the polycentric nature of research. In order to be eligible, applications must include at least 5 different research units. What is your opinion?
I agree with some of the criticisms made after the publication of the call. It is odd, for example, that a maximum number of eligible applications be set for each applicant university. As for polycentric research, it is often true that the most innovative ideas come from the research work of individual researchers or groups. However, after carefully reading the technical document accompanying the Prin calls for applications I got a clearer picture of Minister Profumo's intentions, who rightly says that the new European research programme Horizon 2020 is going to start shortly (2014 is just round the corner) (with a budget of 80 billion euro). Italian researchers will therefore also be asked to participate in polycentric projects and will have to learn to do it a little bit better than they are doing now. Given that - as noted by the minister himself - we participated in the 7th Framework Programme with 15% of resources and brought home only 8,5%. A sign that we can improve. But it takes training.
Critics, especially from the academic milieux, say that universities are not equipped to handle these processes. Or that they follow logics that are not quite based on merit.
The fact that Italian universities are not equipped with an internal screening system for research, and a strategic governance for it, is a very serious flaw; it means giving up one of the most noble university functions. Mr. Profumo is therefore right in asking that universities begin to get equipped with structures capable of establishing priorities, programming research and accordingly choosing the researchers that are most appropriate to these objectives. This call gives universities strategic responsibilities that they must be able to develop from the inside.
The bureaucracy is not helping, including European bureaucracy.
However, let's not put too much blame on others. It is our environment that is still weak and not very competitive. Meanwhile - and the Telethon experience is there to teach us - - we must learn how to prepare the proposals even with the help of specific skills. There is not much training in this respect: projects are sometimes poorly presented, with excessive background information, too many promises and perhaps preliminary information is not provided in a transparent way. All these things are immediately noticed by the evaluators. Grantmanship must be built and is decisive. In addition, polycentric projects are often considered as a "roped party" where the best pull the dead weight. What a horrible thing. On the contrary, if you want to win you have to rely on a real team effort, in which the sum is greater than the individual parts.
How do you learn this?
Through an organization that leaves room to people who know how to coach. Foreign universities do not have ad hoc roles for that purpose. They are researchers who have developed this role, which also includes managerial and administrative functions. Teamwork, actually, also lacks in these major calls for applications. One does not perceive any underlying team work by evaluation experts managing the project in an informed manner. The proper, large research agencies (such as the U.S. NIH) have high-profile administrative roles who can follow the projects from beginning to end. For example, they know how to channel the projects to the right reviewers.
In other words, a Register of Evaluators is needed.
For goodness sake, that is absurd ...I believe the peer review developed by Telethon is a good example. Our scientific office consists of seven people who choose the reviewers one by one among still active researchers and check that the assigned ratings are correct and not biased, for example, by personal or unscientific factors. Foreign reviewers are often chosen, not because we are xenophiles, but because in such a small community as the Italian one, it would be hard to find reviewers at the top of a discipline who are not also the authors of the projects.
In a recent call for applications (by the former government), the Ministry of Health had all projects evaluated by foreign reviewers.
Actually, in that case the whole review process was outsourced to the NIH, a practice which to be honest I found a bit provincial. It was like saying that the government outsourced the entire research and evaluation strategy of such a strategic sector as biomedicine to another country.
Going back to the Telethon method, what are the other prerequisites for a good evaluation?
The anonymous peer review is accompanied by two study sections, i.e. plenary meetings in which the reviewers discuss the projects and often even change their opinion on them. Except for the worst (about 30%) and the best (another 30%) projects, for which opinions are often unanimous, the plenary meetings are needed to reach a consensus on the other 40% of the projects on which there are conflicting assessments. Being face-to-face and supporting one's own evaluations before thirty some of your colleagues (all with voting rights) changes things quite a lot ... for the better, I would say. At the end of this process, we provide researchers with both the initials evaluations of the four reviewers and the outcome, in summary form, of the plenary meeting. I think they may find it instructive too.
Is it unthinkable that mechanisms and expertise of this kind also be adopted in the public sphere, in the ministry itself?
Managing competitive calls requires precise technicalities. The lack of awareness of these mechanisms, which are quite renown, is bothersome. One may just look at how evaluations are managed in centers such as the NIH and the ERC. Currently, the ministry of research is not equipped for this task. Briefly, an organization and a strategy are necessary which could certainly be set up, maybe starting from universities, as Minister Profumo seems to be hoping for. Certainly, even the Advisory Boards seem to be missing , i.e. permanent panels that can establish research priorities at a national level, rather than dragging behind European calls.
The set up of Anvur (Italian Agency for University and Research Evaluation) seems like a step forward towards establishing good evaluation rules and promoting merit.
Certainly, the Anvur is a first good news. Although the actual most relevant and effective evaluation must take place ex ante when the funds are disbursed, through an in-depth analysis of research projects and curricula. In my opinion all this emphasis on bibliometric indices is a bit pathetic; it is as if the H-Index were the magic wand to distinguish between good and bad researchers. It is definitely an indication, but it is not sufficient. For example, a very high H-Index of a researcher who works in a scientifically modest university may even testify to the detriment of that person, who has not been able to set up a team or has preferred to romp in a mediocre setting rather than paying his dues in a center of excellence. Scientific quality is very much a matter of teams, where there is free flow of ideas, organization and leadership.
