Jakub Kůdela of the Faculty of Mechanical Engineering of Brno University of Technology decided to point out a systemic error in his own field. An expert in optimization models and algorithms noticed a major problem a year ago when comparing and analyzing so-called evolutionary algorithms. And he found out why even seemingly correct algorithms do not actually work as they should. A scientific article on this topic has now been published in a prestigious journal from the "family" of Nature, specifically Nature Machine Intelligence.
Evolutionary algorithms are one of the methods of artificial intelligence. They are based on biological principles, inspired by the mechanism of evolution and natural selection. "They are used for tasks where you need to set parameters appropriately, but at the same time the task lacks a clear structure, it is more like a black box. In recent years, evolutionary algorithms have become popular for tackling these tasks. Some of them work great, but others don't. All these methods are tested and compared on a predetermined set of problems, professionally we call it benchmarking," says Kůdela, who works at the Department of Applied Informatics of the Faculty.
In the case of the older, but still widely used set for the above-named benchmarking, there has historically remained a group of tasks for which it is optimal to set the searched parameters to zero. "Which does not matter in case of well-functioning algorithms, they work the same way even if you move the task from zero to a different value. But others tend to search for the optimum at zero. I call this "zero-bias": within the space, which they are searching as a part of the solution, they are driven to zero. And such algorithms do not work as they should in practice. The problem is that when you use an older benchmarking set, it is hard to tell the good and bad ones apart, because at first glance you cannot see how the algorithm arrived at the result," Kůdela explains.
As a result, he found that a number of new methods and the research based on them published in respected journals, contained a fundamental error. "Over the past three years, I described seven methods that contain this error," Kůdela clarifies, and he is aware that he is destroying the imaginary house of cards for many colleagues in the field. "However, that is why I wrote my article. I hope this will change the way these methods are developed and validated. And maybe it will even reduce their number to only the real functional ones," he adds.
Kůdela is not the first to notice the problem, he has already come across two cases in the literature where one of his colleagues pointed out a discrepancy in a particular publication. But only he realized that this was not an individual excess, but a systemic error. Therefore, he offered the topic to the best journal in the field, which has the greatest potential to reach the relevant scientific community. The journal Nature Machine Intelligence is a world leader in computer science and artificial intelligence. According to the Journal Citation Reports, it ranks first out of 144 journals in the "Computer Science, Artificial Intelligence" category and first out of 113 journals in the "Computer Science, Interdisciplinary Applications" category.
"The whole process was surprisingly pleasant and smooth, from the first draft I sent in January, we had a full five-round review process ready until November. Unlike other professional journals, I was impressed by the approach of editors, who write their explanation for each reviewer's note, why they think the note is important and how relevant it is. They also take great care to ensure that the tone of the text is scientifically completely correct and they cross out some of the stronger expressions," Kůdela concludes with a smile.
The article "A critical problem in benchmarking and analysis of evolutionary computation methods" can be found in full on https://rdcu.be/c1tmf.