Gesucht: innovative Online-Suchdienste für Europa
Technologie:Daten
Forschungsbereich:Big Data & KI
Mehr als neun Milliarden Webseiten in 185 Sprachen durchsucht und so etwa 1330 Tebibytes (das sind etwa 1462 Terabytes) an Inhalten indiziert und katalogisiert: Auf dieser Arbeit baut der offene Webindex OWI auf, den das von der EU geförderte Projekt OpenWebSearch.EU (OWS.EU) im Mai 2025 veröffentlichte. „Dieser Index kann von Einzelpersonen oder Organisationen genutzt werden, um eigene Suchmaschinen zu entwickeln“, sagt Prof. Michael Granitzer von der Universität Passau und Projektleiter von OWS.EU.
Die technischen Grundlagen für neue, spezialisierte Suchdienste zu schaffen, die nicht nur die sprachliche und soziokulturelle Vielfalt in Europa widerspiegeln können, das war die Vision, mit der OWS.EU im Herbst 2022 startete. 14 Organisationen aus sieben europäischen Ländern – neben Universitäten, Forschungsinstituten und Unternehmen auch Supercomputing-Zentren wie das Leibniz-Rechenzentrum (LRZ) – erarbeiteten zusammen eine Infrastruktur zur Erfassung und Katalogisierung von Internetdaten. Ohne diese sind weder Suchmaschinen noch Suchdienste denkbar, die auf großen, künstlich intelligenten Sprachmodellen basieren. OWI ist das zentrale Ergebnis dieser Anstrengungen – im Gegensatz zu seinen kommerziellen Kollegen ist der Webindex öffentlich zugänglich und wie Open-Source-Software frei verfügbar. Außerdem ist OWI transparent aufgebaut, die Quellen von Suchergebnissen können leicht nachvollzogen und überprüft werden.
Web indexing is resource‑intensive
New services are primarily a question of “resources and costs,” as Granitzer explains. For OWS.EU, servers at several supercomputing centers were in constant use, crawling around 100 million web addresses per day. Even so, this represents only a fraction of the effort invested by commercial search engine operators such as Google, Bing, or Baidu. “To keep up, we would have to increase our efforts by a factor of 20 to 30,” says Granitzer. That would be feasible, “but we would have to hire staff to maintain the service and buy more storage.” Operating a web index 24 hours a day, year‑round is costly—and naturally beyond the scope of a publicly funded research project. Nevertheless, there is hope that entrepreneurs and investors will emerge who use OWI to build innovative businesses and services.
That OWI provides a solid foundation has already been demonstrated. Seven community projects funded by OWS.EU developed business ideas, search services, and tools for companies or organisations based on the web index – for example, a fact‑checking service for current topics or a tool for building online shops from data in enterprise resource planning (ERP) systems. The Know Research Center in Graz, which develops solutions for business and society from research results, built a health‑focused search engine using 200,000 websites indexed by OWI and AI models. Tilde does not rank results solely by popularity or reach, but by reliability: “For each search result, Tilde lists the sources and assesses trustworthiness,” explains project lead Dr. Michael Jantscher. “Users can also decide for themselves which information they consider more important—scientific studies, specialist articles, blogs, or social media.” Based on OWI, the service has become a blueprint for further thematic search engines as well as follow‑up projects: “We can reuse the experience and strategies from the OWS project for other tasks,” says Jantscher.
Like Tilde, the other use cases and technical solutions for indexing web content are intended to inspire founders and companies. And that can pay off—for businesses as well as for Europe and society. According to a study by the Munich‑based consultancy Mücke, Roth & Company, investments based on OWI could become profitable after around four years of operation. The profit the EU could gain from online search services, the resulting economic and social improvements, and increased technological competitiveness is estimated at around €4.5 billion.
Research also continues: the project “Scalable, Open, and Comprehensive Recognition of Disinformation Campaigns on the Web,” or SOURCE for short, will build on OWS.EU and develop a Europe‑wide, open research and analysis infrastructure for identifying and investigating disinformation campaigns. Its goal will be, among other things, to continuously collect large volumes of web and social media content using OWI and analyze them with the help of AI. This will create a freely available database that stores disinformation content and can be used to verify online texts and images or to train AI‑based fact‑checking tools. (vs | LRZ)