DFKI AI Next 2 englisch

About us Human-Centric AI Intelligent solutions for the knowledge society DFKI has more than 35 years of experience in human-centric AI, conducting research in key areas of basic research and future- oriented applications, while always maintaining a focus on social relevance and scientiﬁc excellence in the ﬁeld of artiﬁcial intelligence. The German Research Center for Artiﬁcial Intelligence (DFKI) was founded in 1988 as a non-proﬁt public-private partnership. It operates facilities in Kaiserslautern, Saarbrücken, Bremen, Lower Saxony (Osnabrück and Oldenburg), with laboratories in Berlin, Darmstadt and Lübeck, and a branch ofﬁce in Trier. It supports 27 research areas, ten competence centers, and eight living labs. Based on application-oriented research, it develops product functions, prototypes, and patentable solutions in the informa- tion and communication technology (ICT) sector. Funding is provided by public funding sources and industrial development contracts. Project results and milestones are periodically reviewed institutionally and by a Scientiﬁc Advisory Board, an interna- tional panel of experts. In addition to the German Federal Ministry of Education and Research and the federal states of Rhineland-Palatinate, Saarland, Bremen, and Lower Saxony, several leading German and international high-tech companies from a wide range of industries also have seats on the board. www.dfki.de

Editorial Trustworthy AI for the Economy 1 August 2024, allowing for different transition periods for impacted companies and organizations. The European Union used a risk-based approach in drafting the AI Act: applications be considered trustworthy and meet strict requirements in terms of transparency, correctness, explainability, technical and content robustness, human control, and data protection before their deployment in critical infrastructures, in the Vendors and commercial users can be certain that every high-risk AI application is subject to review before use. The review determines that it is possible to understand how a system works and how it makes decisions (transparent and explainable). In operational use, AI systems must be techni- cally stable, fault-tolerant, and protected against manipulation Prof. Dr. Antonio Krüger (robust and secure). In highly sensitive areas, no fully autono- CEO mous decisions may be made without human control. The current state of technology only partially meets these requirements. Large language models, in particular, often lack takes on an important role here as the driver of trustworthy AI. In the development of algorithms and procedures, current research initiatives focus on providing guarantees that decisions made by the AI application are reliable and fair – even in uncertain or unpredictable environments. Trustworthy AI is the product of research that provides the knowledge and tools to make AI systems robust, fair, and transparent. Trustworthy AI systems can be safely and respon- meeting regulatory, ethical, and societal requirements. As a result, our research creates new business opportunities, reduces risk, and builds trust. 3 3

Trustworthy AI The Potential in Trustworthy Language Technology for the European Economy Implementing AI applications without knowing the exact consequences of the legislative regulation imposed by the EU AI Act is a challenge - worthy AI requires a joint dialog with science, industry, and politics to create security for users and the incentive to innovate for companies. Author: Prof. Dr. Antonio Krüger Innovative AI processes are very important for val- ue creation in various industries and for companies of Requirements for Trustworthy AI Trust is especially important when risk is present: The different sizes. As the use of AI technologies becomes European approach to Trustworthy AI recognizes differ- more widespread in healthcare, energy supply, or trans- ent levels of sensitivity of AI applications. Consequently, portation, there is a growing awareness that these sys- the AI systems developed or used within the European tems have to be judged as trustworthy so that they can Union will have different requirements depending on the be used with conﬁdence in mission-critical decisions. risk classiﬁcation. To improve the trustworthiness of AI However, many of the conclusions such systems reach systems, features such as robustness, algorithmic fair- are sometimes difﬁcult to understand, the database is ness, explainability, and transparency are discussed. In unknown, and the algorithms are not transparent. Mar- addition, trustworthiness must be systematically deter- ketable solutions require reliable criteria and the re- mined and evaluated throughout the entire life cycle of quirements for trustworthy AI, both of which must be an AI system. developed against the backdrop of government regula- tion through the EU AI Act. Companies need practical tools and standards as well as testing and certiﬁcation procedures if they are to meet the technological re- DFKI researches the trustworthiness of large language models The ﬁeld of AI-based language technology serves as a quirements for trustworthy AI applications. good example to discuss the various requirements of 4

Trustworthy AI Trustworthy AI. Large Language Models (LLMs) have dis- the goal of digital language fairness in Europe: Occiglot, ruptive potential, and language-based AI tools such as the academic, non-proﬁt research initiative, is commit- GPT-4 are now ubiquitous in the private and public sec- ted to open, transparent, and fully documented science tor. Unfortunately, the requirements for trustworthiness and the development of open-source LLMs. are only met to a limited extent, which severely restricts the usability of applications in scenarios with high-risk security requirements (such as in medicine or legal ad- Encouraging innovation through regulation The European Union has been working on standards judication). This limitation is the result of not knowing to implement the AI Act as European law and to assist the parameters, training data, training methods, and in- its application in practice. However, European com- ference settings. panies can only fully exploit the potential of innova- tive AI-based language technologies when veriﬁable DFKI is currently researching these and other open standards and robust test procedures are available questions for basic research on the trustworthiness of and based on the technical and ethical requirements large language models: The DFKI Speech and Language for trustworthy systems. Over time, higher security re- Technology Lab in Berlin develops reliable evaluation quirements may remove barriers to innovation and criteria for Trustworthy AI-based language technology. can lead to wider acceptance among users and en- The evaluation includes robustness and reliability, and sure more clarity for companies in making investment addresses how to measure the performance of language decisions. In the best case, smooth synchronization models in speciﬁc domain applications or across lan- will occur between innovative design, legal structure, guage boundaries. More research is needed with regard and technical implementation. This requires a dy- to transparency and explainability, for example, how namic dialog between science, industry, and politics. knowledge can be localized in large language models and how certain model behaviors can be assigned to speciﬁc model components. At the Centre for European Research on Trusted AI Food security (CERTAIN), DFKI experts are currently using mechanis- tic interpretability methods to investigate whether LLMs Climate protection can also learn complex rules or simply retrieve facts. In project OpenGPT-X, DFKI develops various approach- Energy transition es, including open science concepts. The open approach offers a possible way forward to develop safer and more Trustworthy AI and promotes the participation of every- one in shaping the future of this technology. Direct ac- cess to language models and the data used to train them is required for top-quality basic research and commer- cially viable technology applications. In this sense, a re- search initiative launched by DFKI is intended to pursue Resource efficiency 5 Economic impact Demography Knowledge Industry 4.0 Machine Learning Speech & Language Technology Robotics Mobility Digital sovereignty

Interview Trusted AI – Trustworthiness and Large Language Models Interview with Dr. André Meyer-Vitali and Dr. Simon Ostermann ChatGPT, LLaMA, or Mistral Meyer-Vitali: Generative pre- Trusted AI is characterized by a yield astonishing expertise trained transformers (GPT) are high degree of reliability, security, and, quite obviously, wrong large, highly interconnected mod- transparency, robustness, fairness, answers. What is the reason els, which makes them opaque and and veriﬁability, all while improving for this unreliability? difﬁcult to control. The underlying the functionality of existing sys- technology – deep learning – does tems. The performance and reliabil- not provide a true understanding of ity of AI systems can be trusted by the problem but merely maps developers, users, and regulators complex statistical relationships. – even in complex socio-technical environments. Ostermann: Not only is the technol- ogy a data-based black box, many Do we need a technological vendors do not make the source reboot? code of their systems available. As Meyer-Vitali: Not entirely, but the a consequence, this lack of insight new generation of AI systems will into parameters, training data, be based on hybrid systems that training methods, and inference do not rely solely on data-driven settings makes it difﬁcult to under- approaches, but rather utilize the stand how these models arrive at full range of AI techniques, includ- their results. ing search, reasoning, planning, and How can Generative AI be made more trustworthy? What to expect from hybrid symbolic AI methods. Dr. André Meyer-Vitali re- Meyer-Vitali: The term “Trusted AI” AI systems? searches hybrid and distri- is used for a new overall approach Meyer-Vitali: The use of neuro- buted artificial intelligen- to advance the development of symbolic models facilitates valida- ce. He is Principal Investi- reliable systems. The goal is a new tion, creates more transparency, gator at CERTAIN, the Cen- # tre for European Research generation of AI that guarantees and promotes greater account- functionality, especially in high-risk ability, whereas causal models on Trustworthy AI. applications. provide understandable explana- 6

Interview tions and offer insights into the AI investigate whether LLMs are a language model by inserting decision-making processes. These capable of learning complex rules “intermediate prompts.” Studies Hybrid AI systems achieve “Trust or whether they can only reproduce are also looking at the use of con- by design” by combining machine facts. The goal is to better under- crete examples and how carefully learning with symbolic reasoning stand the capabilities of these mod- selected examples reveal the limi- and the explicit representation els through “reverse engineering” tations of the model. of knowledge. Semantic or other and to deepen the human under- explicit methods can represent standing of how they work. knowledge that does not need to be learned by machine. Such an Does the concept of mech- approach improves generalization, anistic interpretability robustness, and interpretability also help in mastering the by steering the learning process bias problem in AI systems? in the right direction. This hybrid Ostermann: Scientiﬁc studies have approach is often referred to as the shown that sexist responses can “Third Wave of AI.” be traced back to speciﬁc model components that, when deactivat- What research questions for ed, can suppress such behavior. We reliable LLMs is language are working to ﬁnd an approach to technology currently inves- localize knowledge in large lan- tigating? guage models and learn how to Ostermann: We focus on the trace speciﬁc responses back to question of how language models individual model components. make deterministic and appropriate decisions, particularly in high-risk Can this prevent language safety and security areas such as models from giving manipu- medicine or legal adjudication. lated answers? At the Centre for European Re- Ostermann: A particular focus of search on Trusted AI (CERTAIN), our work is on protecting against we are currently using mechanistic manipulation through prompt interpretability, a novel approach to injections, i.e., how to prevent third understanding neural networks, to parties from falsifying the output of 7 Dr. Simon Ostermann is a compu- tational linguist and researches transparent and robust language models at DFKI. He is a senior re- searcher in the Language Techno- logy and Multilingualism research department and co-head of the Generative AI Competence Center.

Mission KI Quality and Testing Methods for Greater Trust- worthiness in Medical AI Systems Authors: Ludger van Elst and Adriano Lucieri « The establishment of the Innovation and Quality Center at DFKI allows us to use improve this situation: 1) How can vendors ensure high quality and trustworthiness in their AI-based our existing research departments and the systems? and 2) How can users broad expertise in the field of trustworthy easily recognize which systems in AI available in our extensive networks to the marketplace can actually be create valuable synergies. trusted? As a research partner to companies from The EU AI Act provides important a wide range of industries, we are aware guidelines for assessing the trust- of the particular challenges of developing worthiness of AI components and marketable AI applications in terms of se- systems. The aim is to develop vol- curity, reliability, and ethical acceptability. untary, transparent, and uniform The Innovation and Quality Center repre- quality and testing standards in sents a one-stop shop for all companies to access cutting-edge AI research. » accordance with European values. The TrustifAI project, currently ongoing at DFKI-Kaiserslautern, develops speciﬁc quality criteria as well as test procedures and methods to meet a deﬁned mini- Lack of trust in AI systems is a po- Users are generally unable to ver- mum standard. The standard cov- tential barrier to their successful ify the quality of the results or to ers areas such as transparency, adoption and use. Companies as- assess the safety and security fairness, privacy data, robustness, sociate an investment in AI-based risks, which makes them reluctant and cyber security while following applications with high economic to accept the new technology. the “trust by design” principle: risk, with the major risk factors Quality assurance is integrated at being technical complexity and Finding the answers to two fun- an early stage of the development uncertain market opportunities. damental questions will help to process for AI software. Methods 8

Mission KI worthy AI at the interface between cutting-edge research and practical applications. The Kaiserslautern center serves as a central point of contact for all questions relating to trustworthy AI and responds main- ly to the needs of SMEs and start- ups. The IQZ offers a wide range of programs and consulting services designed to promote knowledge transfer in the research areas of AI testing and trustworthy AI. A regu- lar lecture program, practical work- shops, and cross-sector networking events are designed to encourage the development of a strong, re- gional AI community and enhance the location as a long-term innova- tion hub. The facility is not limited to the medical sector but is open to innovators from all industries. The MISSION KI initiative plans to es- tablish a second center in Berlin by the end of the year. « If we are to build trust in new applications, developers must consider the quality of AI from the very beginning. We look forward to working on methods to improve the qual- ity of AI applications and to es- tablishing testing criteria that will benefit other sectors be- yond healthcare in the future. » 9 to measure the trustworthiness by the German Federal Ministry for of AI systems are used to ensure Digital and Transport with a total the development of useful appli- budget of 32 million euros and co- cations. Validation is carried out in ordinated by acatech – the German speciﬁcally designed test environ- Academy of Science and Engineer- ments based on ﬁve selected use ing. The initiative creates a plat- cases from the ﬁelds of medicine form where leading organizations and healthcare. from the ﬁelds of testing, standard- ization, and technology can work DFKI‘s work initially focuses on together. AI in the healthcare sector, mainly because of the preponderance of The Kaiserslautern Innovation and companies and research institutes Quality Center (IQZ) is the ﬁrst of in Rheinland-Pfalz from the medi- two MISSION KI centers planned cal, pharmaceutical, and biotech- by the German government specif- nology sectors and the availability ically to promote the use of trust- of a wide range of use cases. The use cases studied in TrustifAI in- clude the detection of skin cancer using dermatoscopic images, deci- sion support for tumor treatment, support for demand forecasting and planning for emergency servic- es, the improvement of treatment plans for psychiatric patients, and the detection of potential problems using medical imaging data during preoperative intubation. The TrustifAI project falls under MISSION KI, an initiative to pro- mote artiﬁcial intelligence research in Germany. The initiative is funded

More on the topic Occiglot: Open-Source Language Models from Europe for Europe Authors: Prof. Dr. Georg Rehm, Dr. Patrick Schramowski Recent advances in large language models (LLMs) high-performance computing infrastructure is being demonstrate their potential as a disruptive innovation. created by the EuroHPC Joint Undertaking, access and However, because of the high development and train- use are often associated with high costs and techni- ing costs as well as the required expertise, the ﬁeld is cal hurdles. To avoid these issues, Occiglot uses pre- dominated by a few large tech companies and deep tech trained, openly available language models such as META start-ups. Their decisions often overlook central Euro- (LLaMA) or MistralAI. However, the data used is not pean values such as linguistic diversity and cultural rich- documented, which injects a lack of transparency that ness in favor of economic considerations. makes understanding difﬁcult and hinders the ability to To ensure Europe‘s digital sovereignty and competitiveness in the ﬁeld of AI, DFKI was in- strumental in creating the Occiglot research initiative. The aim is to emphasize the impor- reproduce work or make further advances. Occiglot closes this gap by curating robust, mul- tilingual data sets and giving them transpar- ency. The data collec- tion already contains tance of dedicated language modeling solutions that around 230 million cleaned documents based on ex- are critical for digital language equality in Europe. The isting work in 10 languages. All documents have been European Parliament and projects such as European processed using special ﬁltering methods, resulting in Language Equality (ELE) are also working toward this a German-language data set of over 60 billion tokens. goal. Open access to language models and the under- This data is publicly available, as are the resulting lan- lying training data is essential for high-quality research guage models, which are pre-trained by partners such and the development of commercially relevant tech- as HessianAI. nologies. As a non-proﬁt research collective, Occiglot is committed to open science and the development of open-source LLMs. Current research Several research areas at DFKI are currently working on large language models, including the BMWK-fund- Open Data – Occiglot Fineweb Two major challenges in the development of powerful ed OpenGPT-X project. The Occiglot research collective now includes other organizations such as HessianAI, TU LLMs must be resolved: the need for high-performance Darmstadt, Common Crawl, Ontocord.AI, the Barcelona computing centers and the need for large amounts Supercomputing Center, and the European Language of high-quality language data. Although a European Grid and European Language Equality networks. 10

More on the topic Trustworthy Research Environment for Sensitive Data at DFKI The term “Trustworthy AI” applies nical and organizational measures In contrast to cloud solutions, SEM- not only to the functionality and al- (TOMs) required for data protection LA stores and processes the data gorithms of large language models and data security in data-sensitive exclusively “on-premises,” i.e., but to the entire development pro- research projects. cess of AI systems, which includes the machine learning processes for training the neural networks. In the medical ﬁeld, the training data we use generally involves high- ly protected patient data such as ul- trasound, X-ray, MRI, and CT imag- es. The data from social interactions with medical staff, for example, the photos and audio/video recordings of patients, are also useful for train- ing the AI systems that support di- agnosis and treatment of psychiat- ric or psychosomatic illnesses. All this sensitive data is subject to the provisions of the GDPR and other data protection regulations. Accord- ing to the principles of good scien- Access to the SEMLab is biometrically secured with a palm vein scanner. within the DFKI facilities. In addi- tion to the biometrically secured an- notation and experimentation lab- oratory (SEMLab) in Saarbrücken, the research environment consists of a computing infrastructure (CPU, GPU, memory) that is operated and safeguarded in Kaiserslautern. The infrastructure is designed to sup- port research projects classiﬁed at Tier 4 and higher, according to the sensitivity classiﬁcation scheme of the Alan Turing Institute. Future plans call for the data sets hosted by SEMLA to be available for use by third parties for model computations on the Internet. The applications are being submitted tiﬁc practice, trustworthy research “SEMLA gives us the ﬁrst opportuni- for certiﬁcation according to ISO environments are needed before ty to work with highly sensitive data. 2700X and TISAX as well as for the this data can be included in AI mod- First, we evaluate medical data from European Data Protection Seal of els. Such a research infrastructure projects in which DFKI is involved, Approval (EuroPriSe, 2022). – the Secure Machine Learning Ar- and then we use this data to train chitecture (SEMLA) – exists at the neural networks. In the future, SEM- DFKI facilities in Saarbrücken and LA will be made available as open Kaiserslautern. The main purpose source,” says SEMLA project man- of SEMLA is to implement the tech- ager Dr. Jan Alexandersson. 11

Lübeck Oldenburg Bremen Berlin Osnabrück DFKI Site DFKI Laboratory Trier Darmstadt Kaiserslautern Saarbrücken Imprint dfki ai next; Issue September | 2024; Published by: German Research Center for Artiﬁcial Intelligence GmbH (DFKI); Trippstadter Str. 122, D-67663 Kaiserslautern; Phone: +49 631 20575 0; E-Mail: news@dfki.de; Responsible: Heike Leonhard; Editing: Sandra Antakli, Armindo Ribeiro; Layout: Lando Lehmann; Production: One Vision Design Credits Cover: AI-generated image, Stable Diffusion, Adobe Firefly; Page 3: DFKI, Jürgen Mai; Page 4: metamorworks - stock.adobe.com; Page 5: DFKI; Page 6: DFKI, Armindo Ribeiro; Page 7 top: AI-generated image, Adobe Stock, bottom: private; Page 8: DFKI; Page 9, top: acatech, bottom: DFKI, Jürgen Mai; Page 10: https://occiglot.eu; Page 11: DFKI, Jaron Hollax; Page12: DFKI.