This Keynote explores the unique challenges and risks associated with AI-based functionality in security-critical applications. As AI systems become increasingly vital in various domains such as automotive, cybersecurity, and healthcare, it is crucial to incorporate new, complex vulnerabilities into risk management. Thora and Vasilios discuss best practices for risk assessment, highlighting strategies to identify AI-specific threats and vulnerabilities. Additionally, the session examines the impact of upcoming AI regulations on the development and deployment processes across the entire life cycle of an AI system.