Safe & Trustworthy AI Agents and Evidence-Based AI Policy

Key Topics

• Exponential growth in LLMs and their capabilities.
• Broad spectrum of risks associated with AI systems.
• Challenges in ensuring trustworthiness, privacy, and alignment of AI.
• Importance of science- and evidence-based AI policy.

Broad Spectrum of AI Risks

• Misuse/Malicious Use: Scams, misinformation, bioweapons, cyber-attacks.
• Malfunction: Bias, harm from system errors, loss of control.
• Systemic Risks: Privacy, labor market impact, environmental concerns.

AI Safety vs. AI Security

• AI Safety: Prevent harm caused by AI systems.
• AI Security: Protect AI systems from external threats.
• Adversarial Settings: Safety mechanisms must withstand attacks.

Trustworthiness Problems in AI

• Robustness: Safe, effective systems, including adversarial and out-of-distribution robustness.
• Fairness: Prevent algorithmic discrimination.
• Data Privacy: Prevent extraction of sensitive data.
• Alignment Goals: Ensure AI systems are helpful, harmless, and honest.

Training Data Privacy Risks

• Memorization: Extracting sensitive data (e.g., social security numbers) from LLMs.
• Attacks: Training data extraction, prompt leakage, and indirect prompt injection.
• Defenses: Differential privacy, deduplication, and robust training techniques.

Adversarial Attacks and Defenses

• Attacks:
  • Prompt injection, data poisoning, jailbreaks.
  • Adversarial examples in both virtual and physical settings.
  • Exploiting vulnerabilities in AI systems.
• Defenses:
  • Prompt-level defenses (e.g., re-design prompts, detect anomalies).
  • System-level defenses (e.g., information flow control).
  • Secure-by-design systems with formal verification.

Safe-by-Design Systems

• Proactive Defense: Architecting provably secure systems.
• Challenges: Difficult to apply to non-symbolic components like neural networks.
• Future Systems: Hybrid symbolic and non-symbolic systems.

AI Policy Recommendations

Key Priorities:

1. Better Understanding of AI Risks:

   • Comprehensive analysis of misuse, malfunction, and systemic risks.
   • Marginal risk framework to evaluate societal impacts of AI.

2. Increase Transparency:

   • Standardized reporting for AI design and development.
   • Examples: Digital Services Act, US Executive Order.

3. Develop Early Detection Mechanisms:

   • In-lab testing for adversarial scenarios.
   • Post-deployment monitoring (e.g., adverse event reporting).

4. Mitigation and Defense:

   • New approaches for safe AI.
   • Strengthen societal resilience against misuse.

5. Build Trust and Reduce Fragmentation:

   • Collaborative research and international cooperation.

Call to Action

• Blueprint for Future AI Policy:
  • Taxonomy of risk vectors and policy interventions.
  • Conditional responses to societal risks.
• Multi-Stakeholder Collaboration:
  • Advance scientific understanding and evidence-based policies.

Resource: Understanding-ai-safety.org