Questions to consider

• Do you believe that advanced AI poses an existential risk?
  • What are the core pieces of evidence/information/intuition that inform your overall opinion?
  • These “core pieces” of information are also known as “cruxes”. A crux is defined as “any fact that if an individual believed differently about it, they would change their conclusion in the overall conclusion.”
• How soon do you think we will achieve transformative AI, if ever? Why – what pieces of evidence/information/intuition are you using?
  • “Transformative AI, which we have roughly and conceptually defined as AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution.” – Holden Karnofsky (Open Philanthropy, 2016)
  • Perhaps consider forecasting work on AI timelines.
    • AI Timelines: Where the Arguments, and the "Experts," Stand (Holden Karnofsky, 2021) (post - 13 mins)
• What actions should humanity be taking (if any) with respect to potential risks from advanced AI, concretely? If you found this problem important, what actions would you take and why?

Resources

If you’re interested in learning more about technical AI safety or AI governance, probably the single most useful resource is the AISF (AI Safety Fundamentals) Program. You can read through the (technical) AI alignment curriculum or the AI governance curriculum on your own, or ideally apply to the program to be in one of the reading group cohorts!

<aside> 👉 Thus, the recommended readings for this week are: read through the AISF alignment curriculum or AISF governance curriculum to see what’s covered each week and pick 3+ readings that interest you.

</aside>

Prioritize interesting readings from the curricula. Here are some example readings (from ~Week 1 in the AISF curriculum) that may or may not be in the above curricula:

• Existential Risk from Power-Seeking AI (Joe Carlsmith, 2022) (talk - watch the first 36 mins; skip Q&A); or read the transcript (post - 35 mins).
• AI could defeat all of us combined (Holden Karnofsky, 2022) (post - 20 mins)
  • Or generally the Most Important Century series. Another good one from the series: Why AI alignment could be hard with modern deep learning (Ajeya Cotra, 2021) (post - 18 mins)
• Current work in AI alignment (Paul Christiano, 2019) (talk - 35 mins / transcript - 26 mins)
• The alignment problem from a deep learning perspective (Richard Ngo, 2022) (paper - 33 mins)
• Intro to AI Safety, Remastered (Robert Miles, 2021) and other Robert Miles videos

If you already know a lot about potential risks from advanced AI, some next steps to get involved are:

• Apply to one of the AISF programs. Besides accountability, this program provides mentors and a peer community.
• Apply for career advising by 80,000 Hours.
• Take a look at additional resources:
  • Preventing an AI-related catastrophe (80,000 Hours / Benjamin Hilton, 2022-2023) (problem profile - 58 mins / recording 2 hours 28 mins).
    • In particular, read the section at the end on What you can do concretely to help
  • Take a look at some of the top-rated EA Forum posts on AI governance
• Advice for people interested in working in AI safety (in either technical alignment or in governance):
  • AGI safety career advice (Richard Ngo, 2023) (post - 15 mins)
  • How to pursue a career in technical AI alignment (Charlie Rogers-Smith, 2022) (post - 47 mins)
  • FAQ: Career Advice for Alignment Researchers (Rohin Shah, 2021) (post - 38 mins)
  • AI governance and coordination (80,000 Hours / Cody Fenwick, 2023) (career review - 42 mins)