Concise Argument for ASI Risk

I listened to the debate between Stephen Wolfram and Eliezer Yudkowsky on Machine Learning Street Talk (MLST).

I found the discussion frustrating, since it felt like they were trying to have two very different conversations: Wolfram questioning basic principles and trying to build the argument from the foundations, Yudkowsky taking AI risk as being mostly self-evident and defending particular aspects of his thesis.

Yudkowsky seems reluctant to provide a concise point-wise argument for AI risk, which leads to these kinds of strange debates where he defends a sequence of narrow points that feel mostly disconnected. From his body of work, I infer two general reasons why he does this:

  1. He has learned that different people find different parts of the argument obvious vs. confusing, true vs. false. So rather than reiterate the whole argument, he tries to identify the parts they take issue with, and deal with those. This might work for one-on-one discussions, but for public debates (where the actual audience is the broader set of listeners), this makes it feel like Yudkowsky doesn’t have a coherent end-to-end argument (though he definitely does).
  2. Yudkowsky’s style, in general, is not to just “give the answer,” but rather to lead the reader through a sequence of thoughts by which they should come to the right conclusion. In motivated pedagogy (where the reader is trying to learn), this is often the right way. “Giving the answer” won’t cause the person to learn the underlying pattern; the answer might feel too obvious and be quickly forgotten. Thus one instead tries to guide the person through the right thoughts. But to a resistant listener, this leaves the (incorrect) impression that the person’s arguments are vague.

Let me try to put together a step-wise argument for ASI risk. I think it goes something like:

  1. Humans are actively trying to make AIs smarter, more capable, and more agentic (including giving access/control to real-world systems like computers and robots and factories).
  2. There is no particular ceiling at human intelligence. It is possible in principle for an AI to be much smarter than a human, and indeed there are lots of easy-to-imagine ways that they would outstrip human abilities to predict/plan/make-decisions.
  3. AIs will, generically, “go hard”; meaning they will put maximal effort into achieving their goals.
  4. The effective goals of a powerful optimizer will tend to deviate strongly from the design goals. There are many reasons for this:
    • It is hard to reliably engineer something as fuzzy (and, ultimately, inconsistent) as human values.
    • Optimizers often have a mis-alignment between the intended goal and the realized inner optimization (inner/outer alignment problem, mesa-optimizers, etc.).
      • The analogy to evolution is often offered: evolution is optimizing for replication of genes, yet enacted human values have only a little to do with that (wanting to have children, etc.); humans mostly care about non-genetic things (comfort, happiness, truth), and are often misaligned to genes (using contraception).
    • Even goals perfectly-specified for a modest context (e.g. human-scale values) will generalize to a broader context (e.g. control the light-cone) in an ill-defined way. There is a one-to-many mapping from the small to the large context, and so there is no way to establish the dynamics to pick which exact goals are enacted in the extrapolated context.
  5. In the space of “all possible goals”, the vast majority are nonsense/meaningless. A small subspace of this total space is being selected by human design (making AIs that understand human data, and do human things like solve problems, design technology, make money, etc.). Even within this subspace, however, there is enormous heterogeneity to what the “effective goals” look like; and only a tiny fraction of those possible AI goals involve having flourishing humans (or other sentient minds).
    • To be clear, humans will design AIs with the intention that their effective goals preserve human flourishing, but (c.f. #4) this is a difficult, ill-posed problem. The default outcome is an AI optimizing for something other than human flourishing.
  6. A powerful system pursuing goals that don’t explicitly require humans will, generally speaking, not be good for humans. For instance, a system trying to harness as much energy as possible for its computational goals will not worry about the fact that humans die as it converts all the matter in the solar system into solar cells and computer clusters.
  7. A superhuman (#2) system with real-world control (#1) pursuing (with maximum effort, #3) goals misaligned to human values (#4) will try to enact a future that does not include humans (#5). It will, generically, succeed in this effort, which will incidentally exterminate humans (#6).
    • Moreover, this isn’t a case where one can just keep trying until one gets it right. The very first ASI could spell ruin, after which one does not get another change. It’s like trying to send a rocket to the moon, without being able to do test flights! (And where failure means extinction.)

This argument has many things left unspecified and undefended. The purpose is not to provide an airtight argument for ASI risk; but rather to enumerate the conceptual steps, so that one can focus a discussion down to the actual crux of disagreement.

Posted in AI, Philosophy | Tagged , , | 1 Comment

AI News 2024-11-07

General

Research Insights

  • Agent S: An Open Agentic Framework that Uses Computers Like a Human (code).
  • TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters. Treats model parameters as tokens, so that input queries become attentional lookups to retrieve model parameters. This leads to an efficiency improvement when scaling.
  • How Far is Video Generation from World Model: A Physical Law Perspective (preprint, code, video abstract). They train a video model on simple physics interactions. The model generalizes perfectly within-distribution, but fails in general when extrapolating out-of-distribution. This implies the model is not learning the underlying physics.
    • A valid question is whether they provided enough coverage in training, and enough scale (data, parameters, training compute) to actually infer generalized physics. It’s possible that at a sufficient scale, robust physics modeling appears as an emergent capability.
    • Conversely, the implication might be that generalization tends to be interpolative, and the only reason LLMs (and humans?) appear generalized is that they have enough training data that they only ever need to generalize in-distribution.
  • Mixtures of In-Context Learners. Allows one to extract more value from existing LLMs, including those being accessed via cloud (weights not available). The method creates a set of different “experts” by calling an LLM repeatedly with different in-context examples. Instead of just merging or voting on their final responses, one can try to consolidate their responses at the token level by looking at the distribution of predictions for next token. This allows one, for instance, to provide more examples than the context window allows.
    • It would be interesting to combine this approach with entropy sampling methods (e.g. entropix) to further refine performance.
  • AI swarms require communication between agents, but right now there are many competing methods for multi-agent coordination (Camel, Swarm, LangChain, AutoGen, MetaGPT). Researchers at Oxford have proposed a scheme (Agora) for AI agents can auto-negotiate a structured protocol: A Scalable Communication Protocol for Networks of Large Language Models (preprint).

LLM

  • Anthropic added visual PDF support to Claude. Now, when Claude ingests a PDF, it does not only consider a textual conversion of the document, but can also see the visual content of the PDF, allowing it to look at figures, layout, diagrams, etc.
  • Anthropic releases Claude 3.5 Haiku, a small/efficient model that actually surpasses their older large model (Claude 3 Opus) on many benchmarks.

Tools

  • Google is now making available Learn About, a sort of AI tutor that can help you learn about a topic. (Seems great for education.)

Image Synthesis

Audio

Video

World Synthesis

  • Neural reproductions of video games are impressive. We’ve seen Doom, Super Mario Bros., and Counter-Strike.
    • Now, Decart AI (working with Etched) are showing a playable neural-rendered video game (basically Minecraft). Playable here (500M parameters, code). Right now, this is just a proof-of-principle. There is no way for the game designer to design an experience, and the playing itself is not ideal (e.g. it lacks persistence for changes made to terrain). It feels more like a dream than a video game. But the direction this is evolving is clear: we could have a future class of video games (or, more broadly, simulation environments) that are designed using AI methods (prompting, iterating, etc.), and neural-rendered in real-time. This would completely bypass the traditional pipelines.
      • To underscore why you should be thinking about this result in a “rate of progress” context (rather than what it currently is), compare: AI video 2022 to AI video today. So, think about where neural-world-rendering will be in ~2 years.
    • And we now also have GameGen-X: a diffusion transformer for generating and controlling video game assets and environments.

Science

  • Anthropic’s “Golden Gate Claude” interpretability/control method consists of identifying legible features in activation space. Researchers have applied this mechanistic interpretability to understanding protein language models. They find expected features, such as one associated with the repeating sequence of an alpha helix or beta hairpin (visualizer, code, SAE). More fully understanding the learned representation may well give new insights into proteins.
    • More generally, it is likely a very fruitful endeavor to train large models on science data, and search in a feature space for expected features (confirm it learned known physics), and thereafter search for novel physics in the space.

Robots

Posted in AI, News | Tagged , , , , , , , | Leave a comment

What is an AI Agent?

The importance of AI agents continues to grow, which makes it mildly concerning that there is no agreed-upon definition of “AI Agent.” Some people use it to refer to any LLM activation (where “multi-agent” might then just mean chaining multiple LLM calls) whereas others reserve it for only for generally intelligent AI taking independent actions in the real-world (fully agentic). The situation is further confused by the fact that the term “agent” has been used for decades to just refer to a generic software process.

This thread tried to crowd-source a definition. The ones that resonate with me are those that emphasize memory and tool-use, reasoning, and long-running operation on general tasks. So, I offered:

AI Agent: A persistent AI system that autonomously and adaptively completes open-ended tasks through iterative planning, tool-use, and reasoning.

To further refine definitions:

Raw data is used to train a base model, which can be fine-tuned (e.g. into a chatbot). If we scaffold the LLM with tools (document retrieval, software APIs, etc.), we call it an AI Assistant (or a Co-pilot, if we embed it in an existing application or workflow).

We can also exploit iterative deliberation cycles (of many possible sorts) to give the LLM a primitive sort of “system 2” reasoning capability. We can call this a Reasoning AI (such systems are rare and currently primitive, but OpenAI o1 points in this direction). A Reasoning Assistant thus combines iteration with scaffolding.

An AI Agent, then, is a reasoning AI with tool-use, that runs for a long-horizon so that it can iteratively work on complex problems.

Beyond that, we can also imagine multi-agent ecosystems, which work on even more complex tasks by collaborating, breaking complex problems into parts (for specialized agents to work on), and combining results. Finally (and most ambitiously), we can imagine that this “swarm” of AI agents is deeply integrated into human work, such that it feels more like an exocortex.

Posted in AI | Tagged , , , , | Leave a comment

AI News 2024-10-31

General

Research Insights

  • adi has proposed a new benchmark for evaluating agentic AI: MC bench (code). It consists of having the agent build an elaborate structure in Minecraft. By using humans to A/B rank the visual output, the capability of agents can be ranked.
  • Anthropic have provided an update to their interpretability work, where the activation space is projected concisely into a higher-dimensional space using sparse auto-encoders (SAE). Now, they posted: Evaluating feature steering: A case study in mitigating social biases. Earlier work showed that they can enforce certain kinds of model behaviors or personalities by exploiting a discovered interpretable feature. Now, they further investigate; focusing on features related to social bias. They find that they can, indeed steer the model (e.g. elicit more neutral and unbiased responses). They also find that pushing too far away from a central “sweet spot” leads to reduced capabilities.
  • RL, but don’t do anything I wouldn’t do. In traditional training, parts of the semantic space without data are simply interpolated. This can lead to unintended AI behaviors in those areas. In particular, this means when an AI isn’t sure what to do, they do exactly that undefined thing. This new approach tries to consider uncertainty. So when an AI isn’t sure about an action, it is biased towards not taking that action. This captures a sort of “don’t do anything I might not do” signals.
  • Mixture of Parrots: Experts improve memorization more than reasoning. The “mixture-of-experts” method (of having different weights that get triggered depending on context) seems to improve memorization (more knowledge for a given inference-time parameter budget) but not reasoning. This makes sense; reasoning is more of an “iterative deliberation” process that benefits from single-pass parameters and multi-pass refinement.
  • The Geometry of Concepts: Sparse Autoencoder Feature Structure. Tegmark et al. report on finding the feature space of LLMs spontaneously organizes in a hierarchical manner: “atomic” structures at small-scale, “brain” organization at intermediate-scale, and “galaxy” distribution at large-scale.

LLM

Audio

Image Synthesis

Video

World Synthesis

Robots

Posted in AI, News | Tagged , , , , , , | Leave a comment

AI News 2024-10-24

General

Research Insights

Safety/Policy

LLM

  • The OpenAI Chat Completion API now supports audio input (allowing one to skip a separate transcription step).
  • Google’s Notebook LM has capture much attention, in part due to the useful “chat with my PDFs” feature, but mostly the cool “generate podcast” trick. You can now customize the podcast generation.
  • MotherDuck have added a “prompt()” function to their SQL database, such that you can weave LLM calls into your SQL lookups.
    • BlendSQL appears to be an open-source attempt to do something similar: combine LLM calls with SQL.
  • Meta released Meta Spirit LM an open source multimodal language model that freely mixes text and speech.
  • Anthropic announces a new Claude 3.5 Haiku model, as well as a new version of their excellent Claude 3.5 Sonnet model. This new model can “use a computer” (still experimental), available via API.
    • Ethan Mollick posts about his experience using this experimental mode.
    • An open-source version (using regular Claude 3.5 Sonnet via API) has appeared: agent.exe.
  • Perplexity plans to release a reasoning mode, where it can agentically search and collate information.

Tools

Audio

  • Elevenlabs adds Voice Design, allowing you to generate a new voice by text-prompting what it should sound like.

Image Synthesis

Video

Science

Hardware

Robots

  • A video of Fourier’s GR-2 robot standing up.
  • Video of Engine AI robot walking. As noted, the more upright (locked knees) gait is more energy-efficient, compared to the squatted (bended knee) walking of many other designs.
  • Clone Robotics continue to pursue their micro-hydraulic bio-mechanical approach to robotics; they now have a torso.
Posted in AI, News | Tagged , , , , , , , , , , | Leave a comment

AI News 2024-10-17

General

  • Anthropic CEO Dario Amodei has published an opinion piece about the future of AI: Machines of Loving Grace: How AI Could Transform the World for the Better. While acknowledging the real risks, the piece focuses on how AI could bring about significant benefits for humankind.
    • Max Tegmark uses this as an opportunity to offer a rebuttal to the underlying thesis of “rapidly developing strong AI is a net good”: The AGI Entente Delusion. He views a competitive race to AGI as a suicide race, since efforts to align AI are lagging our ability to improve capabilities. He proposes a focus on Tool AI (instead of generalized AI), so that we can reap some of the benefits of advanced AI, with fewer of the alignment/control problems. This view focuses on government regulation proportionate to capability/risk. So, in principle, if companies could demonstrate sufficiently controllable AGI, then it could meet safety standards and deployed/sold.
  • (Nuclear) Energy for AI:
    • The US Department of Energy is committing $900M to build and deploy next-generation nuclear technology (including small reactors).
    • Google announced it will work with Kairos Power to use small nuclear reactors to power future data centers.
    • Amazon is investing $500M in small modular reactors, to expand genAI.
    • A group (Crusoe, Blue Owl Capital, and Primary Digital Infrastructure) announced $3.4B joint venture to build a 200 MW datacenter (~100k B200 GPUs) in Texas. Initial customers will be Oracle and OpenAI.
    • The growing commitments to build-out power for datacenters makes it increasingly plausible that AI training will reach 1029 FLOPS by 2030 (10,000× today’s training runs).
  • Here is an interesting comment by gwern on Lesswrong (via this), that explains why it is so hard to find applications for AI, and why the gains have been so small (relative to the potential):

If you’re struggling to find tasks for “artificial intelligence too cheap to meter,” perhaps the real issue is identifying tasks for intelligence in general. …significant reorganization of your life and workflows may be necessary before any form of intelligence becomes beneficial.

…organizations are often structured to resist improvements. …

… We have few “AI-shaped holes” of significant value because we’ve designed systems to mitigate the absence of AI. If there were organizations with natural LLM-shaped gaps that AI could fill to massively boost output, they would have been replaced long ago by ones adapted to human capabilities, since humans were the only option available.

If this concept is still unclear, try an experiment: act as your own remote worker. Send yourself emails with tasks, and respond as if you have amnesia, avoiding actions a remote worker couldn’t perform, like directly editing files on your computer. … If you discover that you can’t effectively utilize a hired human intelligence, this sheds light on your difficulties with AI. Conversely, if you do find valuable tasks, you now have a clear set of projects to explore with AI services.

Research Insights

Safety

LLM

AI Agents

Audio

Image Synthesis

  • Abode presented Project Perfect Blend, which adds tools to Photoshop for “harmonizing” assets into a single composite. E.g. it can relight subjects and environments to match.

Vision

Video

World Synthesis

Science

Cars

  • At Tesla’s “We, Robot” event, they showed the design for their future autonomous vehicles: Cybercab and Robovan. The designs are futuristic.

Robots

Posted in AI, News | Tagged , , , , , , , , , , | Leave a comment

AI News 2024-10-10

General

  • Ethan Mollick writes about “AI in organizations: Some tactics”, talking about how individuals are seeing large gains from use of AI, but organizations (so far) are not.
    • Many staff are hiding their use of AI, with legitimate cause for doing so: orgs often signal risk-averse and punitive bureaucracy related to AI, staff worry that productivity gains won’t be rewarded (or indeed punished, as expectations rise), staff worry contributions won’t be regarded, etc.
    • Mollick offers concrete things that orgs can do to increase use of AI:
      • Reduce fear. Do not have punitive rules. Publicly encourage the use of AI.
      • Provide concrete, meaningful incentives to those who use AI to increase efficiency.
      • Build a sort of “AI Lab” where domain experts test all the tools and see whether they can help with business processes.
  • The 2024 Nobel Prize in Physics has been awarded to John J. Hopfield and Geoffrey E. Hinton, for developing artificial neural networks.
  • The 2024 Nobel Prize in Chemistry has been awarded to to David Baker for computational protein design, and to Demis Hassabis and John Jumper for AI protein prediction (AlphaFold).
  • Lex Friedman interviews the team that builds Cursor. Beyond just Cursor/IDEs, the discussion includes many insights about the future of LLMs.

Research Insights

LLM

AI Agents

  • Altera is using GPT-4o to build agents. As an initial proof-of-concept, they have AI agents that can play Minecraft.
  • CORE-Bench is a new benchmark (leaderboard) for assessing agentic abilities. The task consists of reproducing published computational results, using provided code and data. This task is non-trivial (top score right now is only 21%) but measurable.
  • OpenAI released a new benchmark: MLE-bench (paper) which evaluates agents using machine-learning engineering tasks.
  • AI Agents are becoming more prominent; but there is a wide range of definitions being used implicitly, all the way from “any software process” (“agent” is already in use for any software program that tries to accomplish something has been called) all the way to “AGI” (needs to be completely independent and intelligent). This thread is trying to crowd-source a good definition.
    • Some that resonate with me:
      • (1): agent = llm + memory + planning + tools + while loop
      • (2): An AI system that’s capable of carrying out and completing long running, open ended tasks in the real world.
      • (3): An AI agent is an autonomous system (powered by a Large Language Model) that goes beyond text generation to plan, reason, use tools, and execute complex, multi-step tasks. It adapts to changes to achieve goals without predefined instructions or significant human intervention.
    • To me, a differentiating aspect of an agent (compared to a base LLM) is the ability to operate semi-autonomously (without oversight) for some amount of time, and make productive progress on a task. A module that simply returns an immediate answer to a query is not an agent. So, there must be some kind of iteration (multiple calls to LLM) for it to count. So I might offer something like:
      • AI Agent: A persistent AI system that autonomously and adaptively completes open-ended tasks through iterative planning, tool-use, and reasoning.

Image Synthesis

  • FacePoke is a real-time image editor that allows one to change a face’s pose (code, demo), based on LivePortrait.
  • A few months ago, Paints-UNDO (code) unveiled an AI method for not just generating an image, but approximating the stepwise sketching/drawing process that leads up to that image This is fun, maybe useful as a sort of drawing tutorial; but also undermines one of the few ways that digital artists can “prove” that their art is not AI generated (by screen-capturing the creation process).

Video

World Synthesis

Science

Posted in AI, News | Tagged , , , , , | Leave a comment

How Smart will ASI be?

The development of AI is pushing towards AGI. To many, once you have AGI, you quickly and inevitably achieve ASI (superintelligence), since AGI can do AI research and thus AI iteratively self-improves (at an exponential rate). Others sometimes doubt that ASI can exist; they wonder how AI could ever be smarter than humans.

Here, let us try to enumerate how AI might be smarter than a human.

0. Human-like

A basic assumption herein is that human-level general intelligence can be reproduced. So a sufficiently advanced AI would be able to do anything a human can do. This captures more than just book smarts and mathematical or visual reasoning; our ability to socialize is also an aspect of intelligence (theory of mind, etc.).

1. Focused

A simple way in which a human-level AI would immediately be superhuman is due to focus. The AI can be more motivated, focused, attentive, single-minded, etc. Removing myriad foibles/weaknesses from a human would make them superhuman in output.

2. Fast

Once we can simulate human-level thinking in silico, there’s no reason it can’t be accelerated (through better hardware, algorithmic optimizations, etc.). A single human, if thinking sufficiently quickly, is already quite superhuman. Imagine if, for every reply you need to give, you are allowed to spend endless time researching the best answer (including researching the background of the person asking the question, tailoring it to their knowledge and desires). You can scour the literature for the right information. You can take your time to make sure your math is right. In fact, you can write entire new software stacks, invent new data analysis procedures, tabulate endless statistics. Whatever you need to make your answer just a little bit better.

3. Replicated

The computational form of AI makes it easy to replicate. So, in addition to “spending time thinking,” one can also launch numerous parallel copies to work on a problem. The diverse copies can test out different approaches (using different assumptions, different subsets of the data). The copies can double-check each other’s work. In principle, for any question asked to the AI, a vast hierarchy of agents can be launched; some reading sources, others analyzing data, others collecting results. Imagine, that for every decision, you could leverage a planetary-scale of intellectual effort, all focused precisely on solving your task.

There is little doubt that human organizations exhibit some form of superhuman capability in terms of the complexity of projects they execute. The AI equivalent is similar, but far more efficient since the usual organization frictions (lack of motivation in individuals, misaligned desires among participants, mistrust, infighting, etc.) are gone.

The sheer potential scale of AI parallel thinking is a staggering form of superhuman capability.

4. Better

In principle, an AI brain could be better than a human one in a variety of ways. Our cognition is limited by the size of working memory, by how convoluted a chain we can hold in our heads, by the data-rates of our sensory inputs, by fragility to distractions, etc. All of these are, in principle, improvable.

5. Tunable

Human brains are subject to various modifications, including some that can improve task performance. Certain drugs can induce relaxed or heightened states that might maximize focus, or creativity, or emotions, etc. Certain drugs (e.g. psychedelics) seem even able to “anneal” a mind and help one escape a local minimum in thought-space (for better or worse). In humans, these interventions are brute and poorly-understood; yet even here they have predicable value.

In AI, interventions can be much more precise, reproducible, controllable, and powerful. (And need not have side-effects!) One can, in principle, induce target mental states to maximize particular behaviors or capabilities. In this sense, AI could always have the “ideal” mental state for any particular task.

6. Internalized Scaffolding

It is worth noting that a large fraction of human intelligence comes not from our “raw brainpower” but from the scaffolding we have put on top, which includes language, concepts, math, culture, etc. For instance, our brains are roughly equivalent to the brains of ancient humans. We are much smarter, in large part, because we have a set of heuristics (passed down through culture, books, etc.) that allow us to unlock more “effective intelligence.” Some of our most powerful heuristics (math, logic, science, etc.) do not come so easily to us.

For AI, there is no need for this scaffolding to be external and learned. Instead, it could be more deeply embedded and thus reflexive. Arguably, modern LLMs are some version of this: the complexity of modern concepts (encoded in language) become built-in to the LLM. More generally, an AI could have more and more of this complex scaffolding internalized (including reflexive access to myriad source documents, software, solvers, etc.).

7. Native Data Speaker

Humans can speak (and think) “natively” using language, and learn to understand certain concepts intuitively (everyday physics, “common sense,” etc.). We then attempt to understand other concepts in analogy to those that are intuitive (e.g. visual thinking for math). An AI, in principle, can be designed to “think natively” in other kinds of data spaces, including the complex data/statistics of scientific data sets (DNA sequences, CAD designs, computer binary code, etc.). And these diverse “ways of thinking” need not be separated; they could all be different modalities of a single AI (much as humans can think both linguistically and visually).

By being able to “natively” think in source code, or symbolic math, or complex-high-dimensional-data-structures, the AI could exhibit vastly improved reasoning and intuition in these subject areas.

8. Goes Hard on Important Tasks

Humans are, mostly, unaccustomed to what can be accomplished by truly focusing on a task. The counter-examples are noteworthy as they are so rare: humans were able to design an atomic weapon, and send a person to the moon, in a relatively short time owing to the focus. The organizations in question “went really hard” on the task they were assigned. Most organizations we see are enormously inefficient in the sense that they are not, really, single-mindedly focused on their nominal task. (Even corporate entities trying to maximize profit at all cost are, in the details, highly fractured and inefficient since the incentives of the individual players are not so aligned with the organizational goal. Many organizations are not, in actual fact, pursuing their nominal/stated goal.) The jump in effective capability one sees when an organization (or entity) “really goes hard” (pursues their goal with unrestrained focus) can be hard to predict, as they will exploit any and all opportunities to advance their objective.

9. Goes Hard on Small Tasks

It is also worth considering that an AI can (due to its focus) “go hard” on even the small things that we normally consider trivial. Humans routinely ignore myriad of confounders, give up on tasks, or otherwise “don’t sweat the small stuff.” This is adaptive for ancestral humans (avoid wasting effort on irrelevant things) and modern humans (don’t stress out about minutia!). But an AI could put inordinate effort into each and every task, and sub-task, and sub-sub-task. The accumulation of expert execution of every possible task leads to enormous gains at the larger scales. The flip side to the curse of small errors compounding into enormous uncertainty, is that flawless execution of subtasks allows one to undertake much more complex overall tasks.

A simple example is prediction. As a human predicts what someone else will do, their thinking quickly dissolves into fuzzy guesses; and they give up predicting many moves ahead. The space of options is too large, and the error on each guess in the chain too large. But, with sufficient effort in each and every analysis, one could push much, much harder on a long predictive chain.

10. Unwaveringly Rational

Humans know that rational thinking is, ultimately, more “correct” (more likely to lead to the correct answer, achieve one’s aims, etc.). Yet even those humans most trained in rational thinking will (very often!) rely on irrational aspects of their mind (biases, reflexive considerations, intuitions, inspiration, etc.) when making decisions. Simply sticking to “known best practices” would already improve effectively intelligence. An AI could go beyond this even, by exploiting rigorous frameworks (Bayesian methods, etc.) to be as rationale as possible.

(This need not compromise creativity, since this is also subject to rigorous analysis: optimal amount of creativity, efficient randomization schedules, maximize human enjoyment, etc.)

11. Simulation

With sufficient computer power, a broad range of things can be simulated as part of a thinking process. Complex physical setups, social dynamics, and even the behavior of a single person could be simulated as part of the solution to a problem. Uncertainties and unknowns can be handled by running ensembles of simulations covering different cases. Humanity has reached superhuman weather forecasting by exploiting dense data and complex simulations. An ASI could, in principle, leverage simulations tailored to every subject-area to similarly leverage superhuman predictions to inform all decisions.

12. Super-Forecasting

A combination of the features described above (rational, going hard, simulation) should enable ASI to be incredible forecasters. By carefully taking account of every possible factor (large and small), and predicting the possible outcomes (using logic, detailed simulation, etc.) one can generate chains of forward-predictions that are incredibly rich. Uncertaintites can be handled with appropriate frameworks (Bayesian, etc.) or compute (Monte Carlo, etc.). Of course one is always limited by the available data. But humans are provably very far from fully exploiting the data available to them. An ASI would be able to make predictions with unnerving accuracy over short timescales, and incredible utility for long timescales.

13. Large Concepts

There are certain classes of concepts which can make sense to a human, but which are simply too “large” for a human to really think about. One can imagine extremely large numbers, high-dimensional systems, complex mathematical ideas, or long chains of logical inference being illegible to a human. The individual components make sense (and can be thought about through analogy), but they cannot be “thought about” (visualized, kept all in memory at once) by a human owing to their sheer size.

But, in principle, an AI could be capable (larger working memory, native visualization of high dimensions, etc.) of intuitively understanding these “large” concepts.

14. Tricky Concepts

There are some intellectual concepts that are quite difficult to grasp. For the most complex, we typically observe that only a subset of humans can be taught to meaningfully understand the concept, with an even small subset being sufficiently smart to have discovered the concept. One can think of physics examples (relativity, quantum mechanics, etc.), math examples (P vs. NP, Gödel incompleteness, etc.), philosophy examples (consciousness, etc.), and so on.

If AGI is possible, there is no reason not to expect AI to eventually be smart enough to understand all such concepts, and moreover to be of a sufficient intelligence-class to discover and fully understand more concepts of this type. This is already super-human with respect to average human intelligence.

Plausibly, as AI improves, it will discover and understand “tricky concepts” that even the smartest humans cannot easily grasp (but which are verifiably correct).

15. Unthinkable Thoughts

Are there concepts that a human literally cannot comprehend? Ideas they literally cannot think? This is in some sense an open research question. One can argue that generalized intelligence (of the human type) is specifically the ability to think about things symbolically; anything meaningfully consistent can be described in some symbolic way, hence in a way a human could understand. Conversely, one could argue that Gödel incompleteness points towards some concepts being unrepresentable within a given system. So, for whatever class of thoughts can be represented by the system of human cognition, there are some thoughts outside that boundary, which a greater cognitive system could represent.

Operationally, it certainly seems that some combination of large+tricky concepts could be beyond human conception (indeed we’ve already discovered many that are beyond the conception of median humans). So, it seems likely that there are thoughts that a sufficiently powerful mind would be able to think, that we would not be able to understand. What advanced capabilities would such thoughts enable? It’s not easy to say. But we do know, from the course of human history, that progressively more subtle/refined/complex/powerful thoughts have led to corresponding increases in capabilities (math, science, technology, control over the world, etc.).

16. Emergence

The enumerated modes of increased intelligence will, of course, interact. A motif we can expect to play out is the emergence of enhanced capabilities due to synergy between components; some kind of “whole greater than the sum of the parts” effect. For humans, we of course see this, where a synergy between “raw intelligence” and “cultural scaffolding” (education, ideas, tools, etc.) leads to greatly improved capabilities. For ASI, the advantages in multiple directions could very well lead to emergence of surprising capabilities, such as forecasting that feels precognitive or telepathic, or intuition that feels like generalized genius.

Conclusion

The exact nature of future ASI is unknown. Which of the enumerated “advantages” will it possess? How will they interact? To what extent will capabilities be limited by available computation, or coherence among large computational systems (e.g. lag times for communicating across large/complex systems)? These are unknowns. And yet, it seems straightforward to believe that an ASI would exhibit, at a minimum, a sort of “collection of focused geniuses” type of super-intelligence, where for any given task that it seeks to pursue, it will excel at that task and accomplish it with a speed, sophistication, and efficiency that our best organizations and smartest people can only dream of.

Overall, we hope this establishes that ASI can, indeed, be inordinately capable. This makes it correspondingly inordinately useful (if aligned to humans) and inordinately dangerous (if not).

Posted in AI, Philosophy | Tagged , | 1 Comment

AI News 2024-10-03

General

  • A reminder that Epoch AI has nice graphs of the size of AI models over time.
  • Microsoft blog post: An AI companion for everyone. They promise more personalized and powerful copilots. This includes voice control, vision modality, personalized daily copilot actions, and “think deeper” (iterative refinement for improved reasoning).
  • OpenAI Dev Day: realtime, vision fine-tuning, prompt caching, distillation.
  • OpenAI have secured new funding: $6.6B, which values OpenAI at $157B.

Policy/Safety

  • California governor Gavin Newsom vetoed AI safety bill SB1047. The language used in his veto, however, supports AI legislation generally, and even seems to call for more stringent regulation, in some ways, than SB1047 was proposing.
  • Chatterbox Labs evaluated the safety of different AI models, finding that no model is perfectly safe, but giving Anthropic the top marks for safety implementations.
  • A Narrow Path. Provides a fairly detailed plan for how international collaboration and oversight could regulate AI, prevent premature creation of ASI, and thereby preserve humanity.

Research Insights

  • The context length of an LLM is critical to its operation, setting the limit on how much it can “remember” and thus reason about.
    • A succession of research efforts demonstrated methods for extending context:
    • Modernly, LLMs typically have >100k context, with Google’s Gemini 1.5 Pro having a 2M window. That’s quite a lot of context!
    • Of course, one problem arising with larger contexts is “needle-in-haystack”, where the salient pieces get lost. Attentional retrieval seems to be best for token near the start and end of the context, with often much-worse behavior in the large center of long contexts. So there is still a need for methods that correctly capture all the important parts from long context.
    • Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction. Early LLM layers are used to compress the context tokens, into semantically meaningful but more concise representations. Should allow scaling to larger contexts. (Though one might worry that are some edge-case tasks, this will eliminated needed information/nuance.)
  • Looped Transformers for Length Generalization. Improves length generalization; useful for sequential tasks that have variable length (e.g. arithmetic).
  • Addition is All You Need for Energy-efficient Language Models. Very interesting claims. They show how one can replace floating-point matrix multiplications with a sequence of additions as an approximation. Because additions are so much easier to compute, this massively reduces energy use (95%), without greatly impacting performance. (Which makes sense, given how relatively insensitive neural nets are to precision.) Huge energy savings, if true.
  • Evaluation of OpenAI o1: Opportunities and Challenges of AGI. An overall evaluation of o1-preview confirms that it excels at complex reasoning chains and knowledge integration (while sometimes still failing on simpler problems). o1 represents a meaningful step towards AGI.
  • A few months old, but interesting: The Platonic Representation Hypothesis. Various foundation models appear to converge to the same coarse-grained/idealized representation of reality. And the convergence improves as the models get larger, including across modalities (e.g. language and vision models converge to the same world model). This is partly an artifact of human-generated training data (i.e. they are learning our world model), but also partly due to the intrinsic “useful partitioning” of reality (c.f. representational emergence).

LLM

Audio

Image Synthesis

Video

  • Bytedance unveils two new video models: Doubao-PixelDance and Doubao-Seaweed (examples show some interesting behaviors, including rack focus and consistent shot/counter-shot).
  • Pika release a v1.5 of their model. They have also added Pikaffects, which allow for some specific physics interactions: explode, melt, inflate, and cake-ify (examples: 1, 2, 3, 4, 5, 6). Beyond being fun, these demonstrate how genAI can be used as an advanced method of generating visual effects, or (more broadly) simulating plausible physics outcomes.
  • Runway ML have ported more of their features (including video-to-video) to the faster turbo model. So now people can do cool gen videos more cheaply.
  • Luma has accelerated their Dream Machine model, such that it can now generate clips in ~20 seconds.
  • Runway ML (who recently partnered with Lionsgate) announce Hundred Film Fund, an effort to fund new media that leverage AI video methods.
  • More examples of what genAI video can currently accomplish:

3D

Brain

Hardware

Robots

Posted in AI, News | Tagged , , , , , , , , | Leave a comment

AI News 2024-09-26

General

Research Insights

LLM

Tools

Audio

Image Synthesis

Video

Science

Hardware

Robots

Posted in AI, News | Tagged , , , , , , , , | Leave a comment