Kevin G. Yager | Academic Summary

AI News 2024-12-12

Posted on 2024-12-12 by KevinYager

OpenAI

Dec 5: o1 is out of preview. The updated o1 is faster (uses fewer tokens) while improving performance. And they have introduced a “Pro” version of o1 (thinks for even longer).
- Here’s an example from a biomedical professor about o1-pro coming up with a legitimately useful and novel research idea.
Dec 5: There is now a ChatGPT Pro tier, $200/month for unlimited access to all the best models (including o1 Pro).
Dec 6: Reinforcement Fine-Tuning Research Program. Selected orgs will be able to RL OpenAI models for specific tasks. This is reportedly much more sample-efficient and effective than traditional fine-tuning. It will be reserved for challenging engineering/research tasks.
Dec 9: Sora officially released (examples).
Dec 10: Canvas has been improved and made available to all users.
Dec 11: ChatGPT integration into Apple products.
Dec 12: ChatGPT can pretend to be Santa.

Google

Google releases Gemini 2.0.
- Jules is an experimental code agent.
New “Deep Research” feature can search the web and pull together a coherent research report.
Imagen 3 and Veo image and video models are now available on Googl’es Vertex cloud platform.
Multimodal Live API in Google AI Studio. You can share your webcamera or screen to allow it to provide more directed help. (Example of using it as a research assistant.)

Research Insights

Google DeepMind: Mastering Board Games by External and Internal Planning with Language Models. Search-based planning is used to help LLMs play games. They investigate both externalized search (MCTS) and internalized (CoT). The systems can achieve high levels of play. Of course the point is not to be better than a more specialized/dedicated neural net trained on that game; but to show how search can unlock reasoning modalities in LLMs.
Training Large Language Models to Reason in a Continuous Latent Space. Introduces Chain of Continuous Thought (COCONUT), wherein you directly feed the last hidden state as the input embedding for the next token. So instead of converting to human-readable tokens, the state loops internally, providing a continuous thought.
New preprint considers how “capability density” is increasing over time: Densing Law of LLMs. They find that, for a given task, every 3 months the model size needed to accomplish it is halved. This shows that hardware scaling is not the only thing leading to consistent improvements.

LLM

Meta released Llama 3.3 70B, which achieves similar performance to Llama 3.1 405B. Meta also announced plans for a 2GW datacenter in Louisiana, for future open-source Llama releases.
Ruliad introduces Deepthought 8B (demo), which claims good reasoning for the model size.
Stephen Wolfram released a post about a new Notebook Assistant that integrates into Wolfram Notebooks. Wolfram describes this as a natural-language interface to a “computational language”.
GitIngest is a tool to “turn codebases into prompt-friendly text”. It will take a github repository, and turn it into a text document for easy inclusion into LLM context.
While we haven’t seen a “new class of model” (bigger/better than GPT4) in quite a while, it’s worth remembering the substantial improvements we’ve seen from perfecting the existing systems (from Epoch AI benchmarks). On Ph.D.-level Q&A, over the last year we’ve gone from no-better-than-random to roughly human-expert:

AI Agents

Article: Emergence’s AI orchestrator launches to do what big tech offerings can’t: play well with others. Of course there are many other scaffolding (LangChain, Pydantic, Flow, etc.) and orchestration (ell, swarm, AG2, etc.) frameworks (not to mention commercial attempts thereof: Amazon, Crew AI, MultiOn, etc.). But it’s good to see more development in this space.

Audio

ElevenLabs added GenFM to their web product: you can now generate AI podcasts, and listeners can tune in on the ElevenReader app.

Image Synthesis

Spawning AI is developing an image model based only on public domain data. It will be made available on Source.Plus. Preliminary images seem quite good (examples), suggesting that public data may be enough. Preprint: Public Domain 12M: A Highly Aesthetic Image-Text Dataset with Novel Governance Mechanisms.
Midjourney releases Patchwork, a multi-player world-building tool.

Vision

Nvidia introduces: NVILA: Efficient Frontier Visual Language Models.

Monumental Labs is using AI-enabled robotic stone carving to make Renaissance-style sculpture more common.

Science

Nature writeup: Virtual lab powered by ‘AI scientists’ super-charges biomedical research: Could human–AI collaborations be the future of interdisciplinary studies? Preprint: The Virtual Lab: AI Agents Design New SARS-CoV-2 Nanobodies with Experimental Validation. They use a team of AI assistants to accelerate work.
ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization (video).

Posted in AI, News | Tagged agents, audio, Google, OpenAI, research, Science, vision | Leave a comment

AI News 2024-12-05

Posted on 2024-12-05 by KevinYager

General

The End of Productivity: Why creativity is the new currency of success. The essay argues that focus on pure productivity (and metrics) misses the things that humans value most. And that, potentially, the era of AI will actually shift in an emphasis from human productivity to human creativity being the focus of value.
An interesting experiment (assuming it’s true): an AI jailbreaking contest. An AI agent was tasked with not approving an outgoing money transfer. Anyone can spend a small amount of money to send the AI a message. The money is added to the pool, and the cost-per-message increases slightly. It started at $10/message, and quickly grew to $450/message with a prize-pool of $50k. At that point, someone tricked the AI by sending a message that explained an inverted meaning of approveTransfer. So, they won the money.
- This acts as the usual reminder that modern LLMs are not robust against dedicated attackers that seek to trick them and extract information.
Reportedly: Elon Musk lands priority for Nvidia GB200 delivery in January with US$1.08 billion. Paying a premium to get earlier access to next-gen chips may well be a good strategy.
An interesting blog post by Lilian Weng: Reward Hacking in Reinforcement Learning. Some notes about modern RLHF applied to LLMs (based on this paper):
- RLHF increases human approval, but not necessarily correctness.
- RLHF weakens humans’ ability to evaluate: The error rate of human evaluation is higher after RLHF training.
- RLHF makes incorrect outputs more convincing to humans. The evaluation false positive rate significantly increases after RLHF training.
Andrej Karpathy provides an interesting historical look at how the transformer architecture was invented (c.f. Attention Is All you Need.)
A critical analysis of “openness” in AI: Why ‘open’ AI systems are actually closed, and why this matters. They note that the current version of “open” does not preclude concentration of power.

Research Insights

Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?
Reverse Thinking Makes LLMs Stronger Reasoners. Humans reason not just from problem-to-solution, but also from solution backwards.
Last week saw many results attempting to replicate OpenAI o1’s reasoning ability. Now we also have: o1-Coder: an o1 Replication for Coding (code).

LLM

Amazon enters the fight with Nova (docs, benchmarks). Although not leading on benchmarks, they promise good performance-per-dollar; will be available on Amazon Bedrock.

AI Agents

The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use (code).

Audio

Hume adds a voice creation mode where one can adjust intuitive sliders to pick out the desired voice.
ElevenLabs previously announced intentions to build a conversational AI platform. This capability is now launching; they claim it their interface makes it extremely easy to build a conversational voice bot, and allows you to select the LLM that is called behind-the-scenes.

Video

Google et al. show off: Generative Omnimatte: Learning to Decompose Video into Layers (preprint). It can separate a video into distinct layers, including associating affects (e.g. shadows) with the correct layer (parent object), and inpainting missing portions (e.g. occluded background). Obvious utility for visual effects work: can be used to make a particular person/object invisible (including their shadows), to apply edits to just one component (object or background), etc.
Invideo are demoing a system where a single prompt generates an entire video sequence telling a story (example). I think that creators generally want more granular control of output so they can put together a precise narrative. But there are use-cases where this kind of fully automated generation may make sense.
- It’s easy to look at the output and find the visual or narrative flaws. But also interesting to remember how advanced this is compared to what was possible 6-9 months ago. There is obviously a huge amount of untapped potential in these kinds of systems, as they become more refined.
Runway tease a prototype for a system to enable control over generative video, where videos are defined by keyframes and adjusting the connection/interpolation between them (blog post).
- In October 2023, there were some prototypes of a “prompt travel” idea wherein a video was generated by picking a path through the image-generation latent space. One would define keyframe images, and the system would continually vary the effective prompt to interpolate between them (preprint, animatediff-cli-prompt-travel). This provided a level of control (while not being robust enough to actually enforce coherent temporal physics). Runway’s approach (leveraging a video model) may finally enable the required control and consistency.
Tencent announce an open-source video model: Hunyuan Video (example, video-to-video example).

World Synthesis

World Labs (which includes Fei-Fei Li) is working on 3D world generation from a single image (examples, more examples).
Not to be outdone, Google then announced: Genie 2: A large-scale foundation world model, which can generate playable worlds.

Science

Google Introduces A.I. Agent That Aces 15-Day Weather Forecasts. Scientific paper: Probabilistic weather forecasting with machine learning.

Brain

Whole-brain mapping is advancing. We recently saw release of a fly brain map (140,000 neurons). Now, a roadmap effort claims that whole-brain mapping for mammalian brains should be possible in the coming years.

Hardware

ASML released a hype-video describing the complexity of modern lithography (in particular the computational lithography aspect). There is no new information, but it’s a nice reminder of the nature of the state-of-the-art.
I never grow tired of looking at plots of Moore’s Law:

Robots

MagicLab released a video purporting to show multi-(humanoid)robot collaboration on tasks.

Posted in AI, News | Tagged agents, audio, hardware, LLM, research, robots, video, world synthesis | Leave a comment

AI News 2024-11-28

Posted on 2024-11-28 by KevinYager

General

Google releases an essay on the potential of AI for science: A new golden age of discovery: Seizing the AI for Science opportunity. In addition to outlining an optimistic future (not dissimilar from Dario Amodei’s Machines of Loving Grace), it provides practical insight about what problems are best attacked using modern AI.
Aidan McLaughlin essay: The Problem with Reasoners. He notes three trends that suggest AI will progress more slowly that suggested by naive/optimistic scaling arguments:
- It was hoped that multi-modal models (ChatGPT 4o, voice+text models, etc.) would exhibit significant capability improvement from transfer learning across modalities. This has not borne out.
- Iterative/reasoning models (OpenAI o1, DeepSeek r1, etc.) show that using RL can yield gains in narrow domains with clear metrics (contrived math problems), but we are not seeing evidence of this leading to generalized improvements in intelligence (in areas without easy verification).
- No large model (larger than GPT4 or Claude 3 Opus) have been released, suggesting major challenges there.
Attitudes and perceptions of medical researchers towards the use of artificial intelligence chatbots in the scientific process: an international cross-sectional survey (Nature commentary: Quest for AI literacy). Overall, the study finds substantial interest in AI chatbots among researchers, but also a lack of understanding of these systems.

Research Insights

Replication of “o1-style” chain-of-thought reasoning is heating up:
- Last week saw announcement of DeepSeek-R1-Lite-Preview.
- Update from Walnut Plan’s attempt to replicate o1 (c.f. part 1, code): O1 Replication Journey — Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?
- Paper from Alibaba: Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions.
- Alibaba Qwen releases: Qwen QwQ 32B (weights, demo). This appears to be a separate implementation of the “o1-style” reasoning chain-of-thought approach.
Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models. There is always debate about whether LLMs “truly reason” or “simply memorize”. This paper proposes that reasoning is based on extracting procedures from training data, rather than simply memorizing outputs. So it is a matter of finding, memorizing, and using “templates” rather than specific results.
LLMs Do Not Think Step-by-step In Implicit Reasoning. They argue that while explicit chain-of-thought (CoT) generates stepwise reasoning, implicit reasoning (e.g. model trained to reproduce CoT outputs) does not internally invoke the same stepwise process.
Inference Scaling FLaws: The Limits of LLM Resampling with Imperfect Verifiers. Notes that inference-time scaling is limited by the quality of the verifier (at least for approaches relying on verification).

LLM

Nvidia releases: Hymba Hybrid-Head Architecture Boosts Small Language Model Performance (code). Combines transformer attention mechanism with state-space models (SSMs, c.f. Mamba) to achieve high performance.
Ethan Mollick provides some practical advice for prompting LLMs: Getting started with AI: Good enough prompting (Don’t make this hard).
A sub-culture of AI enthusiasts has developed around the idea of simply giving modern LLMs (limited though they may be) autonomy; or at least semi-persistence by allowing them to run for long time periods. Often, the AIs behave in strange and unexpected ways, as they attempt to continue a token-chain well beyond their original training/design.
- Infinite Backrooms generates extremely long conversations by creating chat-rooms where different LLMs talk to each other endlessly. Conversations often veer into strange and unexpected topics; with some LLMs even outputting tokens describing distress.
- truth_terminal is an 𝕏 handle that is reportedly an LLM given free reign to post. However, there is speculation that the human in charge (Andy Ayrey) is selective about what it actually posts.
- Venture capitalist Marc Andreessen gave the AI a $50,000 no-strings grant (in Bitcoin), so that it could pursue whatever actions it wanted.
- The bot started a memecoin (GOAT) that briefly reached a market cap of $1.3B (currently still at >$700M). The coin’s name is a reference to a (NSFW) shock-meme. The AI itself (or the human behind it) likely netted many million $.
- The AI reportedly “kept asking to play video games”; so it was given access to an “arcade” where the games are text-based games generated by another LLM. You can watch the streaming interactions: Terminal TV.
- It also has its own web-page (that it, ostensibly, authored).
- While it is hard to know how much human tampering is occurring in these implementations, it is interesting to see the bizarre and unexpected outputs that LLMs generate when unleashed.
AI models work together faster when they speak their own language. Letting AI models communicate with each other in their internal mathematical language, rather than translating back and forth to English, could accelerate their task-solving abilities.
- Preprint: DroidSpeak: Enhancing Cross-LLM Communication.
- Although allowing AIs to converse in an invented language could increase efficiency, it undercuts the legibility and auditability aspects of natural-language inter-communication. Overall, this approach could thus hamper both safety and capabilities of complex AI ecosystems.
Anthropic describes Model Context Protocol: an open standard for secure, two-way connections between data sources and AI (intro, quickstart, code).
Anthropic adds a style feature, where it will try to mimic a provided writing example.
Further evidence that model quantization can subtly impact performance: Aider reports that Details matter with open source models.
As a follow-up to last week’s paper on poetry (AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably); Colin Fraser provides this summary graphic, highlighting that humans objectively prefer AI poetry, but when told authorship (real or not), they rate things more highly when (ostensibly) made by humans and lower when (ostensibly) made by AI.

AI Agents

DynaSaur: Large Language Agents Beyond Predefined Actions. The agent improves capabilities over time by progressively writing more functions/code.

Image Synthesis

Black Forest Labs released FLUX.1 Tools, a suite of models to enable more control over image generation/editing (inpainting, outpainting, conditioning).
Runway Frames is a new image model, with good style control.
OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on (code, demo). Allows one to modify a person/character’s clothes in an image.
- There are other codebases to do similar things; e.g.: Kolors Virtual Try-On in the Wild.

Audio

ElevenLabs announces a podcast generator (competing with Google’s Notebook LM).

Video

Meta’s Segment Anything Model 2 (SAM2) has been adapted, adding motion-aware memory, which allows it to do zero-shot video masking (another example): SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory (code).
Runway adds Expand Video, allowing one to change aspect ratio by outpainting (e.g.). Includes prompt guidance, allowing one to change a shot significantly.
LTXStudio announce LTX Video, an open-source video model (code, docs). Although the quality is not quite state-of-the-art, it is remarkably good and it is real-time. Of course, not all generations are excellent; but the real-time generation speed points towards neural world simulation in the not-too-distant future.
Luma Dream Machine v1.6, including Luma Photon image generation and consistent characters.
A group claims to have leaked access to a turbo version of OpenAI’s Sora video model (examples).

World Synthesis

An interesting result: using Runway’s outpainting on video where a person’s face is barely visible (and distorted through refraction); the reconstructed face is remarkably coherent/correct. This implies that the model is implicitly building a valid world model.
Google et al present: CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (project page with examples). Follow up to early CAT3D; but now the 3D objects can evolve in time.

Science

Large language models surpass human experts in predicting neuroscience results (writeup: AI can predict neuroscience study results better than human experts, study finds). This once again shows that LLMs can implicitly learn valid generalizations, picking up on subtle trends spread across a dataset.

Hardware

Epoch AI: Introducing Epoch AI’s Machine Learning Hardware Database.

Robots

Although the Unitree G1 humanoid robot was announced with a price of $16k (c.f.), the latest price chart shows a range of configurations, with prices from $40k to $66k.
Mercedes is running a trial for use of Apptronik robot in their Austin lab.

Posted in AI, News | Tagged agents, hardware, image synthesis, LLM, research, robots, video, world synthesis | Leave a comment

AI News 2024-11-21

Posted on 2024-11-21 by KevinYager

General

Elon Musk’s xAI raising up to $6 billion to purchase 100,000 Nvidia chips for Memphis data center. This is in addition to their existing 100,000 H100 GPU cluster (~100 exaflops FP16). If these are B100 GPUs, that would increase total compute to ~274 exaflops.
A US government commission released a report; among other things, it calls for a Manhattan-Project style AI initiative. (C.f. Leopold Aschenbrenner‘s Situational Awareness.)

Max Tegmark offers a rebuttal to this report: AGI Manhattan Project Proposal is Scientific Fraud. He contends that the report-writers misrepresent the scientific consensus, in that they seem to report that AGI will be easily controlled.

Research Insights

LLM

New study: AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably. At least part of the effect may come from non-experts judging the simpler and more conventional AI poems as being more understandable and superior (and thus human), while the complexity and inconsistency of human-generated poetry is perceived as incoherence.
- Nevertheless, this again shows that for short-form generation, AI has already reached human-level, and can be considered super-human in certain narrow ways.
Mistral releases a new large model (Mistral-Large-Instruct-2411, 123B) and Pixtral Large multimodal model (weights).
DeepSeek announces DeepSeek-R1-Lite-Preview. This is a “reasoning” model (inference-time chain-of-thought) that seems to be similar to OpenAI’s o1. Like o1, it achieves impressive results on math and science benchmarks. Some of the CoT reasoning traces are quite interesting (e.g.). The weights are not yet available, but they claim they will release it open-source.
- Also interesting to consider the rate of progress. A couple years ago, the prediction was we might reach 46% in the MATH benchmark by 2025. Instead, we now have a general LLM getting 92%. And o1 has also scored 97% on a challenging math exam (with novel questions that are nowhere in the training data).

AI Agents

Stripe adds mechanisms for AI agents to trigger payments.
Generative Agent Simulations of 1,000 People (code). They interview humans, using those to define the set of AI agents.
- Builds on their prior work: 2023-10: Generative Agents: Interactive Simulacra of Human Behavior.
AWS releases a multi-agent orchestrator framework.
Paper: Agent-as-a-Judge: Evaluate Agents with Agents. Argues for using evaluation agents in workflows.
Automated-AI-Web-Researcher-Ollama. Code for using local LLMs to automated online research.
Someone is trying to use a team of AI agents to write a full book autonomously. Different agents are responsible for different characters, or different aspects of writing (consistency, researching facts, etc.).

Image Synthesis

A recent survey of 11,000 people has completed: How Did You Do On The AI Art Turing Test? The median score (to differentiate AI and human art) was 60%, a bit above chance. AI art was often preferred by humans. Overall, AI art has already crossing a Turing-Test threshold.

Audio

Suno releases their v4 music generator.
ElevenLabs now offers ability to build conversational AI agents.

Video

Pickle AI is offering a virtual avatar for your meetings ($30/month). You still attend the meeting, and talk when you want. But your avatar pretends to pay attention, and lip-syncs your speech. So this is an alternative to having your camera turned off.
Runway releases some small updates, including longer (20s) video-to-video, vertical aspect ratio for Act-One, and more camera controls.
Current quality of video generations:
- Coca-Cola holiday ad (c.f. McDonald’s commercial, Aug 2024), and parody thereof.
- A Dream Within A Dream (by PZF, selected for the Czech International AI Film Festival).
- Making Friends (by Everett World; see also Childhood Dream and City Echoes).
- Anime: test shots, Ultimate Ceremony, Echoes of Love.
- Echoes of Grace (KakuDrop using Sora).

Science

Sequence modeling and design from molecular to genome scale with Evo. A 7B genomic multi-modal foundation model trained on 2.7 million genomes. It can interpret DNA, RNA, and protein sequences; and can predict across molecular, system, and genomic scales. Can be used to predict effect of mutations, design CRISPR systems, etc.

Hardware

Google has a history of using deep reinforcement learning for automated chip design. This work has been met with some skepticism. Google has now published a rebuttal, claiming that the era of AI chip design is well upon us: That Chip Has Sailed: A Critique of Unfounded Skepticism Around AI for Chip Design.
- April 2020 blog post: Chip Design with Deep Reinforcement Learning.
- June 2021 paper: A graph placement methodology for fast chip design.
- Sept 2023 blog post: How AlphaChip transformed computer chip design.
- August 2024 preprint: ShortCircuit: AlphaZero-Driven Circuit Design (code).

Posted in AI, News | Tagged agents, audio, hardware, LLM, research, Science, video | Leave a comment

AI News 2024-11-14

Posted on 2024-11-14 by KevinYager

General

OpenAI’s data scraping wins big as Raw Story’s copyright lawsuit dismissed by NY court. The crux is that the plaintiffs could not demonstrate a concrete, actual harm from OpenAI’s actions.
An article on Reuters: OpenAI and others seek new path to smarter AI as current methods hit limitations. It repeats the assertions (disputed by many experts in the community) that next-generation models (under development) are under-performing, and that AI labs are hitting data walls. They also emphasize that the path forward involves more “inference-time compute” to unlock reasoning.
- It is interesting to see the article including a quote from Ilya Sutskever, who has been largely quiet in the public sphere, after his departure from OpenAI and founding of SSI.
The AI Semiconductor Landscape.

Lex Fridman interviews Anthropic: Dario Amodei (CEO), Amanda Askell (develops Claude’s personality), Chris Olah (works on mechanistic interpretability).

Research Insights

The Surprising Effectiveness of Test-Time Training for Abstract Reasoning (code). They implement temporary updates to weights at inference-time, using a loss and gradients in the usual (training) manner. They show strong performance on ARC tasks.
Mansi Sakarvadia’s thesis: Towards Interpreting Language Models: A Case Study in Multi-Hop Reasoning. Develops a system to allow the user to inject prompt-specific information into inference, which can improve multi-step reasoning. Also describes Attention Lens, to convert attention heads into interpretable tokens.
Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding (code).

LLM

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (weights, preprint).
Release of: Qwen2.5-Coder Series: Powerful, Diverse, Practical. Currently at the top of the coding leaderboard.

AI Agents

Microsoft introduces: Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks.
Microsoft releases an experimental library: TinyTroupe 🤠🤓🥸🧐: LLM-powered multiagent persona simulation for imagination enhancement and business insights.
Nous Research announces: Introducing the Forge Reasoning API Beta and Nous Chat: An Evolution in LLM Inference. They claim this provides an easy way to take an existing model and run it in a reasoning mode (using inference-time compute).
Mina Fahmi produced this image listing the ways that human and AI could work together:

Video

AutoVFX: Physically Realistic Video Editing from Natural Language Instructions (preprint, code, examples).
Pollo AI has released a video generator. Outputs are quite good, though not quite challenging the state-of-the-art.
Current quality of video generations:
- Plants dancing.
- Insect on tree.
- Trailers for The Silmarillion and The Fall of Gondolin (by Abandoned Films).
- Moody sci-fi.
- Migration (made by combining Runway ML Gen3-Alpha and traditional animation).
- After the Winter (music made using Suno v4).
- Horror: Ridge to Southwest.
- The Gardener (by Machine Mythos).

World Synthesis

ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model. Just two images of a scene are enough to reconstruct a 3D model.

Science

Robots

New Deep Robotics video shows very good terrain navigation from a quadruped-with-wheels design.

Posted in AI, News | Tagged agents, LLM, research, robots, Science, video, world synthesis | Leave a comment

Concise Argument for ASI Risk

Posted on 2024-11-12 by KevinYager

I listened to the debate between Stephen Wolfram and Eliezer Yudkowsky on Machine Learning Street Talk (MLST).

I found the discussion frustrating, since it felt like they were trying to have two very different conversations: Wolfram questioning basic principles and trying to build the argument from the foundations, Yudkowsky taking AI risk as being mostly self-evident and defending particular aspects of his thesis.

Yudkowsky seems reluctant to provide a concise point-wise argument for AI risk, which leads to these kinds of strange debates where he defends a sequence of narrow points that feel mostly disconnected. From his body of work, I infer two general reasons why he does this:

He has learned that different people find different parts of the argument obvious vs. confusing, true vs. false. So rather than reiterate the whole argument, he tries to identify the parts they take issue with, and deal with those. This might work for one-on-one discussions, but for public debates (where the actual audience is the broader set of listeners), this makes it feel like Yudkowsky doesn’t have a coherent end-to-end argument (though he definitely does).
Yudkowsky’s style, in general, is not to just “give the answer,” but rather to lead the reader through a sequence of thoughts by which they should come to the right conclusion. In motivated pedagogy (where the reader is trying to learn), this is often the right way. “Giving the answer” won’t cause the person to learn the underlying pattern; the answer might feel too obvious and be quickly forgotten. Thus one instead tries to guide the person through the right thoughts. But to a resistant listener, this leaves the (incorrect) impression that the person’s arguments are vague.

Let me try to put together a step-wise argument for ASI risk. I think it goes something like:

Humans are actively trying to make AIs smarter, more capable, and more agentic (including giving access/control to real-world systems like computers and robots and factories).
There is no particular ceiling at human intelligence. It is possible in principle for an AI to be much smarter than a human, and indeed there are lots of easy-to-imagine ways that they would outstrip human abilities to predict/plan/make-decisions.
AIs will, generically, “go hard”; meaning they will put maximal effort into achieving their goals.
The effective goals of a powerful optimizer will tend to deviate strongly from the design goals. There are many reasons for this:
- It is hard to reliably engineer something as fuzzy (and, ultimately, inconsistent) as human values.
- Optimizers often have a mis-alignment between the intended goal and the realized inner optimization (inner/outer alignment problem, mesa-optimizers, etc.).
  - The analogy to evolution is often offered: evolution is optimizing for replication of genes, yet enacted human values have only a little to do with that (wanting to have children, etc.); humans mostly care about non-genetic things (comfort, happiness, truth), and are often misaligned to genes (using contraception).
- Even goals perfectly-specified for a modest context (e.g. human-scale values) will generalize to a broader context (e.g. control the light-cone) in an ill-defined way. There is a one-to-many mapping from the small to the large context, and so there is no way to establish the dynamics to pick which exact goals are enacted in the extrapolated context.
In the space of “all possible goals”, the vast majority are nonsense/meaningless. A small subspace of this total space is being selected by human design (making AIs that understand human data, and do human things like solve problems, design technology, make money, etc.). Even within this subspace, however, there is enormous heterogeneity to what the “effective goals” look like; and only a tiny fraction of those possible AI goals involve having flourishing humans (or other sentient minds).
- To be clear, humans will design AIs with the intention that their effective goals preserve human flourishing, but (c.f. #4) this is a difficult, ill-posed problem. The default outcome is an AI optimizing for something other than human flourishing.
A powerful system pursuing goals that don’t explicitly require humans will, generally speaking, not be good for humans. For instance, a system trying to harness as much energy as possible for its computational goals will not worry about the fact that humans die as it converts all the matter in the solar system into solar cells and computer clusters.
A superhuman (#2) system with real-world control (#1) pursuing (with maximum effort, #3) goals misaligned to human values (#4) will try to enact a future that does not include humans (#5). It will, generically, succeed in this effort, which will incidentally exterminate humans (#6).
- Moreover, this isn’t a case where one can just keep trying until one gets it right. The very first ASI could spell ruin, after which one does not get another change. It’s like trying to send a rocket to the moon, without being able to do test flights! (And where failure means extinction.)

This argument has many things left unspecified and undefended. The purpose is not to provide an airtight argument for ASI risk; but rather to enumerate the conceptual steps, so that one can focus a discussion down to the actual crux of disagreement.

Posted in AI, Philosophy | Tagged AI, ASI, safety | 1 Comment

AI News 2024-11-07

Posted on 2024-11-07 by KevinYager

General

Meta reveals that they are training Llama 4, and are using a cluster with >100k H100 GPUs.
Miles Brundage offers another measured blog post: Should AI Progress Speed Up, Slow Down, or Stay the Same? I don’t know, and you don’t, either.
Computerworld article: Agentic AI swarms are headed your way.
- Needless to say, I agree: Towards a Science Exocortex.
Amazon’s new Alexa has reportedly slipped to 2025. It’s surprising, given Amazon’s lead (existing devices in homes, etc.) and considerable resources, that they have not been able to operationalize modern LLMs. Then again, I suppose the legacy capabilities and customer expectations (replacement must work at least as well, in myriad small tasks, as existing offering) slows down the ability to make changes.
- We might be seeing something similar play out with Apple’s promises of AI features.
Google Claims World First As AI Finds 0-Day Security Vulnerability.
New study on impacts of AI to workers: Artificial Intelligence, Scientific Discovery, and Product Innovation. They find that for R&D materials scientists, diffusion models increase productivity and “innovation” (patents), boost the best performers, but also remove some enjoyable tasks.

Research Insights

Agent S: An Open Agentic Framework that Uses Computers Like a Human (code).
- Similar to Anthropic’s recently-announced computer use ability.
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters. Treats model parameters as tokens, so that input queries become attentional lookups to retrieve model parameters. This leads to an efficiency improvement when scaling.
How Far is Video Generation from World Model: A Physical Law Perspective (preprint, code, video abstract). They train a video model on simple physics interactions. The model generalizes perfectly within-distribution, but fails in general when extrapolating out-of-distribution. This implies the model is not learning the underlying physics.
- A valid question is whether they provided enough coverage in training, and enough scale (data, parameters, training compute) to actually infer generalized physics. It’s possible that at a sufficient scale, robust physics modeling appears as an emergent capability.
- Conversely, the implication might be that generalization tends to be interpolative, and the only reason LLMs (and humans?) appear generalized is that they have enough training data that they only ever need to generalize in-distribution.
Mixtures of In-Context Learners. Allows one to extract more value from existing LLMs, including those being accessed via cloud (weights not available). The method creates a set of different “experts” by calling an LLM repeatedly with different in-context examples. Instead of just merging or voting on their final responses, one can try to consolidate their responses at the token level by looking at the distribution of predictions for next token. This allows one, for instance, to provide more examples than the context window allows.
- It would be interesting to combine this approach with entropy sampling methods (e.g. entropix) to further refine performance.
AI swarms require communication between agents, but right now there are many competing methods for multi-agent coordination (Camel, Swarm, LangChain, AutoGen, MetaGPT). Researchers at Oxford have proposed a scheme (Agora) for AI agents can auto-negotiate a structured protocol: A Scalable Communication Protocol for Networks of Large Language Models (preprint).

LLM

Anthropic added visual PDF support to Claude. Now, when Claude ingests a PDF, it does not only consider a textual conversion of the document, but can also see the visual content of the PDF, allowing it to look at figures, layout, diagrams, etc.
Anthropic releases Claude 3.5 Haiku, a small/efficient model that actually surpasses their older large model (Claude 3 Opus) on many benchmarks.

Tools

Google is now making available Learn About, a sort of AI tutor that can help you learn about a topic. (Seems great for education.)

Image Synthesis

The recently released Recraft V3 (highly ranked in blind tests under the “red panda” name) has the ability to generate SVG (e.g.).
Black Forest Labs have released Flux 1.1 Pro Ultra, a high-resolution version of their image system. It can generate some extremely realistic images.

Audio

Hertz-dev is an open-source audio foundation model, that can be adapted to various tasks.

Video

Runway ML “advanced camera controls” are now available on the Gen-3 Alpha Turbo model.
ByteDance unveils a superior lipsync model: X-Portrait 2: Highly Expressive Portrait Animation. Captures complex and dynamic facial performance (examples 1, 2, 3).
Current quality of video generations:

World Synthesis

Neural reproductions of video games are impressive. We’ve seen Doom, Super Mario Bros., and Counter-Strike.
- Now, Decart AI (working with Etched) are showing a playable neural-rendered video game (basically Minecraft). Playable here (500M parameters, code). Right now, this is just a proof-of-principle. There is no way for the game designer to design an experience, and the playing itself is not ideal (e.g. it lacks persistence for changes made to terrain). It feels more like a dream than a video game. But the direction this is evolving is clear: we could have a future class of video games (or, more broadly, simulation environments) that are designed using AI methods (prompting, iterating, etc.), and neural-rendered in real-time. This would completely bypass the traditional pipelines.
  - To underscore why you should be thinking about this result in a “rate of progress” context (rather than what it currently is), compare: AI video 2022 to AI video today. So, think about where neural-world-rendering will be in ~2 years.
- And we now also have GameGen-X: a diffusion transformer for generating and controlling video game assets and environments.

Science

Anthropic’s “Golden Gate Claude” interpretability/control method consists of identifying legible features in activation space. Researchers have applied this mechanistic interpretability to understanding protein language models. They find expected features, such as one associated with the repeating sequence of an alpha helix or beta hairpin (visualizer, code, SAE). More fully understanding the learned representation may well give new insights into proteins.
- More generally, it is likely a very fruitful endeavor to train large models on science data, and search in a feature space for expected features (confirm it learned known physics), and thereafter search for novel physics in the space.

Robots

Xpeng reveal their humanoid robot Iron (promo video, working in factory).

Posted in AI, News | Tagged image synthesis, LLM, research, robots, Science, tools, video, world synthesis | Leave a comment

What is an AI Agent?

Posted on 2024-11-04 by KevinYager

The importance of AI agents continues to grow, which makes it mildly concerning that there is no agreed-upon definition of “AI Agent.” Some people use it to refer to any LLM activation (where “multi-agent” might then just mean chaining multiple LLM calls) whereas others reserve it for only for generally intelligent AI taking independent actions in the real-world (fully agentic). The situation is further confused by the fact that the term “agent” has been used for decades to just refer to a generic software process.

This thread tried to crowd-source a definition. The ones that resonate with me are those that emphasize memory and tool-use, reasoning, and long-running operation on general tasks. So, I offered:

AI Agent: A persistent AI system that autonomously and adaptively completes open-ended tasks through iterative planning, tool-use, and reasoning.

To further refine definitions:

Raw data is used to train a base model, which can be fine-tuned (e.g. into a chatbot). If we scaffold the LLM with tools (document retrieval, software APIs, etc.), we call it an AI Assistant (or a Co-pilot, if we embed it in an existing application or workflow).

We can also exploit iterative deliberation cycles (of many possible sorts) to give the LLM a primitive sort of “system 2” reasoning capability. We can call this a Reasoning AI (such systems are rare and currently primitive, but OpenAI o1 points in this direction). A Reasoning Assistant thus combines iteration with scaffolding.

An AI Agent, then, is a reasoning AI with tool-use, that runs for a long-horizon so that it can iteratively work on complex problems.

Beyond that, we can also imagine multi-agent ecosystems, which work on even more complex tasks by collaborating, breaking complex problems into parts (for specialized agents to work on), and combining results. Finally (and most ambitiously), we can imagine that this “swarm” of AI agents is deeply integrated into human work, such that it feels more like an exocortex.

Posted in AI | Tagged agents, AI, Exocortex, LLM, swarms | Leave a comment

AI News 2024-10-31

Posted on 2024-10-31 by KevinYager

General

Noam Brown (OpenAI) spoke at TED AI on the importance of system 2 thinking for future AI. For poker, he notes how ~20 seconds of thinking gives the same boost as ~100,000× model scaling.
- There is growing evidence that some version of this also holds true for reasoning AI systems (based on LLMs), as seen in o1.
According to the Verge: Google plans to announce its next Gemini model soon. They also repeat the rumor that OpenAI will release a new model (possibly in December 2024).
Another report showing that uptake of genAI is strong: Growing Up: Navigating Generative AI’s Early Years – AI Adoption Report (executive summary, full report).
- 72% of leaders use genAI at least once a week (c.f. 23% in 2023); 90% agree AI enhances skills (c.f. 80% in 2023).
- Spending on genAI is up 130% (most companies plan to invest going forward).
Sundar Pichai indicates that 25% of all new code at Google is generated by AI.
News that OpenAI is seeking to develop its own chips for accelerated AI compute. This is being done in collaboration with Broadcom and TSMC.
The US White House releases an AI memo, calling upon agencies to harness the power of AI.

Research Insights

adi has proposed a new benchmark for evaluating agentic AI: MC bench (code). It consists of having the agent build an elaborate structure in Minecraft. By using humans to A/B rank the visual output, the capability of agents can be ranked.
Anthropic have provided an update to their interpretability work, where the activation space is projected concisely into a higher-dimensional space using sparse auto-encoders (SAE). Now, they posted: Evaluating feature steering: A case study in mitigating social biases. Earlier work showed that they can enforce certain kinds of model behaviors or personalities by exploiting a discovered interpretable feature. Now, they further investigate; focusing on features related to social bias. They find that they can, indeed steer the model (e.g. elicit more neutral and unbiased responses). They also find that pushing too far away from a central “sweet spot” leads to reduced capabilities.
RL, but don’t do anything I wouldn’t do. In traditional training, parts of the semantic space without data are simply interpolated. This can lead to unintended AI behaviors in those areas. In particular, this means when an AI isn’t sure what to do, they do exactly that undefined thing. This new approach tries to consider uncertainty. So when an AI isn’t sure about an action, it is biased towards not taking that action. This captures a sort of “don’t do anything I might not do” signals.
Mixture of Parrots: Experts improve memorization more than reasoning. The “mixture-of-experts” method (of having different weights that get triggered depending on context) seems to improve memorization (more knowledge for a given inference-time parameter budget) but not reasoning. This makes sense; reasoning is more of an “iterative deliberation” process that benefits from single-pass parameters and multi-pass refinement.
The Geometry of Concepts: Sparse Autoencoder Feature Structure. Tegmark et al. report on finding the feature space of LLMs spontaneously organizes in a hierarchical manner: “atomic” structures at small-scale, “brain” organization at intermediate-scale, and “galaxy” distribution at large-scale.

LLM

Anthropic adds an “analysis tool” to Claude, allowing it to write and run JavaScript.
Google’s Notebook LM “generate podcast” feature has spawned some open-source replication efforts: appeared: PDF2Audio (code), Open NotebookLM (code), and podcastfy (and ZenMic. product). Now, Meta released NotebookLlama, a recipe for building your own (example output).
Microsoft describes: Data Formulator: Exploring how AI can help analysts create rich data visualizations (video, code).
Github Spark is a new system for building apps, using natural language (promo video, demo video).
Github Copilot now offers choice of more models (including Anthropic).
Perplexity and Github Copilot announce an integration, allowing users to ask Perplexity questions from within their Copilot dev environment (through an extension).
At OpenAI dev day, they announced some forthcoming o1 features: function calling, developer messages, streaming, structured outputs, image understanding. This would bring o1 up to the tools-capability of their other models.
OpenAI open-sources SimpleQA (code), a benchmark for assessing factuality.
- Paper: Measuring short-form factuality in large language models.
OpenAI release their web search product to a broad range of users.

Audio

MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization.

Image Synthesis

Stable Diffusion released version 3.5 last week, and have now released the Stable Diffusion 3.5 Medium model.
New image model: Recraft V3 (which was tested as “red panda” and was highly ranked by users).

Video

MarDini: Masked Autoregressive Diffusion for Video Generation at Scale (project page with video examples). Seems to capture “physics” (fluids, gas, fire) quite well.

World Synthesis

Autodesk’s Wonder Studio shows off the latest capabilities: mapping of live-action footage into a 3D environment, including world assets and characters with animations (example from beta tester). This gives the artist the best of both worlds: immediate generation of usable assets, with the ability to alter anything (environment, characters, movements, camera position) afterwards.

Robots

Another video of the EngineAI SE01 robot walking around in a rather humanlike way.
A video of Boston Dynamic’s Atlas robot (new, electric version) performing some autonomous tasks.
Physical Intelligence shows off a simple two-armed robot autonomously performing a rather complicated task: laundry (and making coffee, picking up trash, etc.).
- Paper: π0: A Vision-Language-Action Flow Model for General Robot Control.

Posted in AI, News | Tagged audio, image synthesis, LLM, research, robots, video, world synthesis | Leave a comment

AI News 2024-10-24

Posted on 2024-10-24 by KevinYager

General

University of Michigan has a very AI-forward set of tools for their community.
Time magazine has an article (rare for mainstream media) that talks about AGI as a real possibility: Silicon Valley Takes Artificial General Intelligence Seriously—Washington Must Too.
Ethan Mollick posted: Thinking Like an AI. Although it doesn’t contain anything revelatory for those already deeply familiar with modern AI, it is a useful introduction to those who want to understand heuristically what LLMs can do.
Miles Brundage has left OpenAI. He just published a personal blog post: Why I’m Leaving OpenAI and What I’m Doing Next.
- He’s leaving so that he can publish more and more openly, and work on general AI policy in the non-profit space.
- He says: “In short, neither OpenAI nor any other frontier lab is ready, and the world is also not ready. To be clear, I don’t think this is a controversial statement among OpenAI’s leadership, and notably, that’s a different question from whether the company and the world are on track to be ready at the relevant time…”
- “I think the upsides of AI are already big and could be dramatically bigger, as are the downsides.”
- “I think it’s likely that in the coming years (not decades), AI could enable sufficient economic growth that an early retirement at a high standard of living is easily achievable (assuming appropriate policies to ensure fair distribution of that bounty).”
Transluce launches, as a non-profit AI research lab. To kick things off, they released some research:
The US White House issues a memo: Memorandum on Advancing the United States’ Leadership in Artificial Intelligence; Harnessing Artificial Intelligence to Fulfill National Security Objectives; and Fostering the Safety, Security, and Trustworthiness of Artificial Intelligence.

Research Insights

Looking Inward: Language Models Can Learn About Themselves by Introspection. They test whether LLMs can predict their own responses to questions, compared to another model trained only on the model’s outputs (not inner state). The ability of a model to predict outputs of its inner state can be interpreted as a weak sort of introspection.
The previously-mentioned entropix method (initiated by xjdr) is gaining momentum. The basic idea is that instead of just considering the top-k tokens at each step, one looks at the entropy (and variance of entropy) across tokens to better select. (E.g. high uncertainty can be used to trigger deeper chain-of-thought consideration.) See here for more discussion. The latest is a flurry of posts suggesting that this method is significantly improving evals for a variety of open source models. Nothing is certain, since this is rapidly evolving (these volunteers have only been working on it for a couple weeks), and the possibility of honest self-deception is high. Still, this idea remains worth keeping an eye on.
Automatically Interpreting Millions of Features in Large Language Models.
TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling.
A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration. LLMs can learn improved reasoning from “failure” chain-of-thought traces.

Safety/Policy

Anthropic published a blog post: Sabotage evaluations for frontier models. They consider the different ways a model could try to interfere (steer humans, code sabotage, sandbagging, undermine oversight).
OpenAI adds a chief economist to their staff.

LLM

The OpenAI Chat Completion API now supports audio input (allowing one to skip a separate transcription step).
Google’s Notebook LM has capture much attention, in part due to the useful “chat with my PDFs” feature, but mostly the cool “generate podcast” trick. You can now customize the podcast generation.
- But, also, several open-source implementations of the podcasting feature have appeared: PDF2Audio (code), Open NotebookLM (code), and podcastfy. (And a closed product doing something similar: ZenMic.)
MotherDuck have added a “prompt()” function to their SQL database, such that you can weave LLM calls into your SQL lookups.
- BlendSQL appears to be an open-source attempt to do something similar: combine LLM calls with SQL.
Meta released Meta Spirit LM an open source multimodal language model that freely mixes text and speech.
Anthropic announces a new Claude 3.5 Haiku model, as well as a new version of their excellent Claude 3.5 Sonnet model. This new model can “use a computer” (still experimental), available via API.
- Ethan Mollick posts about his experience using this experimental mode.
- An open-source version (using regular Claude 3.5 Sonnet via API) has appeared: agent.exe.
Perplexity plans to release a reasoning mode, where it can agentically search and collate information.

Tools

Perplexity announced a new feature for combined search over user-provided files and the web (video).
Perplexity Pro Search now has a reasoning mode, for handling more complex queries.

Audio

Elevenlabs adds Voice Design, allowing you to generate a new voice by text-prompting what it should sound like.

Image Synthesis

Adobe shows off the ability to do 3D rotations on 2D vector graphic assets.
Stability AI releases Stable Diffusion 3.5.
Ideogram Canvas is a UI for generating images from existing image assets; e.g. region reprompting (tutorial video).
OpenAI released a result on greatly accelerating image generation: Simplifying, stabilizing, and scaling continuous-time consistency models (preprint).
Midjourney released their image editor, allowing genAI transformation of uploaded images/photos (examples from Grimes: 1, 2) and “retexturing” (effectively depth ControlNet).

Video

Meta released MovieGenBench: benchmarks for video generation.
Haiper AI release their 2.0 video model.
Runway ML announces Act One, which allows performance transfer from a video onto a character (without motion capture).
Genmo AI released an open-source video model, Mochi 1, that appears competitive (weights, try).
Release of Open-Sora-Plan-v1.3.0 (example video).
Current quality of video generations:
- Meta Movie Gen examples.
- Emotional range of Minimax.
- Car commercial: Bear.
- Diner conversation.
- Loved and Lost (a meditation on grief).

Science

Meta has released a large dataset on inorganic materials: Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models (code, datasets, checkpoints, blogpost).
An AI model for cancer diagnosis shows promise (96% accuracy).
FutureHouse have used their PaperQA2 tool to generate “Wikipedia-style” articles for all 19,255 human genes.

Hardware

The US TSMC fab is doing well: TSMC’s Arizona Chip Production Yields Surpass Taiwan’s in Win for US Push.

Robots

A video of Fourier’s GR-2 robot standing up.
Video of Engine AI robot walking. As noted, the more upright (locked knees) gait is more energy-efficient, compared to the squatted (bended knee) walking of many other designs.
Clone Robotics continue to pursue their micro-hydraulic bio-mechanical approach to robotics; they now have a torso.

Posted in AI, News | Tagged audio, hardware, image synthesis, LLM, policy, research, robots, safety, Science, tools, video | Leave a comment