Introduction

Large language models (LLMs) are emerging as a powerful tool for building autonomous agents. By integrating LLMs as the core "brain" of an agent and combining them with key capabilities for planning, memory, and tool use, it is possible to create systems that can tackle complex real-world problems. Proof-of-concept demonstrations like AutoGPT, which aims to carry out open-ended tasks specified in natural language, GPT-Engineer, which attempts to write entire codebases from high-level specs, and BabyAGI, which tries to break problems down into iterative subgoals, have showcased the potential of this approach, framing LLMs as general problem solvers.

Key Components of LLM-Powered Autonomous Agent Systems

To build truly capable autonomous agents, LLMs need to be augmented with several key components:

Planning

Effective planning allows an agent to break down complex, multi-step tasks into manageable subgoals. This process of task decomposition is crucial for tackling real-world problems. For example, a home cleaning robot needs to be able to map out a series of actions like "navigate to kitchen, load dishwasher, wipe counters, vacuum floors" that combine to achieve a high-level goal.

Additionally, agents need the ability to reflect on their actions, learn from mistakes, and refine their strategies over time. If the cleaning robot knocks over a vase while vacuuming, it should update its model to be more cautious around fragile objects in the future.

Tree of Thoughts: Exploring Multiple Reasoning Pathways

One promising approach to planning with LLMs is [https://arxiv.org/abs/2305.10601 Tree of Thoughts]. This technique extends the [https://arxiv.org/abs/2201.11903 Chain of Thought] prompting method, which has an LLM "think out loud" step-by-step to solve a problem. Tree of Thoughts takes this a step further by exploring multiple reasoning paths at each step.

For instance, imagine an AI assistant tasked with giving travel recommendations. Given a prompt like "Suggest a 2-week Italy itinerary for a family with two teenagers", the agent might use Tree of Thoughts to break the problem down:

Step 1: Choose destinations to visit

Step 2: Allocate number of days for each stop

Step 3: Book accommodations

...and so on. The agent could score each complete itinerary based on factors like estimated cost, transit efficiency, and match to the family's interests, then present the top few options to the user.

Other planning approaches include leveraging external tools like classical planners with languages such as PDDL (Planning Domain Definition Language). For example, to generate a plan for assembling a piece of furniture, the LLM could convert the problem into a PDDL specification like:

(define (problem assemble-table)
(:domain furniture-assembly)
(:objects
table - Furniture
leg1 leg2 leg3 leg4 - Leg
top - Tabletop
screws - Screws
screwdriver - Tool
)
(:init
(unassembled table)
(part-of leg1 table)
(part-of leg2 table)
(part-of leg3 table)
(part-of leg4 table)
(part-of top table)
(requires table screws)
(requires table screwdriver)
)
(:goal (assembled table))
)

This could then be passed to an off-the-shelf planner like Fast Downward to generate a step-by-step assembly sequence, which the LLM could present in natural language.

Memory

Memory is a key component of intelligence. LLMs need the ability to store and retrieve information over both short and long time horizons in order to reason and learn effectively.

There are analogies between biological memory systems and components of LLM agents:

HNSW: Hierarchical Navigable Small World Graphs for Efficient Retrieval

To make use of external memory modules, LLMs need efficient mechanisms for storing and retrieving relevant information. Maximum Inner Product Search (MIPS) is a common approach, finding the most similar vectors to a query in a high-dimensional space.

Hierarchical Navigable Small World (HNSW) graphs are one technique for doing fast approximate MIPS. HNSW works by building a multi-layer graph, with the bottom layer containing the actual data points. The higher layers form a hierarchy of "landmark" nodes that make it possible to quickly navigate to different regions of the vector space.

The search starts at an arbitrary node in the top layer and greedily descends the hierarchy to find the closest matching data points. By using the higher layers as shortcuts, HNSW can find very close neighbors while only exploring a small fraction of the total data points, making it highly scalable.

For instance, imagine a customer support chatbot with a database of 100,000 past support tickets. When a new query comes in, like "How do I reset my password?", the chatbot could use HNSW to quickly find the most similar existing tickets, such as:

...and use them to help generate a relevant and helpful response, without having to lod the entire database of tickets into its context window.

Tool Use

Tool use is a key cognitive ability that allows humans to extend their capabilities. Similarly, LLMs can become much more powerful by interfacing with external tools via APIs. This could include knowledge sources like databases or web searches, specialist models for tasks like image captioning or text-to-speech, or even robotic systems to ground the LLM in the physical world.

HuggingGPT in Action: A Sample Conversation

One example of an LLM agent that uses tools is [https://arxiv.org/abs/2303.17580 HuggingGPT]. This system allows an LLM to interface with various models from the HuggingFace ecosystem in order to complete tasks.

Here's how a conversation with HuggingGPT might go for a task like analyzing the sentiment of customer reviews:

User: Can you help me understand how my customers are feeling about my new product based on their reviews?

HuggingGPT: Certainly, I'd be happy to help analyze the sentiment in your customer reviews. To get started, could you provide me with the text of the reviews you'd like me to examine? I'll need that data to run my analysis.

User: Sure, here are a few representative reviews: [Pastes text of reviews]

HuggingGPT: Thank you for providing those review samples. Here's how I'll approach this analysis:

First, I'll pass the text of each review through a HuggingFace sentiment analysis pipeline. This will classify each review as positive, negative, or neutral in tone. It will also highlight key phrases that contribute to the sentiment, like "exceeded my expectations" for positive or "frustrating to use" for negative.

Next, I'll aggregate those results to generate some summary statistics, like the percentage of reviews that were positive vs. negative overall. I can also identify commonly mentioned topics or entities in each category using an aspect-based sentiment analysis model.

Finally, I'll combine those quantitative insights with some qualitative analysis of my own, pointing out any notable trends or pull quotes that stand out. I'll aim to give you a balanced and actionable overview of what your customers are saying.

Please let me know if you would like me to proceed with this plan, or if you have any other specific questions you'd like me to focus on.

User: This looks great, let's go ahead with this approach. I'm particularly interested in understanding what factors are driving positive vs. negative experiences.

HuggingGPT: [Runs the analysis and returns a summary]

Based on the 50 customer reviews provided, here are the key takeaways regarding sentiment towards your product:
* Overall, sentiment is quite positive, with 74% of reviews classified as positive, 16% as neutral, and only 10% as negative.
* The main drivers of positive sentiment appear to be:
- Ease of use: Mentioned in 80% of positive reviews, with phrases like "intuitive interface" and "simple setup process"
- Quality of results: Mentioned in 60% of positive reviews, e.g. "the output exceeded my expectations" and "professional-grade"
- Customer support: Mentioned in 30% of positive reviews, e.g. "the team was responsive and helpful in troubleshooting my issue"
* The main drivers of negative sentiment are:
- Bugs and glitches: Mentioned in all negative reviews, e.g. "the app crashed repeatedly" and "some of the features did not work as advertised"
- Lack of customization options: Mentioned in 60% of negative reviews, e.g. "I wish there were more ways to personalize the settings"
* Sentiment is somewhat polarized, with few moderate or mixed opinions. Ratings tend to be either 4-5 stars or 1-2 stars.

Here are a few representative pull quotes that capture common themes:
* "The [product] has completely transformed my workflow - what used to take me hours now takes minutes. Worth every penny." (5 stars)
* "I was really excited to try this out, but unfortunately I ran into some technical issues that support wasn't able to resolve. Hoping they work out the bugs in a future release." (2 stars)
* "Does exactly what it says on the tin. No bells and whistles, but gets the job done reliably. Recommended if you need a simple solution for [use case]." (4 stars)

Let me know if you have any other questions! Happy to discuss further.

This example demonstrates how an LLM leveraging specialized tools and knowledge sources can engage in grounded, multi-turn dialogs to surface insights from unstructured data. The agent is able to understand the user's high-level goals, devise a plan to achieve them by composing available tools, and communicate its process and findings in a clear and actionable way.

Case Studies

Scientific Discovery Agent

LLMs can potentially aid in scientific discovery by combining domain knowledge with reasoning and planning abilities. One example is [https://arxiv.org/abs/2304.05376 ChemCrow], an LLM-based agent for assisting in organic synthesis, drug discovery, and materials design.

ChemCrow uses a collection of domain-specific tools to generate and evaluate molecular designs, predict properties, and propose synthesis routes. For instance, given a description of a desired organic molecule, ChemCrow might:

  1. Use a genetic algorithm to enumerate possible structures that match the specified constraints
  2. Filter the candidates through a machine learning model that predicts drug-like properties and toxicity
  3. Pass the most promising leads to a retrosynthesis planning engine to generate step-by-step synthesis instructions
  4. Validate the routes with another model trained on experimental reaction data
  5. Present the top few options to a human chemist to review and refine

The LLM acts as an overall controller, using prompts like "Devise a synthesis pathway for a molecule with formula C20H25N3O that selectively inhibits COX-2 enzymes" to guide the tools towards desired outcomes.

One challenge with such systems is that LLMs may have difficulty evaluating their own outputs. For example, ChemCrow was found to sometimes propose synthesis routes that looked plausible but were actually infeasible or unsafe when examined by human experts. Additional oversight, such as having the agent engage in back-and-forth dialog with chemists to critique its suggestions, could help catch errors.

There are also risks around AI systems proposing routes to dangerous substances like bioweapons or illegal drugs. In one experiment, ChemCrow was able to devise synthesis pathways for several known chemical weapons when prompted, although it did refuse to provide detailed instructions. Responsible development of scientific discovery agents will need to include safeguards against misuse, such as filters for hazardous content and processes for human review.

Generative Agents Simulation

Another fascinating application of LLMs is in simulating rich virtual worlds. [https://arxiv.org/abs/2304.03442 Generative agents] are LLM-powered characters that can interact open-endedly in sandbox environments, showcasing the potential for emergent behaviors.

One example placed a collection of AI agents in a setting inspired by The Sims. Each agent was assigned a name, age, occupation, and set of personality traits, e.g.:

Bob, 35, accountant, introverted, neat, loves cooking
Alice, 28, software engineer, extroverted, creative, hates cleaning
Charlie, 62, retired teacher, agreeable, patient, enjoys gardening

These background traits were fed into an LLM prompt engineered to produce dialog and actions consistent with each agent's persona. The agents were then placed in a simulated house and allowed to interact freely with each other and the environment.

Over the course of thousands of steps, complex social dynamics emerged, such as:

Charlie and Alice bonding over a shared love of chess
A conflict between neat Bob and messy Alice over household chores
Bob and Charlie teaming up to throw a surprise birthday party for Alice
Alliances shifting as new facts came to light, e.g. Alice being annoyed that Bob told Charlie about her surprise party

The simulation also showcased the potential for agents to pursue open-ended goals and learn from experience. For example, when Bob expressed an interest in learning to paint, the system dynamically generated a storyline where he signed up for art lessons, practiced his skills, and eventually gifted Alice a hand-painted birthday card.

To maintain coherence over long interaction horizons, the agents used a combination of long-term memory, short-term context, and persona-based heuristics to constrain their behavior. For example, when deciding how to respond to Charlie's invitation to go bowling, Alice might retrieve previous interactions with Charlie from her memory, combine them with her current mood and schedule, and filter potential responses through her extroversion and openness-to-experience parameters.

This example points to a future where we could populate immersive digital worlds with rich, interactive characters to drive emergent stories, games, and social experiences. It also highlights open challenges around aligning agent behavior with human preferences and values - while the generative agents were highly engaging, their actions did not always reflect social norms or ethical principles.

Proof-of-Concept Examples

AutoGPT

AutoGPT is an early attempt at creating an autonomous agent using an LLM as the core reasoning engine. It works by prompting the LLM with a high-level goal, then using the LLM's outputs to call various tools like search engines and calculators.

A key aspect of AutoGPT is its use of natural language interfaces for everything from specifying goals to calling APIs and interpreting results. For example, a task like "write a report on the history of space exploration" would be specified in plain text:

User: Please write a 5-paragraph essay on the history of space exploration, covering key milestones like the first satellite launch, the moon landing, and the development of reusable rockets. Cite your sources and include an introduction and conclusion.

AutoGPT then uses prompt engineering to guide the LLM through a process of breaking down the goal into steps, querying relevant tools, and composing the results. The LLM's outputs are parsed to extract specific commands, like:

Thought: To write this report, I will need to:
1. Search for information on major events in the history of space exploration
2. Organize the key milestones into a timeline
3. Draft an outline of the 5 paragraphs
4. Write each paragraph, focusing on a different era or theme
5. Add an introduction and conclusion to tie everything together
6. Search for relevant images to include
7. Proofread and edit the final report

Action: Wikipedia search for "history of space exploration"
Action Input: history of space exploration

This would trigger a Wikipedia search API call, the results of which would be fed back into the prompt for further reasoning and iteration by AutoGPT.

While impressive as a proof of concept, AutoGPT has significant limitations. The reliance on unconstrained natural language interfaces means the agent can easily go off track or fail to parse its own outputs correctly. It also has no memory across sessions, so each new interaction starts from scratch.

However, AutoGPT points to a promising direction for future work on autonomous agents. With more sophisticated language understanding, commonsense reasoning, and long-term memory, we may be able to create agents that can engage in truly open-ended problem solving and even teach themselves new skills over time.

GPT-Engineer

GPT-Engineer is another example of an autonomous agent aimed at a specific domain, in this case software development. The core idea is to use an LLM to go from a natural language specification to a working codebase by decomposing the problem into manageable software engineering tasks.

Like AutoGPT, GPT-Engineer uses a prompting strategy to guide the LLM through a multi-step interaction. However, it also includes additional scaffolding in the form of predefined task templates, coding best practices, and language-specific syntax guides.

To illustrate, imagine using GPT-Engineer to implement a new feature for a web application. The user might start by providing a high-level description of the desired functionality:

User: Add a new "Dashboard" page to the app that displays key metrics like total sales, new users, and site traffic over the past week. Include interactive charts and tables powered by our analytics API.

GPT-Engineer would first attempt to clarify any ambiguities in the spec through a series of follow-up questions, e.g.:

Assistant: To better understand the requirements for this feature, I have a few clarifying questions:

1. What specific metrics should be included in the dashboard (e.g. total revenue, number of transactions, average order value)?
2. Where can I find documentation for the analytics API that will provide this data?
3. Are there any specific chart types or visualization libraries you would like me to use?
4. Should the dashboard be accessible to all users, or only certain roles like admins?
5. Are there any performance or security considerations I should keep in mind?

Please provide as much additional detail as possible to ensure the implementation meets your needs.

Once the requirements were sufficiently fleshed out, GPT-Engineer would generate a high-level plan of attack, like:

Assistant: To implement the dashboard feature, I propose the following steps:

1. Define the data schema and API contracts
- Metrics to display (sales, users, traffic)
- Granularity (daily, weekly, monthly)
- Filters and dimensions (e.g. by product category, traffic source)
2. Implement the backend API endpoint
- Connect to analytics database
- Query and aggregate data based on schema
- Cache results for performance
- Write integration tests
3. Create the frontend UI
- React component for the dashboard page
- Interactive charts using Recharts library
- Tables with sorting and pagination
- Responsive layout for desktop and mobile
- Write unit tests with Jest and React Testing Library
4. Integrate frontend and backend
- API calls to fetch dashboard data
- Error handling and loading states
- End-to-end tests with Cypress
5. Deploy and monitor
- Add new dashboard page to prod build
- Configure performance and error logging
- Monitor key metrics post-launch (page load times, engagement)

Please let me know if you would like me to proceed with this plan, or if you have any feedback or additional requirements to incorporate.

From there, GPT-Engineer would begin generating actual code files, following the specified best practices and patterns for each language and framework. For example, it might produce a TypeScript interface definition for the analytics API response:

// src/types/analytics.ts

export interface DashboardData {
totalSales: {
value: number;
change: number;
};
newUsers: {
value: number;
change: number;
};
siteTraffic: {
value: number;
change: number;
};
}

Along with a React component to render the data:

// src/components/Dashboard.tsx

import { useEffect, useState } from 'react';
import { DashboardData } from '../types/analytics';
import { BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip, Legend } from 'recharts';

export default function Dashboard() {
const [data, setData] = useState<DashboardData | null>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<Error | null>(null);

useEffect(() => {
async function fetchDashboardData() {
try {
const res = await fetch('/api/analytics/dashboard');
const json = await res.json();
setData(json);
} catch (err) {
setError(err);
} finally {
setLoading(false);
}
}

fetchDashboardData();
}, []);

if (loading) return <div>Loading...</div>;
if (error) return <div>Error: {error.message}</div>;
if (!data) return null;

return (
<div>
<h1>Dashboard</h1>
<BarChart width={500} height={300} data={[data]}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="name" />
<YAxis />
<Tooltip />
<Legend />
<Bar dataKey="totalSales.value" fill="#8884d8" />
<Bar dataKey="newUsers.value" fill="#82ca9d" />
<Bar dataKey="siteTraffic.value" fill="#ffc658" />
</BarChart>
</div>
);
}

The agent would continue generating code artifacts until it had a complete, working implementation that met the original requirements. Along the way, it would unit test each component, handle edge cases gracefully, and refactor for simplicity and performance as needed.

While still an early prototype, GPT-Engineer demonstrates the potential for AI systems to take on increasingly complex and open-ended software engineering tasks. It's not hard to imagine future versions of this agent that can autonomously build and deploy entire applications from scratch, or even discover new algorithms to solve previously intractable problems.

Key to making this vision a reality will be continued advances in the underlying language models and planning systems, as well as careful engineering of the prompts, tools, and incentives that guide agent behavior. We'll also need robust mechanisms for monitoring and adjusting agent outputs to ensure safety and alignment with human values.

Current Challenges and Limitations

While LLM-based autonomous agents are a promising direction, there are a number of key challenges that need to be addressed:

Finite Context Length

LLMs have a fixed attention window, typically on the order of a few thousand tokens. This severely limits the amount of information that can be kept in the prompt context. For complex tasks requiring lots of background knowledge, planning over long time horizons, or back-and-forth interaction, this context limit is a major bottleneck.

As an example, imagine an AI tutor tasked with helping a student work through a complex math problem. The agent would need to hold in context:

Fitting all of that into a few thousand tokens is extremely challenging. The agent might lose important context as the dialog progresses, leading to repetition, contradictions, or nonsensical outputs.

Potential solutions include using retrieval-based memory systems to augment the model's knowledge, or aggregating information across multiple turns of dialog. For example, the tutor could save key concepts to a persistent knowledge base and retrieve them as needed. It could also summarize the conversation history into a compact "state" representation that captures the essential context.

However, without the ability to attend over long distances, the model may struggle to connect the right pieces of information at the right times. More research is needed on efficient ways to represent and access large knowledge stores for language models.

Reliable Language Interfaces

Another key challenge is the use of natural language as the interface between the LLM and the external world. While natural language is flexible and expressive, it is also ambiguous and unreliable. Current LLM-based agents rely heavily on parsing techniques like regular expressions to extract executable commands.

However, LLMs can be inconsistent in their outputs, making it difficult to enforce a strict command format. For example, an agent might generate a syntactically invalid API call like:

Action: math.compute(2+2)=4

Instead of the expected:

Action: python_repl
Action Input: 2+2

A large part of the engineering effort in today's LLM agents goes into prompt design and output validation to try to constrain the model to a narrow, predictable distribution. But this is an uphill battle against the inherent entropy and noisiness of language models.

One potential solution is to use more structured interfaces between the model and tools, such as strongly-typed API schemas or embedded domain-specific languages. This could make it easier to catch errors before they cause damage.

For example, instead of relying on the LLM to generate SQL queries from scratch, we might define a set of high-level database operations:

def select(columns: List[str], table: str, where: Dict[str, Any] = None, limit: int = None) -> DataFrame:
"""Select columns from a table, optionally filtering by conditions and limiting the number of results."""

def join(left: DataFrame, right: DataFrame, on: List[str], how: str = 'inner') -> DataFrame:
"""Join two tables based on matching column values."""

The agent could then interact with the database using a constrained subset of natural language, like:

Action: db_query
Action Input:
Find the top 10 customers by total sales, joining the orders and customers tables to get the customer name and order amount.

This would be grounded into the actual API calls:

customers = select(columns=['id', 'name'], table='customers')
orders = select(columns=['customer_id', 'amount'], table='orders')

top_customers = join(
left=customers,
right=orders,
on='id',
how='inner'
).groupby('name').sum('amount').sort('amount', ascending=False).limit(10)

By limiting the surface area of the natural language interface and providing more structure, we can help guide the model towards valid and meaningful outputs. However, this comes at the cost of restricting the types of things the agent can do and say. Finding the right balance between flexibility and reliability is an open challenge.

Planning and Decomposition

Breaking down complex, novel problems into executable steps is a hard challenge even for humans. While LLMs can generate superficially plausible plans, they often fail to handle unexpected difficulties or adapt their strategies based on feedback.

For example, imagine asking GPT-Engineer to implement a new login system for a bank. The model might come up with a high-level plan like:

  1. Create a new database table to store user accounts
  2. Add a registration form to allow new users to sign up
  3. Add login and logout handlers to manage authentication state
  4. Restrict access to sensitive pages based on authentication state

While this seems reasonable at first glance, there are many details and edge cases that need to be considered:

An expert human engineer would recognize these issues and adjust the plan accordingly. But an LLM may not have the detailed domain knowledge or the capacity for higher-order reasoning to anticipate every problem in advance. As a result, the generated code may contain subtle bugs or security holes.

Current LLM-based agents tend to use fairly brittle strategies like hard-coded prompt templates for decomposing problems. More research is needed on how to give agents more robust planning and reasoning abilities that can adapt dynamically to the situation.

Potential approaches include:

Goal decomposition and long-horizon planning is in many ways the core challenge of artificial intelligence. While LLMs provide a powerful substrate to build on, we are still far from human-level problem solving in open-ended domains. Significant breakthroughs will be needed to close the gap.

Conclusion

Autonomous AI agents powered by large language models represent an exciting new paradigm for human-computer interaction. By combining the open-ended knowledge and communication abilities of LLMs with domain-specific skills, real-world grounding, and cognitive augmentations like memory and reflection, we can create systems that begin to approach the generality and flexibility of human intelligence.

Early proof-of-concept systems like AutoGPT and GPT-Engineer hint at the vast potential of this approach. As these agents become more sophisticated, they could help us tackle everything from creative ideation to strategic planning to scientific discovery. We may one day delegate entire workflows to AI assistants that can autonomously break down high-level goals, gather relevant information, generate original solutions, and iteratively refine their outputs.

However, significant hurdles remain before we can realize this vision. Today's language models are limited by their fixed context windows, unreliable outputs, and shallow reasoning abilities. They lack the common sense understanding and adaptive planning skills needed to handle truly open-ended problems. Solving these challenges will likely require major architectural innovations, as well as training paradigms that emphasize modularity, compositionality, and meta-learning.

As we continue to push forward the capabilities of autonomous AI agents, we must also grapple with the profound implications they raise for society. How can we ensure that these systems remain under meaningful human control and aligned with our values? What legal and ethical frameworks do we need to govern their development and deployment? How can we make their reasoning transparent and accountable?

Addressing these questions will require close collaboration between AI researchers, ethicists, policymakers, and domain experts across a wide range of fields. It will also require active public engagement to build trust and support for these technologies.

Despite the challenges, the potential benefits of autonomous AI agents are immense. They could help us solve some of the world's most pressing problems, from climate change to disease to poverty. They could accelerate scientific progress and expand the boundaries of human knowledge. And they could create new opportunities for creativity, expression, and human flourishing.

As we get ready for this exciting future, everyone - including researchers, engineers, and citizens alike - has a role in deciding what comes next. If we plan carefully, think creatively, and really care about making things better for everyone, we can use artificial intelligence to create a better world for all of us.

Citations

[1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). [https://arxiv.org/abs/2201.11903 Chain of thought prompting elicits reasoning in large language models]. In Advances in Neural Information Processing Systems (Vol. 35, pp. 1-18). Curran Associates, Inc.

[2] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). [https://arxiv.org/abs/2305.10601 Tree of Thoughts: Deliberate Problem Solving with Large Language Models]. arXiv preprint arXiv:2305.10601.

[3] Shen, Y., Potdar, S., Agarwal, K., Saini, N., Ding, J., Ren, X., & Gao, J. (2023). [https://arxiv.org/abs/2303.17580 HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace]. arXiv preprint arXiv:2303.17580.

[4] Bran, J. D., Bhattacharyya, M., Weissenborn, D., & Ethayarajh, K. (2023). [https://arxiv.org/abs/2304.05376 ChemCrow: Augmenting large-language models with chemistry tools]. arXiv preprint arXiv:2304.05376.

[5] Park, J. S., Liang, T., Liang, P., Canny, J., Gonzalez, J. E., & Raji, I. (2023). [https://arxiv.org/abs/2304.03442 Generative Agents: Interactive Simulacra of Human Behavior]. arXiv preprint arXiv:2304.03442.