AI Agents and Automation: Multiagent Frameworks and My Observations

Generated by ChatGPT using keywords “Open source frameworks for agentic AI”

In today’s IT landscape, AI agents are a hot topic. New agent frameworks and innovations seem to emerge almost daily in newsletters and discussions.

In this blog, I want to share my observations on some open-source multi-agent frameworks, providing readers with a quick view of current approaches to build AI agents.

The agent frameworks I evaluated can be found at the end of this blog.

Generic vs Domain Specific

Most of these agent frameworks are designed to build general-purpose agents, while some are for specific domains. For example:

1. MetaGPT focuses on IT development, transforming requirements into user stories, APIs, data structures, code, and test cases.
2. OpenHands serves as a software development AI platform, capable of modifying / coping codes, executing commands, browsing the webs, calling APIs.
3. Pipecat is a framework to build multimodal conversational agents.
4. Eliza is a Web3-friendly AI Agent OS that is now expanding its scope to be more general-purpose.

Observations:

1. In the current AI community, both generic and domain-specific agent frameworks are gaining popularity, though the majority are designed to be general purpose.
2. Nearly all of these frameworks allow users to configure agents with a variety of LLM models.
3. Looking ahead, I believe we will see a growing number of domain-specific agent frameworks emerging in the community and domain specific markets, for example, customer care, retail.

Multiagent Flows and Design

All these frameworks support multi-agent, with most organizing agents into a hierarchical structure like:

actions/tasks -> agents -> (orchestrate, supervise) agents.

And their workflows follow a structure similar to the architecture diagram in this Gartner blog:

https://www.gartner.com/document/5799415

Just, different frameworks use various terms for actions, agents, and supervisory agents. For example:

1. OpenAI Swarm introduces the concept of “routine” for flows and “handoff” for communication (data transfer) between agents.
2. AgentLite uses a “ManagerAgent” to oversee a set of agents within a “teamagents” list.
3. Bee Agent Framework employs a “workflow” to host multiple agents.
4. Microsoft AutoGen organizes multiple agents under a “team.”
5. AWS Multi-Agent Orchestrator, by its name, utilizes an “orchestrator” to manage multiple agents, leveraging agent characteristics and conversation history to select the most suitable agent for a task.
6. Hugging Face smolagents enables an agent to host multiple (sub) agents and tools for various actions.
7. crew.ai supports “Tools” and “Agents,” using “Process” and “Crew” to manage multiple agents and execute tasks.
8. AutoGPT defines “agent” and “workflow,” using “block” for actions.
9. Pipecat uses a “pipeline” and “flow_manager” to structure the flow of voice and multimodal conversations.

However, some frameworks adopt different design approaches. For example:

PydanticAI’s multi-agent application support operates at four levels:

1. Single Agent Workflows — Managing the tasks and actions owithin a single agent.
2. Agent Delegation — Agents manually delegate tasks to other agents via tools, facilitating collaboration between agents.
3. Programmatic Agent Handoff — In this case, one agent runs a task and then hands off control to another agent through application codes.
4. Graph-Based Control Flow — For more complex scenarios, a graph-based state machine is utilized to control the execution flow of multiple agents.

Part of an image generated by ChatGPT with “human society”

Microsoft TinyTroupe, which is different from other frameworks, simulates real-world personas through a unique framework:

Microsoft TinyTroupe is an experimental Python library that allows the simulation of people with specific personalities, interests, and goals. These artificial agents — TinyPersons - can listen to us and one another, reply back, and go about their lives in simulated TinyWorld environments.

Some key concepts in TinyTroupe include:

1. TinyPerson: An agent designed to simulate a person with specific personality traits, interests, and goals.
2. TinyWorld: An environment that simulates the interactions between multiple TinyPersons. A TinyWorld instance enables multi-agent interactions and flows among the different agents.
3. TinyTool: Tools that TinyPersons can use to complete tasks within their simulated world.

MetaGPT follows a similar approach to TinyTroupe by simulating real-world personas, but with a focus on software engineering roles and teams. Its multi-agent framework allows for the definition of distinct personas, such as coder, tester, and reviewer, each performing tasks like coding, writing test cases, and reviewing code. These personas are organized into a “team” that can hire agents to execute a project based on a given input idea. In my view, MetaGPT adopts a style of "mixture of experts" approach, where specialized agents work together to solve specific tasks.

team = Team()
    team.hire(
        [
            SimpleCoder(),
            SimpleTester(),
            SimpleReviewer(),
        ]
    )

    team.invest(investment=investment)
    team.run_project(idea)
    await team.run(n_round=n_round)

Eliza is a multi-agent framework designed to build and deploy AI agents with consistent personalities across platforms like Discord, Twitter, and Telegram. As the framework is focused on social media interactions, its Agent object includes a Character attribute, which allows the agent to be defined as a specific persona. For example, an agent can represent someone like Jingdong (me), with distinct characteristics for interaction on these platforms.

Observations:

1. While the detailed designs vary across frameworks, the core approaches of most these frameworks remain fundamentally similar.
2. The hierarchical structure will extend. For example, for the tasks -> agents -> orchestrate agents design, I believe it will extend to have “agent mesh”; for TinyTroupe’s persona design, I think it may introduce more levels like TinyTeam, TinyOrg or TinySociety between TinyPerson and TinyWorld to make it more like real world.
3. Multi-agent workflows are typically created manually within these frameworks, example like LangGraph code snippet below, with automation being a potential next step for future enhancements.

graph_builder = StateGraph(State)
graph_builder.add_node("chatbot", chatbot)
graph_builder.add_conditional_edges("chatbot", tools_condition)
graph_builder.add_edge(START, "chatbot")
... ...

4. Workflow updates or enhancements often require modifications to the source codes. Once a workflow is created, it cannot be dynamically updated or modified during runtime. Another improvement area for future!

5. Most frameworks offer limited memory or session support, except LangGraph, which provides more robust memory and checkpoint capabilities.

6. The reliability of communication among agents — such as whether data can be dropped or if guarantees like exactly-once or at-least-once delivery are ensured — remains unclear.

Agent Deployment and Execution

Generated by ChatGPT with keywords “multiagent flows and deployment”

Most of these open-source frameworks support easy deployment of agents locally through simple commands or function calls. For example:

• IBM Bee Agent: agent.run() and workflow.run()
• CrewAI: Crew.kickoff()
• OpenAI: Swarm().run()
• AutoGen: team.run_stream()

OpenAI Swarm supports defining a method run_full_turn(agent, messages) to enable more customized agent execution, including communication among agents.

LangGraph can integrate with the LangGraph Platform, a commercial solution for deploying agent-based applications to production. This allows long-running deployments and the ability to stream results back to end users, even as a background process. LangGraph support parallel processing with multiple approaches.

Microsoft TinyTroupe simulates real-world personas, their communications, and actions. Its design diverges from the general IT deployment approaches used by many other agent frameworks. TinyTroupe enables persona interactions within a TinyWorld environment, which, in my view, creates a more virtual world-style experience.

AWS Multi-Agent Orchestrator supports orchestrator deployment both locally and remotely on the AWS cloud through CDK. It also supports running agents in parallel.

Crew.ai allows for both synchronous and asynchronous execution of crew operations.

Observations:

1. All frameworks support running agents locally.
2. Some frameworks allow remote agent deployment to distributed (cloud) environments, typically through relatively simple approaches, with little configurations for execution environments.
3. To meet evolving business needs, I think the next phase for many frameworks will involve supporting agent deployment and execution in diverse environments with varying configurations and business needs. Borrowing a popular term, we need to support Agent-as-a-Service (AaaS).

Agent Monitoring, Analysis, and AgentOps

Not many of these open-source frameworks provide robust support for monitoring and metrics.

LangGraph offers monitoring capabilities through LangSmith and LangFuse.

Crew.ai, on the other hand, enables monitoring with built-in tools like agentOps, and external libraries like Langtrace, Langfuse, and others.

Observations:

1. Many frameworks offer limited logging and tracing capabilities, either through in-house libraries or third-party integrations.
2. A few provide more advanced logging, tracing, metrics support, and even analytics dashboards, for example LangGraph.
3. However, none of the frameworks currently offer full AgentOps support at runtime, which could be a key area for future improvements.

Agent Frameworks:

(Listed not by any specific order)

General Purpose:

1. LangGraph: https://github.com/langchain-ai/langgraph
2. LlamaIndex: https://docs.llamaindex.ai/en/stable/
3. OpenAI swarm: https://github.com/openai/swarm
4. Salesforce AgentLite: https://github.com/SalesforceAIResearch/AgentLite/
5. IBM bee agent: https://github.com/i-am-bee/bee-agent-framework
6. Microsoft tinytroupe: https://github.com/microsoft/tinytroupe
7. Microsoft AutoGen: https://github.com/microsoft/autogen
8. AWS Multi-Agent Orchestrator: https://github.com/awslabs/multi-agent-orchestrator
9. Hugging Face smolagents: https://github.com/huggingface/smolagents
10. Crewai: https://www.crewai.com/open-source
11. PydanticAI: https://ai.pydantic.dev/
12. Phidata: https://github.com/phidatahq/phidata
13. AutoGPT: https://github.com/Significant-Gravitas/AutoGPT

Domain specific:

1. MetaGPT: https://github.com/geekan/MetaGPT
2. OpenHands: https://github.com/All-Hands-AI/OpenHands
3. Pipecat: https://github.com/pipecat-ai
4. Eliza: https://github.com/elizaOS/eliza