Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe

Executive Summary

The future of AI agents hinges on their ability to interact robustly with the real world, and that increasingly means chaining calls to diverse external APIs. Amidst rising data-sovereignty regulations, public institutions are often compelled to deploy open-source, on-premise LLM agents. Yet, these models consistently falter when tasked with complex, multi-step API interactions, a critical gap for which no adequate benchmark previously existed.

This research, “Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe,” directly addresses this pressing challenge. It introduces KOPA-Bench, a pioneering benchmark built on live Korean public APIs, and EDGE, an innovative data-synthesis methodology. The core breakthrough? Enabling smaller, more efficient open-source LLMs to navigate intricate API sequences with a proficiency that rivals, and often surpasses, much larger proprietary models. This isn’t just an academic exercise; it’s a foundational step towards deployable, compliant, and highly capable AI agents for critical public services and enterprise operations.

Technical Deep Dive

The fundamental problem tackled here is the inherent complexity of multi-step tool-calling. It’s not enough for an LLM to understand a single API’s function; it must intelligently parse an initial user query, identify a sequence of tools, extract the necessary parameters for the first tool, execute it, then dynamically interpret its output to parameterize the next tool in the chain, and so on, until the overall goal is achieved. This process is fraught with potential pitfalls, from API schema mismatches to real-time data variances and execution failures.

To quantify and address this, the authors introduce the Korean Open Public API Benchmark (KOPA-Bench). Comprising 145 real-world tasks, KOPA-Bench differentiates itself by focusing on live, multi-step interactions with actual Korean government APIs. This pragmatic approach exposes the true performance limitations of existing models far more effectively than synthetic or single-step benchmarks. KOPA-Bench reveals that even sophisticated open-source LLMs struggle profoundly in this authentic, chained execution environment.

Enter EDGE (Execution-grounded Dynamic Graph for tool-calling data synthEsis). This methodology is a sophisticated approach to generating high-quality training data for multi-step tool-calling. Imagine a meticulous cartographer who doesn’t just draw roads based on blueprints, but sends scouts to verify every path and connection in real-time.

1. Graph Construction: EDGE first builds a preliminary graph, mapping potential output-to-input relationships between various tools based on their API schemas. This identifies which tool’s output could potentially serve as an input for another.
2. Live Execution Verification (The Key Innovation): This is where EDGE truly distinguishes itself. Instead of relying solely on theoretical links, EDGE actually executes these proposed tool calls against the live APIs. Only those connections that successfully execute and produce valid, usable outputs are retained in the graph. This “execution-grounded” approach weeds out dead ends, erroneous assumptions, and schema discrepancies that are common in real-world APIs, ensuring the synthesized data is robust and realistic.
3. Trajectory Synthesis: Once a verified graph of executable links is established, EDGE traverses these links to synthesize complete, multi-step trajectories. These trajectories represent successful, real-world task completions, providing a rich dataset of sequential tool usage.

This highly curated dataset is then used to fine-tune an LLM. Specifically, the research reports using GRPO (Gradient Regularized Policy Optimization), a technique common in reinforcement learning, to train models on these execution-grounded trajectories. The results are compelling: a 9B parameter model, fine-tuned with EDGE data, nearly matches the performance of an untuned 27B model from the same family. This substantial leap in efficiency and capability extends not only to KOPA-Bench but also to other benchmarks like BFCL, demonstrating the generalizability of the approach. This represents a significant advancement in Machine Learning for agentic systems.

Real-World Applications

The implications of this research are profound for the deployment of practical AI agents:

• Public Sector Automation: Imagine citizens using an LLM agent to navigate complex government services – applying for multiple permits, checking eligibility for various benefits, or resolving multi-departmental queries – all through a single interface that chains live public APIs. Data sovereignty is respected by using on-premise, open-source models.
• Enterprise Resource Orchestration: Within large organizations, AI agents could automate complex business processes by chaining internal APIs across different departments (e.g., pulling customer data from CRM, cross-referencing with ERP inventory, and initiating a support ticket in a separate system).
• Specialized Knowledge Retrieval: For domains requiring access to vast, interconnected databases and services, agents could perform sophisticated searches and data aggregations that require chaining multiple queries and filtering steps, presenting synthesized answers rather than raw API outputs.
• Autonomous IoT and Robotic Control: Beyond information retrieval, the principles of execution-grounded multi-step tool-calling could be extended to agents controlling physical systems, where verifying the successful execution of each step is critical for safety and reliability.

Future Outlook

Looking ahead 2-3 years, this research points to several exciting directions for intelligent systems:

• Generalized Execution-Grounding: The EDGE methodology, while demonstrated with Korean APIs, is inherently generalizable. We can expect similar techniques to emerge for diverse API ecosystems globally, pushing the reliability of AI agents across all sectors.
• Smaller, More Capable Models: The ability of smaller LLMs to achieve outsized performance on complex tasks through targeted data synthesis will democratize access to advanced agentic capabilities, making them viable for resource-constrained environments and specialized deployments. This is a critical step for on-premise, sovereign AI.
• Adaptive Agent Architectures: Future AI agents will likely incorporate dynamic data-synthesis and self-correction mechanisms directly into their learning loops, continuously improving their tool-use proficiency as they interact with live environments. This moves beyond static training datasets towards agents that learn continuously from their own experiences.
• Enhanced Auditability and Safety: By grounding tool-calling in verified execution, this research inherently improves the transparency and auditability of agentic actions. Knowing that a model was trained on actually executable sequences rather than theoretical ones provides a crucial layer of safety for critical applications.
• Hybrid AI Systems: The line between LLMs and traditional software engineering will blur further. Expect sophisticated frameworks that blend the generative power of LLMs with robust, execution-validated decision-making engines for higher reliability and predictability in complex real-world tasks.

Key Takeaways

• Multi-step tool-calling is a critical, yet underdeveloped, capability for practical LLM AI agents, especially in regulated, on-premise environments.
• KOPA-Bench is a pioneering benchmark addressing this gap, using real-world, live Korean public APIs to evaluate multi-step task completion.
• EDGE offers an innovative, execution-grounded data synthesis methodology that verifies tool connections against live APIs, producing highly reliable training data.
• This approach enables smaller LLMs (e.g., 9B parameters) to achieve performance comparable to much larger, untuned models (e.g., 27B parameters) on complex, real-world tasks.
• The research accelerates the deployment of efficient, compliant, and highly capable open-source AI agents for public services and enterprise applications, advancing the field of Machine Learning.