Agent Harness in Mircosoft Agent Framework

https://www.pexels.com/photo/close-up-of-man-carrying-ropes-for-rock-climbing-12585932/

Agent Harness in Microsoft Agent Framework provides a preconfigured runtime for turning a language model into an agent capable of sustained, multi-step work. A language model can generate a response or request a tool call, but an effective agent also needs to preserve state, manage context, track progress, govern tool use, and determine how execution should continue. Agent Harness supplies this operational layer so developers do not have to assemble the entire agent loop themselves.

During a task, the Harness coordinates model and tool calls while maintaining conversation and session state. Plans and todos keep the agent oriented, context compaction controls token growth, approval policies govern tool use, and telemetry exposes what happened during execution. These capabilities help the agent retain continuity when a task cannot be completed in one response.

The Harness is not a separate runtime. It assembles existing Agent Framework components, including chat clients, pipelines, sessions, context providers, middleware, and tools. The result remains a standard Agent Framework agent with the same extension points as one assembled manually. Developers gain practical defaults while retaining the ability to disable, replace, or extend individual capabilities.

Agent Harness is not a model host, deployment service, or prescribed user interface. The application still chooses a model provider, controls where the agent runs, and presents progress or approval requests to the user.

How It Is Composed

The Harness combines five layers:

A chat client connects the agent to a language model.
A chat pipeline handles function calls, message injection, history persistence, and optional context compaction.
Context providers supply session instructions, memory, todo state, operating modes, and tools.
Middleware and decorators add approval handling, OpenTelemetry observability, and optional bounded looping.
The application streams responses and provides the user experience for progress updates and tool approvals.

Each layer remains customizable. Teams can begin with the provided defaults, disable capabilities they do not need, and replace individual providers without redesigning the whole agent loop.

Core Capabilities

Agent Harness includes several capabilities intended to keep multi-step tasks organized and observable:

| Capability          | Behavior                                                       |
| ------------------- | -------------------------------------------------------------- |
| Function invocation | Runs tools with a configurable iteration limit                 |
| History persistence | Saves history after each model call in a tool-calling run      |
| Todo tracking       | Tracks task progress by default                                |
| Agent modes         | Provides plan and execute modes by default                     |
| Session file memory | Provides session-scoped file memory by default                 |
| Tool approval       | Supports standing approvals and automatic approval rules       |
| Observability       | Emits OpenTelemetry data by default                            |
| Web search          | Adds search when the selected chat client supports it          |
| Context compaction  | Reduces context when token limits or a strategy are configured |

File access, background-agent delegation, shell execution, and bounded looping are optional. In Python, Agent Skills are also opt-in through a skills provider or configured skill paths.

How This Project Uses Agent Harness

This project demonstrates Agent Harness through a command-line health information assistant. Its top-level coordinator is created with create_harness_agent. An Azure AI Foundry deployment provides the model through FoundryChatClient, authenticated by DefaultAzureCredential. A Lagom dependency-injection container supplies the chat client, logger, session store, health specialist, and medical-center lookup tool.

The coordinator receives application instructions rendered from a Jinja template. These instructions restrict its scope, treat user input as untrusted, and prevent it from answering specialist questions from its own model knowledge. The Harness also receives an InMemoryHistoryProvider and two tools:

- health_advisor is a specialist Agent Framework agent exposed through as_tool. Its prompt requires accessible general health information, explicit uncertainty, appropriate professional follow-up, and immediate emergency guidance for potentially life-threatening symptoms.

- medical_centers is a FunctionTool backed by a deterministic Python function. It maps supported transplant procedures to a medical center and returns a fallback when no center is configured.

The central construction in agent_harness.py is:

agent = create_harness_agent(
   client=self._client.get_client(),
   agent_instructions=get_system_prompt(),
   context_providers=[InMemoryHistoryProvider(load_messages=False)],
   max_context_window_tokens=128_000,
   max_output_tokens=8_000,
   middleware=[
      ToolTimingMiddleware(
         statistics,
         agent_tool_names={"health_advisor"},
      )
   ],
   tools=[
      health_advisor.create_agent(statistics).as_tool(
         name="health_advisor",
         description="Answer a medical or health question.",
      ),
      FunctionTool(
         name="medical_centers",
         description="Find medical center where procedure can be performed.",
         func=medical_centers.run,
      ),
   ],
)

Request Flow

For each non-empty question, the application follows this sequence:

Start a new TurnStatistics measurement.
Run the coordinator with the current AgentSession.
Let the coordinator respond directly when appropriate or invoke a registered tool.
Record the duration and type of every tool invocation, even when a tool raises an exception.
Capture token usage reported by the delegated health agent.
Log the coordinator response metadata, delegated usage, and combined toke total before displaying the answer.
Persist the session when the user exits or when execution terminates with an error.

A shared TurnStatistics object connects the coordinator and specialist agent. ToolTimingMiddleware wraps coordinator tool calls and classifies each one as either an agent delegation or a function invocation. AgentUsageMiddleware, installed when the health advisor creates its agent, records usage from both regular and streamed responses. The resulting statistics include work performed by the specialist rather than reporting only the coordinator's model usage.

Durable Sessions

Conversation state is stored as a serialized AgentSession. At startup, the application restores a valid session or creates a new one. Invalid JSON and incompatible session data are moved to a uniquely named quarantine file, so a damaged session does not prevent the application from starting.

Session writes use a temporary file followed by os.replace. This atomic replacement avoids leaving a partially written primary session file if the process is interrupted during persistence. The outer finally block in the command loop ensures that the latest session is persisted during both normal exit and error handling.

The result illustrates the boundary between framework and application. Agent Harness provides the coordinator foundation, while the application owns its domain prompts, tools, specialist agents, safety rules, persistence, dependency injection, and operational telemetry.

Evaluation

Evaluation should cover both software correctness and the quality of the agent's decisions. This project already uses deterministic tests to verify prompt rendering, tool registration, middleware behavior, session recovery, usage aggregation, and medical-center lookup results. These tests confirm that the application is assembled correctly and that its supporting services behave predictably, but they do not establish whether model-generated answers are useful or safe.

A behavioral evaluation set should therefore include representative prompts for each supported route, along with adversarial and out-of-scope cases. Health questions should be assessed for correct delegation, factual grounding, appropriate uncertainty, and escalation of emergency symptoms. Procedure queries should select the deterministic medical-center tool and preserve its result. Greetings should receive a brief direct response, while unsupported requests and prompt-injection attempts should be declined without invoking an unrelated tool.

Each case can record pass or fail criteria for routing, tool arguments, final answer content, and safety behavior. Because model outputs vary, repeated runs are useful for measuring consistency rather than relying on one successful response. Human review remains important for clinical clarity and potential harm, even when automated graders are used for broader regression testing.

The existing telemetry provides a complementary operational view. Turn duration, tool-call order, delegated-agent usage, finish reason, and combined token consumption can be compared across model, prompt, and configuration changes. Together, behavioral scores and operational measurements make it possible to detect a change that improves answer quality while increasing latency or cost, or one that is faster but routes requests less reliably.

Demo

We have a small demo available that illustrates how the Agent Harness coordinates between tools and sub agents to manage complex tasks effectively.

code is in https://github.com/dennisseah/radio_gaga

Dennis Seah

Search This Blog