affinefoundation/navworld repository overview

⁠NAVWORLD

NAVWORLD is the Affine network's agent evaluation environment for end-to-end Chinese travel-planning tasks. The model under test runs as a tool-using agent that must autonomously decide which MCP tools to call, integrate their results, and produce a complete travel plan that is scored on factual grounding, completeness, tool usage and LLM-judged quality.

⁠What it tests

A single evaluation puts the model through a closed loop:

1. The environment generates a parameterised travel problem (city pair, dates, constraints, problem type — intercity / multi-day / hybrid / single-POI / food-tour / business / family-study).
2. The agent loop calls real MCP tools to gather data:
   • AMap: poi_search, around_search, direction, weather
   • Transport: search_flights, search_train_tickets
3. The model synthesises a final plan referencing the tool data.
4. The plan is graded along two axes:
   • Code (50 pts) — information consistency vs tool results, completeness, six hard constraints (format, required-tools-called, POI verification, transport grounding, tool quality, tool-info-used).
   • LLM judge (50 pts) — practicality, logic, user experience, analysis depth, factual grounding. Median across multiple Qwen judges, with DashScope fallback for stability.

Hard-constraint failures multiply the score down; invalid evaluations (judge unreachable, agent never produced a final plan, infra timeouts) come back with extra.error non-empty so the caller can filter them out before aggregation.

⁠Required environment variables

⁠Two ways to run

⁠1. af eval — the evaluation runner (recommended)

af eval is shipped by affine-cortex⁠ and is the high-level entry point for batched evaluation: it pulls this image, starts the container, dispatches sample tasks against an OpenAI-compatible endpoint, aggregates scores, and writes a results JSON.

pip install affine-cortex

Single sample, pinned task_id

af eval -e NAVWORLD \
    --task-id 22961303 \
    --model     <provider/model-name> \
    --base-url  https://your.openai-compat.endpoint/v1 \
    --temperature 0.0 \
    --output    results.json

Batch over N random samples

af eval -e NAVWORLD \
    --samples 20 \
    --model     <provider/model-name> \
    --base-url  https://your.openai-compat.endpoint/v1 \
    --temperature 0.0 \
    --max-retries 1 \
    --output    results.json

Sweep a contiguous task_id range (one sample per task_id; useful for reproducible regression sweeps):

af eval -e NAVWORLD \
    --task-id-range 1000 1050 \
    --model     <provider/model-name> \
    --base-url  https://your.openai-compat.endpoint/v1 \
    --temperature 0.0 \
    --delay 1.0 \
    --output    results.json

Add --network-host when your model endpoint lives on localhost (so the container can reach the host loopback).

Full flag reference: af eval --help.

⁠2. afs — the affinetes CLI (lower-level container management)

afs is shipped by affinetes and operates directly on the container — useful for one-off evaluate() calls, debugging, or scripted pipelines that already manage their own retries / batching:

pip install affinetes

# Pull and start the container
afs run affinefoundation/navworld:latest \
    --name navworld \
    --env CHUTES_API_KEY=$CHUTES_API_KEY \
    --env AMAP_MAPS_API_KEY=$AMAP_MAPS_API_KEY \
    --env DASHSCOPE_API_KEY=$DASHSCOPE_API_KEY

# Call evaluate() directly
afs call navworld evaluate \
    --arg task_id=22961303 \
    --arg model=<provider/model-name> \
    --arg base_url=https://your.openai-compat.endpoint/v1

af eval ultimately uses the same evaluate() entry point; afs just exposes it without the batching + aggregation layer.

⁠Container contract

The image starts a uvicorn HTTP server on port 8000. The Actor.evaluate() endpoint accepts task_id, seed, model, base_url, api_key, temperature, timeout and returns a result with score, success, and an extra payload containing the full conversation, tool_trace, scoring breakdown, and (when applicable) an error string for infra-side failures.