Task 4.1: Define success criteria and evaluation signals for agent tasks

• AThe number of session.idle events, one per model call

  Incorrect

  session.idle is emitted when the tool-use loop ends, not once per model call.

• BThe number of assistant.turn_start and turn_end pairs

  Correct

  Each pair is exactly one LLM API call, and there are no hidden calls for planning or completion checks.

• CThe number of tool calls, since each tool call is one model call

  Incorrect

  One turn can request several tools; turns, not tool calls, map to model calls.

• DThe count of session.task_complete events in the log, one per model call made during the task

  Incorrect

  task_complete is an optional signal that the model considers the task done.

In the Copilot agent loop, each iteration is exactly one LLM API call, visible as one assistant.turn_start and assistant.turn_end pair in the event log. There are no hidden calls for planning, evaluation or completion checks, so counting the pairs, for example with grep in the session's events.jsonl, gives the number of model calls a task needed.

• ARun each task once and compare the agents by the length of their final answers

  Incorrect

  A single run ignores nondeterminism, and answer length isn't one of the key metrics.

• BAsk each agent to grade its own output on a 1-to-10 scale

  Incorrect

  Self-grading isn't the method described; tasks are scored on resolution and efficiency metrics.

• CRun each task several times and compare resolution rate, token efficiency, latency and tool-call reliability

  Correct

  GitHub runs multiple independent runs to account for nondeterminism and tracks these key metrics.

• DCompare them on tasks copied from real customer conversations and private code, since those reflect daily use best

  Incorrect

  GitHub's benchmark tasks come from public repositories and synthetic scenarios, not real user queries or customer code.

GitHub evaluates its agents with industry benchmarks such as SWE-Bench and internal suites built from public open-source repositories and synthetic scenarios. Each evaluation includes multiple independent runs to account for nondeterminism, and key metrics are resolution rate (percentage of tasks completed), token efficiency, latency and tool call reliability.

• AA line in copilot-instructions.md telling Copilot to fix every alert

  Incorrect

  Instructions guide the agent; they don't stop a pull request from being merged.

• BA ruleset with the Require code scanning results rule for the target branch

  Correct

  The rule blocks merges when a required tool finds alerts of the defined severity, is still running, or isn't configured.

• CThe agent's built-in CodeQL validation, which blocks merging when it finds a problem

  Incorrect

  The built-in validation lets the agent try to fix issues before finishing; it isn't a merge gate.

• DA preToolUse hook that runs CodeQL before each edit

  Incorrect

  Hooks act on the agent's tool calls; the merge requirement belongs in a ruleset.

If code scanning is configured, a ruleset's Require code scanning results rule prevents merging when a required tool finds an alert of a severity you define, when its analysis is still in progress, or when a required tool isn't configured. GitHub's governance guidance suggests this rule for code produced by people and agents alike; Copilot cloud agent is subject to rulesets like any developer.

• AThe dashboard mainly reflects IDE activity, and Copilot CLI usage isn't included

  Correct

  Usage in Copilot CLI, code review, Chat on GitHub.com and GitHub Mobile is not in dashboard data.

• BThe dashboard only shows the last 24 hours

  Incorrect

  The dashboard shows 28 days of aggregated data.

• CCLI usage is shown only to enterprise owners

  Incorrect

  The data simply isn't included in the dashboard.

• DCLI usage appears in the dashboard only after a user also installs the VS Code extension

  Incorrect

  Installing an IDE extension doesn't add CLI usage to the dashboard.

The Copilot usage metrics dashboard shows 28 days of aggregated IDE telemetry for licensed users, supplemented by server-side telemetry. Usage in Copilot Chat on GitHub.com, GitHub Mobile, Copilot code review or Copilot CLI is not included in dashboard data, so evaluate those surfaces with other signals.

• AA per-run budget set in the copilot-setup-steps.yml file of the repository

  Incorrect

  Agentic workflows are capped in their own frontmatter.

• Bmax-ai-credits in the frontmatter, which defaults to 1,000 AIC per run

  Correct

  You can cap inference per run with max-ai-credits; the default cap is 1,000 AIC.

• CThe --max-ai-credits flag on gh aw compile

  Incorrect

  The cap is a frontmatter setting of the workflow.

• DNo cap; costs are reviewed after the run

  Incorrect

  max-ai-credits caps usage per run.

GitHub Agentic Workflows use AI Credits (AIC) to monitor and budget inference across engines, where 1 AIC is $0.01. You can set max-ai-credits in workflow frontmatter to cap inference for a single run; the default cap is 1,000 AIC per run.

• AAdd a feedback label to the pull request and describe the problem in a comment

  Incorrect

  Feedback is given with the thumbs buttons.

• BMention @github-feedback in a comment

  Incorrect

  Feedback is given with the thumbs buttons.

• CUse the thumbs up or thumbs down buttons, optionally with a reason

  Correct

  Feedback buttons on Copilot's pull requests and comments rate the output.

• DClose the pull request without merging

  Incorrect

  Closing gives no rating; use the feedback buttons.

On a pull request or comment from Copilot, click thumbs up or thumbs down; with thumbs down you can choose a reason and leave a comment before submitting. This feedback helps improve Copilot's quality.

• ARequire linear history

  Incorrect

  That rule controls merge commits, not coverage.

• BRestrict file size

  Incorrect

  That rule limits file sizes, not coverage.

• CBlock force pushes

  Incorrect

  That rule stops force pushes, not coverage drops.

• DRestrict code coverage, with a minimum percentage or a maximum drop

  Correct

  The rule blocks merges below a minimum line coverage or when coverage drops more than a set number of points.

With GitHub Code Quality and uploaded coverage data, the Restrict code coverage rule (public preview) blocks merging when line coverage on the pull request branch is below a minimum percentage, or drops by more than a set number of points relative to the default branch. It only evaluates coverage already uploaded, so make the upload checks required.