Phase 3: Pilot Implementation - Building Use Cases That Earn the Right to Scale

Picture a pilot that reaches Phase 3 in good shape. The foundation work from Phase 2 is solid: SSO is live, the API gateway is routing, cost attribution is working, and the smoke test has passed. The development team is ready to build.

Weeks later, the pilot fails the gate. Not because the technology did not work. The demos are impressive. Users like the output. But the security review finds a prompt injection vector in the document store MCP server, a vector that was not caught because the security review happened after the build was finished rather than being embedded in the build process. Cost has run well over the projected budget because nobody set acceptance thresholds until the end, which means the team kept iterating on quality when the outputs were already good enough. And when SMEs sit down to formally validate the use case outputs, there are no golden test sets to validate against. The team built use cases without defining what good looked like in a form the business could sign against.

Three gate conditions fail. The rework takes weeks. A pilot that should have advanced to hardening in month three advances in month five instead, at the point where the executive sponsor's patience has noticeably thinned.

This is the third deep-dive in the series on the enterprise Claude deployment roadmap. Phase 1 covered discovery and assessment. Phase 2 covered the infrastructure layer. Phase 3 is where use cases get built, MCP servers connect Claude to internal systems, and evaluation frameworks determine whether the pilot has earned the right to move to hardening.

Phase 3 runs weeks five through eight, fifteen to twenty working days, across fourteen tasks. The tasks are not independent. The sequencing is load-bearing. Build before security review, and the security review becomes rework. Define evaluation criteria after the build, and the build does not know what it is optimizing for.

The fourteen tasks organize into four functional groups.

3.1: Prompt Engineering and System Design

Before any code is written, the system prompts for each pilot use case get designed, versioned, and stored.

This is Task 3.1, and it sets the frame for everything that follows. Each prompt design includes the system prompt itself, input and output schemas using structured outputs via output_config.format, and the evaluation criteria the use case will be measured against at the gate. Version control goes into the prompt registry established in Phase 2. The prompt is not an artifact of the build process. It is the specification the build process implements.

The failure mode is treating prompt engineering as something that happens informally during development, adjusted incrementally as the build progresses, with the final version being whatever was in place when the demos went well. That process produces prompts that work for the use cases the developer thought to test. It does not produce prompts with documented evaluation criteria, which means the gate has no formal specification to sign against.

3.2: Document Store MCP Server

The MCP server for the primary document repository, SharePoint, Google Drive, or Confluence depending on the organization. Three tools at minimum: search, get_document, and list_recent. Authentication via service account. Read-only access enforced at the server level, not through prompt instructions. Pydantic input schemas on all parameters. Pagination on any tool that returns a list.

3.3: Ticketing System MCP Server

The MCP server for JIRA, Linear, or ServiceNow. Five tools: search_tickets, get_ticket, create_ticket, update_ticket, add_comment. The write tools, create_ticket, update_ticket, and add_comment, get marked with destructiveHint: true in the tool definition. Input validation on every field that gets written to the ticketing system, not just the fields the developer expects users to fill in.

3.4: Internal Database MCP Server

A read-only query server. Parameterized queries only: no raw SQL pass-through, ever. Result size limits at 100 rows per call by default. Connection pooling. Query timeout enforcement. The technical constraints here are not preferences. Raw SQL pass-through is a command injection surface. Unbounded result sets are a data exfiltration vector.

3.5: MCP Security Review

This task depends on Tasks 3.2, 3.3, and 3.4 being complete, which means it runs after the servers are built. It does not run after the use cases are built. The distinction matters.

The review covers four threat categories: prompt injection via tool results, command injection in tool parameters, data exfiltration via tool chaining, and permission escalation. Industry data on this is stark: a large share of MCP implementations contain command injection vulnerabilities at the point of first security review. The question is not whether a vulnerability exists. The question is whether it gets found before production.

Tool sandboxing per MCP 2.4 gets implemented based on the review findings. All tool invocations get added to the audit log. The security review is a joint task: the delivery partner owns the methodology and the findings, the client owns the organizational security sign-off.

3.6: Use Case 1 Build

The first pilot use case gets built end to end: prompt integration with the approved system prompt from Task 3.1, MCP server integration, error handling and retries on tool calls, structured output parsing, and an evaluation harness that runs against real data. Anonymized real data, where the use case touches sensitive information. Not synthetic data. Synthetic data does not surface the edge cases that matter.

3.7: Use Case 2 Build

Task 3.7 introduces a decision point: if the second use case is non-real-time, it should be configured to use the Batch API. Batch workloads at current pricing represent a meaningful cost reduction relative to synchronous API calls for the same token volume. This is a routing configuration decision, not a code change, but it needs to be made at build time, not during Phase 4 when cost optimization becomes the primary concern.

3.8: Use Case 3 Build

Task 3.8 commonly produces either a Claude Code deployment for an engineering business unit or a Claude.ai workflow for an operations business unit. Both paths are valid. The goal is to ensure the pilot spans at least two business unit types, which provides breadth evidence at the gate.

3.9: Claude Code Pilot Rollout

Five to ten pilot engineers. Claude Code is deployed with managed-settings.json enforcing the organization's API endpoint, project-level CLAUDE.md files pre-configured for each team's context, and MCP server registrations already in place. A one-hour onboarding session covers the tool, the organizational policies from the AUP, and how to report issues. Daily check-ins run for the first week.

The check-in cadence is not optional. The first week of a Claude Code rollout surfaces configuration issues, prompt questions, and user behavior patterns that cannot be predicted in advance. The team that skips daily check-ins in the first week discovers those issues in week three, when they have compounded.

3.10: Evaluation Framework

Before the pilot user feedback period begins, the evaluation framework has to be in place. Metrics per use case: quality, measured as accuracy and completeness against golden test sets; latency, measured at p50 and p95; cost per request; and user satisfaction. Baselines come from the manual process being replaced, not from the first week of pilot output. Acceptance thresholds, the numbers that must be met for the use case to pass the gate, get documented and signed off before the pilot period begins.

If the thresholds are set after the pilot period, they will be set to match the pilot output rather than to reflect the business requirement. That is not evaluation. That is documentation.

3.11: Pilot User Feedback Collection

The formal two-week pilot period. Structured feedback across three channels: daily usage logs capturing sessions, completion rates, and failure patterns; weekly surveys covering quality perception, time savings, and friction points; end-of-pilot interviews with a sample of users. The output is a dataset, not anecdotes. The gate condition requires NPS above 8 from pilot users. 'The users liked it' is not sufficient. A named score with a documented methodology is.

3.12: MCP Server Integration Testing

Every MCP server gets tested against MCP Inspector before it is connected to a use case build. The test covers tool schema accuracy against implementation, pagination behavior at boundary conditions (the last page, empty results, and a result set of exactly the page size limit), error response quality, authentication failure handling, concurrent request behavior, and timeout enforcement. These are not developer unit tests. They are integration tests against the actual MCP specification.

3.13: Use Case Acceptance Testing

Client subject matter experts validate output quality against the golden test sets established in Task 3.10. The edge case battery covers malformed input, empty results from tool calls, oversized documents that approach context limits, multilingual content if applicable, and rate limit behavior. Structured outputs get validated end to end through downstream parsing, not just inspected for visual correctness.

3.14: Prompt Regression Test Suite

An automated evaluation harness for each use case, with a minimum of 50 test cases per prompt. Coverage spans expected outputs on standard inputs, edge cases identified during Task 3.13, adversarial inputs designed to test robustness, and prompt injection attempts. The suite runs on every prompt change. Results are stored for trend analysis. A prompt change that degrades performance on 10 percent of test cases while improving the demo case is a regression, not an improvement.

The Gate

Phase 3 has five gate conditions. All five are required before Phase 4 begins.

All three pilot use cases must meet the acceptance criteria documented in Task 3.10. Pilot users must confirm value at an NPS above 8. The MCP security review from Task 3.5 must be passed with documented remediation of all findings. No data incidents can have occurred during the pilot period. Cost must be within 20 percent of the projected budget.

The cost condition is the one that most often fails without warning. Teams track quality and user satisfaction actively. They track cost less consistently. A use case that is performing well on quality and satisfaction but running at 150 percent of projected cost has not earned the right to scale. Scaling it multiplies the budget problem.

What Gets Deferred and Why

Three deferrals are common in Phase 3.

The MCP security review gets deferred to the end of the build because teams want to show working use cases before going through review. The consequence is that security findings require code changes to the already-built use cases, not just to the MCP servers. The delta in rework cost between reviewing servers before use case integration and reviewing after is not small.

Evaluation criteria get defined after the pilot period rather than before. Teams rationalize this as wanting to see real output before deciding what good looks like. The actual effect is that the criteria get set to match the output rather than the business requirement. The gate cannot distinguish a use case that achieved the target from a use case that had the target set to match what it achieved.

The prompt regression suite gets scoped down or deferred because 50 test cases per prompt is more effort than the team expected. The consequence surfaces in Phase 4 and beyond: prompts get updated to fix observed issues and introduce new ones, with no automated detection. A regression that ships to 200 enterprise users in Phase 5 is not a prompt problem. It is a testing gap.

Phase 4, Hardening and Compliance, opens next. It is where the pilot that passed the Phase 3 gate gets prepared for the organizational load of a full rollout: penetration testing, load validation, formal compliance review, and the security documentation that procurement and legal need to sign against. Next in the series: why hardening is not a single review but a structured sequence, and what it costs to defer it.