Contract Testing for AI Microservices: Don't Let Your APIs Break in the Dark

Your recommendation engine service returned a valid JSON response. The schema is correct. The field names are right. But the values are hallucinated. Your consuming service accepted the contract violation, and now your customer is looking at a completely wrong recommendation.

Contract testing, the practice of validating that services meet each other's expectations, is standard in deterministic microservices architectures. But AI microservices break the contract testing paradigm. Traditional tests verify exact outputs. AI services output probabilities, embeddings, and variable text.

The question isn't "did you return the right string?" It's "did you return semantically valid output that respects our contract boundaries?" That's a different kind of testing entirely.

Why Traditional Contract Testing Fails for AI

Traditional Pact contracts look like this:

test('returns user recommendations', () => {
 expect(response).toEqual({
 recommendations: [
 { id: 1, title: 'Product A', score: 0.95 },
 { id: 2, title: 'Product B', score: 0.87 }
 ]
 });
});

This passes or fails. There's no middle ground. But AI APIs generate variable outputs. The same input produces different recommendations every time. Traditional contract testing creates false test failures and forces you to disable contracts entirely.

I've seen teams say "contracts don't work for AI" and drop them completely. Then they integrated breaking changes and didn't realize until production.

Contract testing for AI isn't about exact matches. It's about verifying that boundary contracts, data schemas, output formats, semantic requirements, are respected.

Semantic Contract Testing: The Framework

Semantic contracts validate that outputs meet requirements without requiring exact matches.

Layer 1: Schema Validation

First, verify the response structure is valid. Does it have the required fields? Are types correct? Are values within expected ranges?

const semanticContract = {
 // Consumer: downstream service (e.g., recommendation UI)
 consumer: 'recommendation-ui',

 // Provider: AI service (e.g., recommendation engine)
 provider: 'recommendation-engine',

 // Contract: what the consumer expects from the provider
 contract: {
 request: {
 userId: 'number',
 limit: { type: 'number', min: 1, max: 100 }
 },

 response: {
 // Schema validation: must be present and correctly typed
 recommendations: {
 type: 'array',
 minItems: 1,
 maxItems: 100,
 items: {
 type: 'object',
 required: ['id', 'title', 'score'],
 properties: {
 id: { type: 'number' },
 title: { type: 'string', minLength: 1 },
 score: {
 type: 'number',
 minimum: 0,
 maximum: 1
 }
 }
 }
 },

 // Metadata for debugging
 modelVersion: 'string',
 generatedAt: 'string' // ISO timestamp
 }
 }
};

This validates structure. Every response from your AI service must pass this schema, regardless of what recommendations are generated.

Layer 2: Semantic Validation

Beyond schema, validate semantic requirements. Are the recommendations coherent? Do the scores correlate with quality?

const validateSemanticContract = async (response, context) => {
 const contract = {
 // Recommendations must be diverse
 diversity: {
 validate: (recs) => {
 const embeddings = recs.map(r => embed(r.title));
 const distances = computePairwiseDistances(embeddings);
 const minDistance = Math.min(...distances);
 return minDistance > 0.3; // embedding distance threshold
 },
 error: 'Recommendations too similar'
 },

 // Score distribution must be reasonable
 scoreDistribution: {
 validate: (recs) => {
 const scores = recs.map(r => r.score);
 const variance = calculateVariance(scores);
 return variance > 0.01; // not all same score
 },
 error: 'Scores lack variation'
 },

 // Rankings must be monotonic (higher score = appears earlier)
 monotonicity: {
 validate: (recs) => {
 for (let i = 0; i < recs.length - 1; i++) {
 if (recs[i].score < recs[i + 1].score) {
 return false; // violated monotonicity
 }
 }
 return true;
 },
 error: 'Scores not properly ranked'
 },

 // No hallucinated IDs
 knownIds: {
 validate: async (recs) => {
 const validIds = await getValidProductIds();
 return recs.every(r => validIds.includes(r.id));
 },
 error: 'Contains unknown product IDs'
 }
 };

 const violations = [];

 for (const [rule, validator] of Object.entries(contract)) {
 const passes = await validator.validate(response.recommendations);
 if (!passes) {
 violations.push(validator.error);
 }
 }

 return {
 passes: violations.length === 0,
 violations
 };
};

Layer 3: Version Compatibility Testing

As you update your AI service, contracts ensure backward compatibility. Consumers need to know when breaking changes are coming.

// Pact/PactFlow example for AI services
describe('Recommendation Engine Contract', () => {
 const provider = new Pact({
 consumer: 'recommendation-ui',
 provider: 'recommendation-engine'
 });

 it('returns recommendations matching semantic contract', async () => {
 await provider.addInteraction({
 state: 'user has recommendation history',
 uponReceiving: 'a request for recommendations',
 withRequest: {
 method: 'POST',
 path: '/api/v1/recommend',
 body: { userId: 123, limit: 10 }
 },
 willRespondWith: {
 status: 200,
 // Instead of exact body match, use matcher that validates semantics
 body: {
 recommendations: Matchers.arrayContaining([
 {
 id: Matchers.number(),
 title: Matchers.string(),
 score: Matchers.numberBetween(0,1),
 // Custom matcher for semantic validation
 ...customMatcher(validateSemanticContract)
 }
 ]),
 modelVersion: Matchers.string(),
 generatedAt: Matchers.iso8601DateTime()
 }
 }
 });

 const response = await client.getRecommendations(123,10);
 expect(response).toMatchObject(provider.contract);
 });

 it('maintains backward compatibility across model versions', async () => {
 const v2Schema = { /* new schema */ };
 const v1Schema = { /* old schema */ };

 // Verify v2 output is compatible with v1 consumers
 expect(validateAgainstSchema(v2Response, v1Schema)).toBe(true);
 });
});

Consumer-Driven Contracts for AI APIs

Instead of the provider defining what's correct, consumers define their expectations. This shifts testing upstream.

How Consumer-Driven Contracts Work

Your recommendation UI team defines what they need from the recommendation engine. The engine team verifies they can deliver it. Contract violations are caught before they hit production.

// Consumer (recommendation-ui) defines expectations
const uiContractExpectations = {
 consumer: 'recommendation-ui',
 provider: 'recommendation-engine',
 version: '1.0.0',

 interactions: [
 {
 name: 'Get recommendations for active user',
 given: 'user has browse history',
 uponReceiving: 'request for top 5 recommendations',
 withRequest: {
 method: 'POST',
 path: '/api/recommend',
 body: { userId: 'any-user-id', limit: 5 }
 },
 willRespondWith: {
 status: 200,
 body: {
 recommendations: 'array-with-5-items',
 // Consumer cares about these properties
 items: {
 id: 'must-be-valid-product',
 title: 'must-be-non-empty-string',
 score: 'must-be-between-0-and-1',
 // Consumer doesn't care about exact values
 reasoning: 'any-string-is-ok' // optional explanation
 }
 }
 }
 }
 ]
};

// Provider (recommendation-engine) verifies contract
const verifyContract = async (provider, contract) => {
 const results = [];

 for (const interaction of contract.interactions) {
 const response = await provider.process(interaction.withRequest.body);

 const violation = checkSemanticContract(
 response,
 interaction.willRespondWith,
 interaction.uponReceiving
 );

 results.push({
 interaction: interaction.name,
 passes: !violation,
 violation
 });
 }

 return results;
};

Handling Non-Deterministic Outputs

AI services are non-deterministic. The same input produces different outputs. Your contracts must account for this.

Strategy 1: Repeated Validation

Run the same interaction multiple times. All responses must pass semantic validation.

const validateNonDeterministicContract = async (provider, contract) => {
 const trials = 10;
 const failures = [];

 for (let i = 0; i < trials; i++) {
 const response = await provider.process(contract.request);

 const semanticValidation = await validateSemanticContract(
 response,
 contract
 );

 if (!semanticValidation.passes) {
 failures.push({
 trial: i,
 violations: semanticValidation.violations
 });
 }
 }

 if (failures.length > 0) {
 console.error(`Contract failed ${failures.length}/${trials} times`);
 return {
 passes: false,
 failureRate: failures.length / trials,
 failures
 };
 }

 return { passes: true };
};

Strategy 2: Property-Based Testing

Use property-based testing to validate that contracts hold across a range of inputs, not just fixed examples.

const propertyBasedContractTest = async (provider) => {
 const properties = [
 {
 name: 'Recommendations always have valid IDs',
 property: (response) => {
 return response.recommendations.every(r =>
 validProductIds.includes(r.id)
 );
 }
 },
 {
 name: 'Scores are monotonically decreasing',
 property: (response) => {
 const scores = response.recommendations.map(r => r.score);
 for (let i = 0; i < scores.length - 1; i++) {
 if (scores[i] < scores[i + 1]) return false;
 }
 return true;
 }
 },
 {
 name: 'Scores are between 0 and 1',
 property: (response) => {
 return response.recommendations.every(r =>
 r.score >= 0 && r.score <= 1
 );
 }
 }
 ];

 const userIds = generateUserIds(100); // Generate test inputs

 for (const userId of userIds) {
 const response = await provider.recommend(userId);

 for (const prop of properties) {
 if (!prop.property(response)) {
 console.error(`Property failed: ${prop.name} for userId ${userId}`);
 return false;
 }
 }
 }

 return true;
};

Detecting Contract Violations in Production

Contracts should be validated at the integration boundary, not just in tests. Catch violations immediately when they occur.

const contractEnforcingMiddleware = (contractDefinition) => {
 return async (req, res, next) => {
 const originalSend = res.send;

 res.send = function(data) {
 const violation = checkSemanticContract(data, contractDefinition);

 if (!violation.passes) {
 // Log contract violation
 logger.error('Contract violation detected', {
 path: req.path,
 consumer: contractDefinition.consumer,
 provider: contractDefinition.provider,
 violations: violation.violations,
 response: data
 });

 // Metrics and alerting
 metrics.contractViolations.inc();

 // In strict mode, reject the response
 if (process.env.CONTRACT_MODE === 'strict') {
 return res.status(502).json({
 error: 'Service contract violation',
 details: violation.violations
 });
 }
 }

 return originalSend.call(this, data);
 };

 next();
 };
};

Real-World Contract Testing Workflow

mature teams implement contract testing for AI microservices: