API Live Sync #4: OpenAI Fetcher

Part 4: Building the OpenAPI Retrieval System. You can find the repository here.

---

In our previous articles, we laid the foundation with architecture, data structures, and the core service layer. Now it's time to tackle one of the most challenging parts of live API synchronization: actually fetching those OpenAPI specifications from development servers.

The Challenge: Development Servers Are Chaotic

Imagine this: You're building a FastAPI service, and your OpenAPI spec is served at http://localhost:8000/openapi.json. Simple, right? Wrong! Here's what can go wrong:

• Server restarts mid-request (because you're actively developing)

• CORS issues (because browsers are security-conscious)

• Network timeouts (because your laptop decided to prioritize that YouTube video)

• Different frameworks serve specs at different endpoints

• Authentication requirements (because not all APIs are public)

We needed to build a fetcher that could handle all this chaos gracefully while providing helpful feedback to developers.

The Architecture

The Core Interface: Setting Expectations

export interface OpenAPIFetcher {
  fetchSpec(url: string, options?: FetchOptions): Promise<OpenAPIFetchResult>
  testConnection(url: string, options?: FetchOptions): Promise<ConnectionTestResult>
}

Why two methods? Because sometimes you want to test if someone's home before knocking on the door. testConnection uses a lightweight HEAD request to check if the server is alive, while fetchSpec does the heavy lifting of actually downloading the specification.

The Result Types: Comprehensive Feedback

export interface OpenAPIFetchResult {
  success: boolean
  content?: string // The actual OpenAPI spec (JSON or YAML)
  contentType?: string // Helps us know the format
  errors: string[] // Multiple errors can occur
  metadata: {
    url: string
    timestamp: Date
    responseTime: number
    statusCode?: number
  }
}

The Reasoning: Never leave developers guessing. If something fails, tell them exactly what went wrong, when it happened, and how long it took. It's like having a really good error message that also includes a timestamp and performance metrics.

The Implementation

Challenge 1: Timeout Handling

The Problem: The Fetch API doesn't have built-in timeout support. Your request could hang forever while you wait for a server that's stuck in an infinite loop.

Our Solution: Manual timeout with AbortController

async fetchSpec(url: string, options?: FetchOptions): Promise<OpenAPIFetchResult> {
  const controller = new AbortController()
  const timeout = options?.timeout || 10000

  // Set up our impatience timer
  const timeoutId = setTimeout(() => controller.abort(), timeout)

  try {
    const response = await fetch(url, {
      signal: controller.signal, // This is our "cancel button"
      // ... other options
    })

    clearTimeout(timeoutId) // Cancel the impatience timer
    // ... handle response
  } catch (error) {
    clearTimeout(timeoutId) // Always clean up!

    if (error.name === 'AbortError') {
      throw new Error(`Request timeout after ${timeout}ms`)
    }
    throw error
  }
}

Why AbortController? It's like having a remote control for your HTTP requests. If the server is taking too long, we can say "nope, I'm out" and move on with our lives.

Challenge 2: Retry Logic - Because Servers Have Bad Days

The Problem: Development servers restart frequently, network connections hiccup, and sometimes the first request just doesn't work for mysterious reasons.

Our Solution: Exponential backoff with smart retries

async fetchSpec(url: string, options?: FetchOptions): Promise<OpenAPIFetchResult> {
  const finalOptions = { timeout: 10000, retries: 2, ...options }
  let lastError: any

  for (let attempt = 0; attempt <= finalOptions.retries; attempt++) {
    try {
      const result = await this.attemptFetch(url, finalOptions, startTime)
      return result // Success! No need to retry
    } catch (error) {
      lastError = error

      // Don't retry on certain errors - they won't get better
      if (this.shouldNotRetry(error)) {
        break
      }

      // Wait before retry (exponential backoff)
      if (attempt < finalOptions.retries) {
        await this.delay(Math.pow(2, attempt) * 1000) // 1s, 2s, 4s...
      }
    }
  }

  // All retries failed
  return this.handleFetchError(url, startTime, lastError)
}

The Retry Philosophy:

• Timeout errors: Don't retry (server is too slow)

• CORS errors: Don't retry (configuration issue)

• 4xx errors: Don't retry (client error, won't change)

• 5xx errors: Retry (server error, might recover)

• Network errors: Retry (transient issues)

Exponential Backoff: Like being politely persistent. First retry after 1 second, then 2 seconds, then 4 seconds. It's the digital equivalent of knocking on the door, waiting a bit, then knocking again a bit louder.

Challenge 3: Framework Detection

The Problem: Different frameworks serve OpenAPI specs at different endpoints:

• FastAPI: /openapi.json

• Express: /api-docs.json

• Spring Boot: /v3/api-docs

• NestJS: /api-json

Our Solution: Pattern matching

const FRAMEWORK_PATTERNS = {
  fastapi: ['/openapi.json', '/docs/openapi.json'],
  express: ['/api-docs.json', '/swagger.json', '/api/docs'],
  nestjs: ['/api-json', '/api/docs-json'],
  spring: ['/v3/api-docs', '/v3/api-docs.json'],
  // ... more frameworks
}

export function detectFrameworkFromURL(config: URLSourceConfig): FrameworkDetectionResult {
  const url = config.url.toLowerCase()

  // Check URL patterns
  for (const [framework, patterns] of Object.entries(FRAMEWORK_PATTERNS)) {
    for (const pattern of patterns) {
      if (url.includes(pattern.toLowerCase())) {
        return {
          framework: framework as APIFramework,
          confidence: 0.9,
          indicators: [`URL matches ${framework} pattern: ${pattern}`]
        }
      }
    }
  }

  // Port-based heuristics (because developers are creatures of habit)
  if (url.includes(':8000')) {
    return {
      framework: 'fastapi',
      confidence: 0.6,
      indicators: ['URL suggests FastAPI (port 8000)']
    }
  }

  return { framework: 'unknown', confidence: 0, indicators: [] }
}

Why This Matters: When something goes wrong, we can provide framework-specific guidance:

export function getFrameworkErrorGuidance(framework: APIFramework, error: string): string[] {
  const frameworkSpecific = {
    fastapi: [
      'Make sure FastAPI is running on port 8000',
      'Check if /openapi.json endpoint is accessible',
      'Verify FastAPI version supports OpenAPI generation'
    ],
    express: [
      'Ensure swagger-jsdoc is properly configured',
      'Check that swagger middleware is set up correctly',
      'Verify JSDoc comments are properly formatted'
    ],
    // ... more frameworks
  }

  return [...commonGuidance, ...frameworkSpecific[framework]]
}

It's like having a multilingual guide who knows the local customs of each API framework.

Challenge 4: CORS Detection

The Problem: Browsers block cross-origin requests unless the server explicitly allows them. This is great for security, terrible for development.

Our Solution: HEAD request with CORS header detection

async testConnection(url: string, options?: FetchOptions): Promise<ConnectionTestResult> {
  try {
    // Use HEAD request - faster, less data
    const response = await fetch(url, {
      method: 'HEAD',
      headers: options?.headers,
      // ... other options
    })

    // Check CORS headers
    const corsEnabled = this.checkCORSHeaders(response)

    return {
      statusCode: response.status,
      isReachable: true,
      corsEnabled,
      errors: response.ok ? [] : [`HTTP ${response.status}: ${response.statusText}`],
      suggestions: this.generateSuggestions(url, response.status, corsEnabled)
    }
  } catch (error) {
    // Handle connection failures...
  }
}

private checkCORSHeaders(response: Response): boolean {
  const corsHeader = response.headers.get('access-control-allow-origin')
  return corsHeader !== null && (corsHeader === '*' || corsHeader.includes('localhost'))
}

Why HEAD Requests? They're like knocking on the door instead of barging in. You get all the response headers (including CORS info) without downloading the actual content. Perfect for testing connectivity.

The Testing Strategy

Testing HTTP clients is tricky because you're dealing with external dependencies. The approach:

1. Mock Everything, Test Behavior

describe('OpenAPIFetcherImpl', () => {
  const mockFetch = vi.fn()
  global.fetch = mockFetch

  beforeEach(() => {
    mockFetch.mockReset()
  })

  it('should retry on network failures', async () => {
    // First two calls fail, third succeeds
    mockFetch
      .mockRejectedValueOnce(new Error('Network error'))
      .mockRejectedValueOnce(new Error('Network error'))
      .mockResolvedValueOnce({
        ok: true,
        status: 200,
        headers: {
          get: vi.fn((key: string) => {
            if (key === 'content-type') return 'application/json'
            return null
          })
        },
        text: () => Promise.resolve('{"openapi": "3.0.0"}')
      })

    const result = await fetcher.fetchSpec('http://localhost:8000/openapi.json')

    expect(result.success).toBe(true)
    expect(mockFetch).toHaveBeenCalledTimes(3) // Original + 2 retries
  })
})

The Mock Philosophy: We're not testing the network; we're testing our logic. The mock lets us simulate every possible scenario: timeouts, network errors, HTTP errors, CORS issues, and successful responses.

2. Integration Tests for Real-World Scenarios

it('should complete a full FastAPI integration workflow', async () => {
  // Step 1: Register a FastAPI source
  const source = await service.registerSource({
    name: 'My FastAPI Service',
    type: 'url',
    config: { url: 'http://localhost:8000/openapi.json' },
    syncStrategy: 'replace-all'
  })

  // Step 2: Validate the source
  const validation = await service.validateSource(config, 'url')
  expect(validation.isValid).toBe(true)

  // Step 3: Perform sync
  const syncResult = await service.syncSource(source.id)
  expect(syncResult.success).toBe(true)

  // Step 4: Verify framework guidance
  const guidance = service.getFrameworkGuidance(source.id, 'Connection failed')
  expect(guidance.some(g => g.includes('FastAPI'))).toBe(true)
})

Integration Test Philosophy: Test the entire workflow from registration to sync, ensuring all components work together seamlessly.

The Debugging Adventures

The Great Response Mock Mystery

The Problem: Our tests were failing with Cannot read properties of undefined (reading 'get').

The Investigation: The existing tests were mocking fetch responses like this:

mockFetch.mockResolvedValueOnce({ ok: true, status: 200 })

But our new fetcher expected a proper Response object with a headers.get() method:

const contentType = response.headers.get('content-type')

The Solution: Proper Response mocking

mockFetch.mockResolvedValueOnce({
  ok: true,
  status: 200,
  headers: {
    get: vi.fn((key: string) => {
      if (key === 'content-type') return 'application/json'
      if (key === 'access-control-allow-origin') return '*'
      return null
    })
  },
  text: () => Promise.resolve('{"openapi": "3.0.0"}')
})

The Lesson: When you upgrade your HTTP client, you need to upgrade your mocks too. It's like upgrading your car and then realizing your garage door opener doesn't work with the new model.

The Retry Logic Paradox

The Problem: Our retry test was failing because the retry logic wasn't working.

The Investigation: The issue was subtle. In our attemptFetch method, we were returning error results instead of throwing errors:

// This doesn't trigger retries (returns error result)
if (!response.ok) {
  return this.createErrorResult(url, startTime, [`HTTP ${response.status}`])
}

// This triggers retries (throws error)
if (!response.ok) {
  const error = new Error(`HTTP ${response.status}: ${response.statusText}`)
  throw error
}

The Lesson: Retry logic only works when exceptions are thrown, not when error results are returned. It's the difference between saying "I failed" and actually failing.

The Integration Challenge

The final challenge was integrating our shiny new fetcher with the existing LiveSpecSourceService. This required:

1. Updating the Service Constructor

export class LiveSpecSourceServiceImpl implements LiveSpecSourceService {
  private sources: Map<string, LiveSpecSource> = new Map()
  private storage: LiveSpecStorage
  private eventEmitter: EventEmitter = new SimpleEventEmitter()
  private fetcher: OpenAPIFetcher = new OpenAPIFetcherImpl() // Our new addition!

  constructor(storage?: LiveSpecStorage) {
    this.storage = storage || new LocalStorageLiveSpecStorage()
    this.loadSourcesFromStorage()
  }
}

2. Enhancing the Sync Method

private async syncFromURL(config: URLSourceConfig): Promise<SyncResult> {
  try {
    const fetchOptions = {
      timeout: config.timeout || 10000,
      headers: config.headers || {},
      retries: 2,
      followRedirects: true
    }

    const result = await this.fetcher.fetchSpec(config.url, fetchOptions)

    if (result.success && result.content) {
      return {
        success: true,
        hasChanges: true,
        changesSummary: ['Specification updated from URL'],
        errors: [],
        specVersion: generateContentHash(result.content),
        timestamp: result.metadata.timestamp
      }
    } else {
      return {
        success: false,
        hasChanges: false,
        changesSummary: [],
        errors: result.errors,
        timestamp: result.metadata.timestamp
      }
    }
  } catch (error) {
    throw new Error(`Failed to sync from URL: ${error.message}`)
  }
}

3. Upgrading Validation

private async validateURLConnection(config: URLSourceConfig): Promise<SourceValidationResult> {
  try {
    const testResult = await this.fetcher.testConnection(config.url, {
      timeout: config.timeout || 5000,
      headers: config.headers || {}
    })

    const errors: string[] = []
    const warnings: string[] = []

    // Check both reachability AND errors (for HTTP 404, etc.)
    if (!testResult.isReachable || testResult.errors.length > 0) {
      errors.push(...testResult.errors)
    }

    if (testResult.isReachable && !testResult.corsEnabled) {
      warnings.push('CORS may not be enabled - this could cause issues in browser environments')
    }

    // Add framework-specific suggestions as warnings
    if (testResult.suggestions.length > 0) {
      warnings.push(...testResult.suggestions)
    }

    return {
      isValid: errors.length === 0,
      errors,
      warnings
    }
  } catch (error) {
    return {
      isValid: false,
      errors: [`Connection test failed: ${error.message}`],
      warnings: []
    }
  }
}

The Reasoning: The service layer shouldn't know about HTTP details. It just asks the fetcher to get the spec and handles the business logic of what to do with the result.

What's Next?

With our OpenAPI fetcher in place, we're ready for the next challenge: file system watching. Because sometimes your OpenAPI spec isn't served by a development server—sometimes it's just a file that gets regenerated every time you change your code.

---

Still here? Cool. Look out for our next publication regarding file system watching!