API Live Sync #6: Sync Engine

How we built the sync engine and intuitive UI that makes the API live sync come together

---

Previously in Our Live Sync ...

In our previous posts, we laid the groundwork for live API synchronization by building file watching and collections, and establishing the foundational architecture. We had all these individual components, but no engine to bring them together.

Today, we're diving into the heart of the live sync experience: the Sync Engine Service, the Setup View, and the Status Indicators. These are the components that transform our technical foundation into something developers actually want to use. While writing this post, the analogy of an orchestra conductor kept coming to my mind. I’m gonna use this analogy throughout this post.

🔗 Follow along: All code is available in the this repository.

The Challenge

Building developer tools is like being a translator between the messy reality of software development and the clean abstractions users want to work with. We had built all the pieces:

• File watchers that could detect changes

• Collections that understood frameworks

• Diff engines that could compare specifications

• Framework detection that could identify your stack

But we needed to answer three critical questions:

1. How do we orchestrate all these pieces? (Sync Engine)

2. How do users set this up? (Setup Wizard)

3. How do users know what's happening in real-time? (Status Indicators)

Let's dive into how we solved each of these challenges.

---

The Sync Engine Service

The Problem: Too Many Moving Parts

Imagine trying to conduct an orchestra where the violins don't know when the drums are playing, the brass section is in a different time zone, and the conductor has to manually tell each musician when to start. That's what our live sync system looked like before the Sync Engine Service.

We had:

• File watchers detecting changes

• Diff engines comparing specifications

• Collections stores managing data

• Framework detection identifying stacks

• But no central coordinator to make them work together

The Solution

The Sync Engine Service became our conductor - a single service that coordinates all the moving parts and makes intelligent decisions about when and how to sync.

1. The Architecture: Bringing Order to Chaos

export class SyncEngineService {
  private fileWatcher: FileWatcherImpl
  private diffEngine: SpecDiffEngine
  private config: SyncEngineConfig
  private activeSyncs = new Map<string, Promise<SyncResult>>()

  constructor(config: SyncEngineConfig = {}) {
    this.config = {
      autoSync: true,
      conflictResolution: 'prompt',
      debounceMs: 500,
      maxRetries: 3,
      ...config
    }

    this.fileWatcher = new FileWatcherImpl()
    this.diffEngine = new SpecDiffEngine({
      detectBreakingChanges: true,
      preserveUserCustomizations: true
    })

    this.setupFileWatcherEvents()
  }
}

Think of this as the brain of our live sync system. It knows about all the other components and can coordinate their actions. The activeSyncs Map is particularly useful - it prevents the same source from being synced multiple times simultaneously (like preventing two conductors from trying to lead the same orchestra).

2. Smart Sync Orchestration: The Magic Happens Here

The real magic happens in the startWatching method, which is like teaching the orchestra a new piece:

async startWatching(sourceConfig: {
  sourceId: string
  type: 'url' | 'file'
  path: string
  collectionName?: string
  framework?: string
}): Promise<SyncResult> {
  try {
    // Step 1: Fetch initial spec (like getting the sheet music)
    const initialSpec = await this.fetchSpec(sourceConfig.type, sourceConfig.path)
    if (!initialSpec) {
      return {
        success: false,
        errors: [`Failed to fetch initial spec from ${sourceConfig.path}`]
      }
    }

    // Step 2: Detect framework if not provided (like identifying the musical style)
    let framework = sourceConfig.framework
    if (!framework) {
      const detection = await this.detectFramework(sourceConfig.type, sourceConfig.path)
      framework = detection.frameworks[0]?.name || 'unknown'
    }

    // Step 3: Create or update collection (like arranging the music for our orchestra)
    const existingCollection = findCollectionBySourceId(sourceConfig.sourceId)
    if (existingCollection) {
      return await this.syncExistingCollection(existingCollection, initialSpec)
    } else {
      return await this.createNewCollection(sourceConfig, initialSpec, framework)
    }
  } catch (error) {
    return {
      success: false,
      errors: [`Sync engine error: ${error instanceof Error ? error.message : 'Unknown error'}`]
    }
  }
}

This method is like a master chef's recipe - it follows a precise sequence of steps, handles errors gracefully, and produces consistent results.

3. Conflict Prevention: The Diplomatic Solution

One of the trickiest parts was preventing concurrent syncs for the same source. Imagine if two people tried to update the same document simultaneously - chaos! Our solution:

async triggerSync(sourceId: string): Promise<SyncResult> {
  // Prevent concurrent syncs for the same source
  if (this.activeSyncs.has(sourceId)) {
    return await this.activeSyncs.get(sourceId)!
  }

  const syncPromise = this.performSync(sourceId)
  this.activeSyncs.set(sourceId, syncPromise)

  try {
    const result = await syncPromise
    return result
  } finally {
    this.activeSyncs.delete(sourceId)
  }
}

This is like having a bouncer at a club who ensures only one person can use the bathroom at a time. The second person doesn't get rejected - they just wait for the first person to finish.

4. Event-Driven Architecture

The sync engine listens to file watcher events and responds appropriately:

private setupFileWatcherEvents(): void {
  this.fileWatcher.on('fileChanged', async (event) => {
    if (!this.config.autoSync) return

    // Find collection associated with this file
    const collection = this.findCollectionByFilePath(event.filePath)
    if (!collection?.liveMetadata?.sourceId) return

    // Trigger sync with debouncing (like waiting for the dust to settle)
    setTimeout(() => {
      this.triggerSync(collection.liveMetadata!.sourceId)
    }, this.config.debounceMs)
  })
}

The debouncing is important here. Build tools often write files in chunks, so we wait 500ms for the file to "settle" before triggering a sync.

Testing: Because Orchestras Need Rehearsals

Testing a sync engine is like testing a conductor - you need to verify they can coordinate multiple musicians correctly:

describe('SyncEngineService', () => {
  it('should prevent concurrent syncs for the same source', async () => {
    const collection = {
      liveMetadata: { sourceId: 'test-source', sourceUrl: 'http://localhost:3000/openapi.json' }
    }

    mockCollections.findCollectionBySourceId.mockReturnValue(collection)
    vi.mocked(fetch).mockResolvedValue({
      ok: true,
      json: () => Promise.resolve({ info: { title: 'Test' }, paths: {} })
    } as Response)

    // Start two concurrent syncs
    const sync1 = syncEngine.triggerSync('test-source')
    const sync2 = syncEngine.triggerSync('test-source')

    const [result1, result2] = await Promise.all([sync1, sync2])

    // Both should succeed and be the same result (second one waits for first)
    expect(result1.success).toBe(true)
    expect(result2.success).toBe(true)
  })
})

This test ensures our "bouncer" logic works correctly - the second sync waits for the first one to complete rather than causing conflicts.

---

The Setup UI - Making Complexity Simple

The Problem: Setup Shouldn't Require a Manual

Setting up live sync involves several complex steps:

1. Choose between URL and file sources

2. Configure connection details

3. Detect the framework

4. Preview what will be generated

5. Configure sync options

Without a wizard, users would need to understand our entire architecture just to get started. It's like asking someone to assemble IKEA furniture without the instruction manual.

The Solution

We built a 4-step wizard that guides users through the setup process like a friendly tour guide:

Step 1: Source Type Selection

<template>
  <div class="source-type-options">
    <div
      v-for="option in sourceTypeOptions"
      :key="option.type"
      class="source-option"
      :class="{ 'selected': formData.sourceType === option.type }"
      @click="selectSourceType(option.type)"
    >
      <div class="flex items-start space-x-3">
        <component :is="option.icon" class="svg-icons text-accent mt-1" />
        <div class="flex-1">
          <h5 class="font-medium text-primaryLight">{{ option.title }}</h5>
          <p class="text-sm text-secondaryLight mt-1">{{ option.description }}</p>
          <div v-if="option.examples" class="text-xs text-secondaryDark mt-2">
            <span class="font-medium">{{ t('collections.examples') }}:</span>
            {{ option.examples.join(', ') }}
          </div>
        </div>
      </div>
    </div>
  </div>
</template>

This step is like asking "Are you connecting to a running development server, or watching a generated file?" We provide clear examples for each option because developers think in concrete terms, not abstractions.

Step 2: Configuration

For URL sources, we provide real-time validation and connection testing:

const testConnection = async () => {
  if (!formData.value.sourceUrl || urlValidation.value.status === 'error') return

  urlValidation.value.testing = true

  try {
    const response = await fetch(formData.value.sourceUrl, { method: 'HEAD' })
    if (response.ok) {
      urlValidation.value = {
        status: 'success',
        message: t('collections.connection_successful'),
        testing: false
      }
    } else {
      urlValidation.value = {
        status: 'error',
        message: t('collections.connection_failed', { status: response.status }),
        testing: false
      }
    }
  } catch (error) {
    urlValidation.value = {
      status: 'error',
      message: t('collections.connection_error'),
      testing: false
    }
  }
}

We don't just accept the URL - we actually try to connect to it and give immediate feedback.

We also provide helpful suggestions for common endpoints:

const commonEndpoints = computed(() => [
  '/openapi.json',
  '/api-docs', 
  '/v3/api-docs',
  '/docs/openapi.json',
  '/swagger.json'
])

const suggestEndpoint = (endpoint: string) => {
  const baseUrl = formData.value.sourceUrl.replace(/\/[^\/]*$/, '')
  formData.value.sourceUrl = baseUrl + endpoint
  validateUrl()
  testConnection()
}

It's like having autocomplete for API endpoints - we know where different frameworks typically serve their OpenAPI specs.

Step 3: Framework Detection

This step runs our framework detection and shows the results:

const detectFramework = async () => {
  if (!formData.value.sourceUrl && !formData.value.filePath) return

  frameworkDetection.value.loading = true

  try {
    const detection = await detectFrameworkComprehensive({
      url: formData.value.sourceUrl || undefined,
      filePaths: formData.value.filePath ? [formData.value.filePath] : undefined
    })

    frameworkDetection.value.results = detection.frameworks.map(fw => ({
      name: fw.name,
      displayName: getFrameworkDisplayName(fw.name),
      description: getFrameworkDescription(fw.name),
      confidence: fw.confidence
    }))

    // Auto-select the highest confidence framework
    if (frameworkDetection.value.results.length > 0) {
      formData.value.framework = frameworkDetection.value.results[0].name
    }
  } catch (error) {
    console.error('Framework detection failed:', error)
    frameworkDetection.value.results = []
  } finally {
    frameworkDetection.value.loading = false
  }
}

We show confidence percentages because framework detection isn't always 100% certain. It's like a detective showing their evidence and how confident they are in their conclusion.

Step 4: Confirmation - The Final Check

The final step shows a summary and lets users configure sync options:

<div class="confirmation-summary">
  <div class="summary-item">
    <span class="label">{{ t('collections.source_type') }}:</span>
    <span class="value">{{ getSourceTypeLabel(formData.sourceType) }}</span>
  </div>
  <div class="summary-item">
    <span class="label">{{ t('collections.source_location') }}:</span>
    <span class="value">{{ formData.sourceUrl || formData.filePath }}</span>
  </div>
  <div class="summary-item">
    <span class="label">{{ t('collections.collection_name') }}:</span>
    <span class="value">{{ formData.collectionName }}</span>
  </div>
</div>

This is like reviewing your order before checkout - we show exactly what will be created so there are no surprises.

The User Experience

The wizard uses progressive disclosure - each step only shows what's relevant at that moment. We validate inputs in real-time and provide immediate feedback. The navigation is smart - you can't proceed to the next step until the current one is valid.

const canProceed = computed(() => {
  switch (currentStep.value) {
    case 0:
      return formData.value.sourceType !== ''
    case 1:
      if (formData.value.sourceType === 'url') {
        return formData.value.sourceUrl && urlValidation.value.status === 'success'
      } else {
        return formData.value.filePath && fileValidation.value.status === 'success'
      }
    case 2:
      return true // Framework detection is optional
    case 3:
      return formData.value.collectionName.trim() !== ''
    default:
      return false
  }
})

Testing

Testing a wizard is like testing a GPS - you need to verify it guides users correctly through the entire journey:

describe('LiveSourceSetupWizard', () => {
  it('should guide user through complete setup flow', async () => {
    // Step 1: Select source type
    await wrapper.findAll('.source-option')[0].trigger('click')
    expect(wrapper.vm.formData.sourceType).toBe('url')

    // Step 2: Configure source
    await wrapper.find('.btn-primary').trigger('click')
    const urlInput = wrapper.find('input[type="url"]')
    await urlInput.setValue('http://localhost:3000/openapi.json')

    // Mock successful connection test
    vi.mocked(fetch).mockResolvedValueOnce({ ok: true } as Response)
    await urlInput.trigger('blur')

    // Step 3: Framework detection
    await wrapper.find('.btn-primary').trigger('click')
    expect(wrapper.vm.currentStep).toBe(2)

    // Step 4: Confirmation and creation
    await wrapper.find('.btn-primary').trigger('click')
    expect(wrapper.vm.currentStep).toBe(3)
  })
})

---

Status Indicators

The Problem

Live sync happens in the background, which is great for productivity but terrible for user confidence. Users need to know:

• Is my API connected?

• When was it last synced?

• Are there any errors?

• What framework was detected?

• Are there pending changes?

Without visual feedback, live sync feels like magic - and not the good kind. It's the "I have no idea what's happening" kind of magic.

The Solution: Status Indicators

We built two complementary components: LiveStatusIndicator for individual collections and LiveStatusList for managing multiple sources.

The Status Indicator: A Tiny Dashboard

<template>
  <div class="live-status-indicator" :class="statusClass">
    <div 
      class="status-badge"
      :class="[statusClass, { 'with-tooltip': showTooltip }]"
      @mouseenter="showTooltip = true"
      @mouseleave="showTooltip = false"
      @click="$emit('click')"
    >
      <!-- Framework Icon -->
      <div class="framework-icon" v-if="collection.liveMetadata?.framework">
        <component 
          :is="getFrameworkIconComponent(collection.liveMetadata.framework.name)" 
          class="svg-icons"
        />
      </div>

      <!-- Status Icon -->
      <div class="status-icon">
        <SmartSpinner v-if="status === 'syncing'" class="w-3 h-3" />
        <icon-lucide-check-circle v-else-if="status === 'connected'" class="svg-icons" />
        <icon-lucide-alert-circle v-else-if="status === 'error'" class="svg-icons" />
        <icon-lucide-x-circle v-else-if="status === 'disconnected'" class="svg-icons" />
        <icon-lucide-pause-circle v-else class="svg-icons" />
      </div>

      <!-- Change Indicator -->
      <div 
        v-if="hasPendingChanges" 
        class="change-indicator"
        :title="t('collections.pending_changes')"
      >
        <icon-lucide-dot class="svg-icons pulse" />
      </div>
    </div>
  </div>
</template>

This tiny component contains a lot of information:

• Framework icon: Shows what technology stack was detected

• Status icon: Visual indication of connection state

• Change indicator: Pulsing dot when there are pending changes

• Color coding: Green for good, red for errors, blue for syncing

The Tooltip: Information on Demand

When users hover over the status indicator, they get a rich tooltip with detailed information:

<div class="tooltip-content">
  <!-- Framework Info -->
  <div v-if="collection.liveMetadata?.framework" class="tooltip-section">
    <div class="section-label">{{ t('collections.framework') }}</div>
    <div class="section-content">
      <span class="framework-name">{{ collection.liveMetadata.framework.name }}</span>
      <span v-if="collection.liveMetadata.framework.version" class="framework-version">
        v{{ collection.liveMetadata.framework.version }}
      </span>
    </div>
  </div>

  <!-- Source Info -->
  <div class="tooltip-section">
    <div class="section-label">{{ t('collections.source') }}</div>
    <div class="section-content">
      <div class="source-info">
        <icon-lucide-globe v-if="sourceType === 'url'" class="svg-icons" />
        <icon-lucide-file v-else class="svg-icons" />
        <span class="source-path">{{ getSourcePath() }}</span>
      </div>
    </div>
  </div>

  <!-- Actions -->
  <div class="tooltip-actions">
    <button 
      v-if="status === 'error' || status === 'disconnected'"
      class="action-button primary"
      @click="$emit('retry')"
    >
      <icon-lucide-refresh-cw class="svg-icons" />
      {{ t('action.retry') }}
    </button>

    <button 
      v-if="hasPendingChanges"
      class="action-button secondary"
      @click="$emit('sync')"
    >
      <icon-lucide-download class="svg-icons" />
      {{ t('collections.sync_now') }}
    </button>
  </div>
</div>

The tooltip is like a detailed health report for your API connection. It shows not just what's happening, but provides actions to fix problems.

The Status List

For users with multiple live sources, we built a comprehensive management interface:

<template>
  <div class="live-status-list">
    <!-- Filters -->
    <div v-if="showFilters" class="list-filters">
      <div class="filter-group">
        <label class="filter-label">{{ t('collections.filter_by_status') }}</label>
        <select v-model="statusFilter" class="filter-select">
          <option value="all">{{ t('collections.all_statuses') }}</option>
          <option value="connected">{{ t('collections.status_connected') }}</option>
          <option value="syncing">{{ t('collections.status_syncing') }}</option>
          <option value="error">{{ t('collections.status_error') }}</option>
          <option value="disconnected">{{ t('collections.status_disconnected') }}</option>
        </select>
      </div>

      <div class="filter-group">
        <label class="filter-label">{{ t('collections.filter_by_framework') }}</label>
        <select v-model="frameworkFilter" class="filter-select">
          <option value="all">{{ t('collections.all_frameworks') }}</option>
          <option v-for="framework in availableFrameworks" :key="framework" :value="framework">
            {{ framework }}
          </option>
        </select>
      </div>
    </div>

    <!-- Status Items -->
    <div class="status-list">
      <div 
        v-for="source in filteredSources" 
        :key="source.collection.liveMetadata?.sourceId"
        class="status-item"
        :class="{ 'has-changes': source.hasPendingChanges }"
      >
        <!-- Collection Info with Quick Actions -->
        <div class="collection-info">
          <div class="collection-header">
            <div class="flex items-center space-x-2">
              <icon-lucide-folder class="svg-icons text-accent" />
              <span class="collection-name">{{ source.collection.name }}</span>
            </div>
            <LiveStatusIndicator
              :collection="source.collection"
              :status="source.status"
              size="sm"
              @retry="handleRetry(source)"
              @sync="handleSync(source)"
            />
          </div>
        </div>
      </div>
    </div>
  </div>
</template>

This component is like a dashboard for all your API connections. You can filter by status or framework, see which collections have pending changes, and take actions directly from the list.

Smart Status Calculation

The status calculation logic is particularly useful:

const getCollectionStatus = (collection: LiveSyncCollection) => {
  // Check if sync is in progress
  if (activeSyncs.has(collection.liveMetadata?.sourceId)) {
    return 'syncing'
  }

  // Check for errors
  if (collection.liveMetadata?.syncStatus?.success === false) {
    return 'error'
  }

  // Check for pending changes
  if (hasPendingCodeChanges(collection)) {
    return 'pending'
  }

  // Check for user modifications
  if (hasUserModifications(collection)) {
    return 'modified'
  }

  // All good!
  return 'connected'
}

This logic prioritizes the most important information - errors first, then pending changes, then user modifications, and finally the happy "everything is in sync" state.

Testing

Testing status indicators is like testing a medical diagnostic tool - accuracy is critical:

describe('LiveStatusIndicator', () => {
  it('should show correct status for each collection state', async () => {
    const collections = [
      createMockCollection({
        liveMetadata: {
          syncStatus: { success: true, lastAttempt: new Date() }
        }
      }),
      createMockCollection({
        liveMetadata: {
          syncStatus: { success: false, error: 'Connection failed' }
        }
      })
    ]

    mockCollections.getLiveSyncCollections.mockReturnValue(collections)

    await wrapper.vm.$forceUpdate()
    await nextTick()

    const indicators = wrapper.findAll('.status-indicator')
    expect(indicators[0].attributes('data-status')).toBe('connected')
    expect(indicators[1].attributes('data-status')).toBe('error')
  })
})

---

How It All Works Together

The magic happens when these three components work together:

The Happy Path Flow

1. User opens Setup Wizard → Guided through configuration

2. Wizard calls Sync Engine → syncEngine.startWatching(config)

3. Sync Engine orchestrates → Fetches spec, detects framework, creates collection

4. Status Indicators update → Show "connected" status with framework icon

5. File changes detected → File watcher notifies sync engine

6. Sync Engine processes → Compares specs, applies changes

7. Status Indicators reflect → Show "syncing" then "connected" with change count

The Error Handling Flow

1. Connection fails → Sync engine returns error result

2. Status shows error → Red indicator with error icon

3. User hovers for details → Tooltip shows specific error message

4. User clicks retry → Sync engine attempts reconnection

5. Status updates → Shows "syncing" then result

The Conflict Resolution Flow

1. Code and user both change same thing → Diff engine detects conflict

2. Status shows modified → Yellow indicator with warning

3. User opens tooltip → Shows conflict details and resolution options

4. User resolves conflict → Sync engine applies resolution

5. Status returns to connected → Green indicator, conflict resolved

The Developer Experience

Before these components, setting up live sync was like assembling furniture without instructions. Now it's like having a personal assistant who:

1. Guides you through setup with the wizard

2. Handles all the complexity with the sync engine

3. Keeps you informed with status indicators

4. Helps you fix problems with actionable error messages

The result? Developers can set up live sync in under 2 minutes and always know what's happening with their API connections.

What's Next: The Future of Live Sync

With the orchestration layer and UI complete, we're ready for the advanced features:

• Team Collaboration: Share live sources across team workspaces

• Webhook Integration: Trigger syncs from CI/CD pipelines

• Advanced Conflict Resolution: Visual diff tools for complex conflicts

• Performance Optimizations: Smarter change detection and batching

• Framework-Specific Features: Custom sync strategies for different stacks

The Symphony Complete

What we've built is more than just a sync engine, setup wizard, and status indicators. We've created a cohesive experience that transforms how developers work with APIs. The individual components are good, but together they create something greater than the sum of their parts.

It's like the difference between having individual musicians and having an orchestra. The musicians are talented, but the orchestra creates music that moves people.

The next time you modify an endpoint in your FastAPI application and see your Hoppscotch collection update automatically, with a friendly green indicator showing everything is in sync, remember: it all started with the question "How do we make this feel like magic?"

---

Ready to experience live sync for yourself? Check out the Hoppscotch repository and let us know what frameworks you'd like us to support next! In the next publication, we should be able to try out this live-sync feature.

Technical Deep Dives

For those who want to dive deeper into the implementation details:

• Sync Engine Architecture: See packages/hoppscotch-common/src/services/sync-engine.service.ts

• Setup Wizard Components: See packages/hoppscotch-common/src/components/collections/LiveSourceSetupWizard.vue

• Status Indicator System: See packages/hoppscotch-common/src/components/collections/LiveStatusIndicator.vue

• Collections Store Extensions: See packages/hoppscotch-common/src/newstore/collections.live-sync.ts

Each component includes comprehensive test suites and detailed documentation for future contributors.