Workflows - SG Cars Trends API

Overview

The SG Cars Trends API includes a comprehensive workflow system for automated data updates and social media integration. This guide explains how workflows operate and how to interact with them.

Workflow Architecture

System Components

Data Workflows

Car registration data updates
COE bidding results updates
Automated file processing
Database synchronization

Social Media Integration

LinkedIn posting
Twitter updates
Discord notifications
Telegram messages

QStash Integration

The workflow system uses QStash for reliable message queuing and scheduling:

Scheduled Jobs: Automatic data updates via cron expressions
Retry Logic: Automatic retries for failed operations
Dead Letter Queue: Failed messages are preserved for analysis
Deduplication: Prevents duplicate processing of the same data

Data Update Workflows

Car Registration Workflow

Updates car registration data from LTA DataMall:

# Trigger car data update
curl -X POST "https://api.sgcarstrends.com/workflows/cars" \
  -H "Content-Type: application/json"

COE Bidding Workflow

Updates COE bidding results from official sources:

# Trigger COE data update
curl -X POST "https://api.sgcarstrends.com/workflows/coe" \
  -H "Content-Type: application/json"

Workflow Trigger (Authenticated)

Trigger both car and COE workflows simultaneously:

# Trigger all workflows (requires authentication)
curl -X POST "https://api.sgcarstrends.com/workflows/trigger" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

Workflow Process

1. Data Download and Validation

2. Key Features

Checksum Verification

SHA-256 checksums prevent reprocessing identical files

Batch Processing

Efficient bulk database operations for large datasets

Error Handling

Comprehensive error handling with Discord notifications

Conditional Publishing

Social media posts only when new data is available

3. Data Processing Steps

File Download: Retrieve data from LTA DataMall
Checksum Calculation: Generate SHA-256 hash of downloaded file
Duplicate Check: Compare with stored checksums in Redis
CSV Processing: Parse CSV data with validation
Database Insertion: Bulk insert new records
Timestamp Update: Update last processed timestamp
Social Media Trigger: Publish updates if new data exists

LinkedIn Integration

Post updates to LinkedIn company pages:

curl -X POST "https://api.sgcarstrends.com/workflows/linkedin/post" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "New car registration data available for January 2024",
    "link": "https://sgcarstrends.com/cars/2024-01"
  }'

Twitter Integration

Post updates to Twitter:

curl -X POST "https://api.sgcarstrends.com/workflows/twitter/post" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "📊 January 2024 car registration data is now available! Check out the latest trends in Singapore'"'"'s automotive market.",
    "link": "https://sgcarstrends.com/cars/2024-01"
  }'

Discord Integration

Send notifications to Discord channels:

curl -X POST "https://api.sgcarstrends.com/workflows/discord/post" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "🚗 **New Data Available**\n\nJanuary 2024 car registration data has been updated!\n\n🔗 View details: https://sgcarstrends.com/cars/2024-01"
  }'

Telegram Integration

Send messages to Telegram channels:

curl -X POST "https://api.sgcarstrends.com/workflows/telegram/post" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "📈 New car registration data for January 2024 is available! Electric vehicle registrations continue to grow.",
    "link": "https://sgcarstrends.com/cars/2024-01"
  }'

Scheduled Operations

Cron Schedule

Workflows run on the following schedule:

Workflow	Schedule	Description
Car Data Update	`0 0 15 * *`	15th of each month at midnight
COE Data Update	`0 0 * * 3`	Every Wednesday at midnight
Health Check	`0 /6 * *`	Every 6 hours

Manual Triggers

You can manually trigger workflows for testing or immediate updates:

// Function to trigger workflow and monitor result
async function triggerWorkflow(workflowType) {
  const endpoint = `https://api.sgcarstrends.com/workflows/${workflowType}`;
  
  try {
    const response = await fetch(endpoint, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
    });
    
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
    
    const result = await response.json();
    console.log(`${workflowType} workflow result:`, result);
    
    return result;
  } catch (error) {
    console.error(`Failed to trigger ${workflowType} workflow:`, error);
    throw error;
  }
}

// Usage
await triggerWorkflow('cars');
await triggerWorkflow('coe');

Error Handling and Monitoring

Error Scenarios

Download Failures

Network timeouts
Invalid file formats
Corrupted downloads

Processing Failures

CSV parsing errors
Database connection issues
Validation failures

Social Media Failures

API rate limits
Authentication errors
Content policy violations

Infrastructure Failures

Redis connection issues
QStash delivery failures
AWS service outages

Monitoring and Alerts

Discord Notifications: Critical errors are sent to Discord
QStash Dead Letter Queue: Failed messages are preserved
Health Check Endpoint: Monitor system status
Workflow Logs: Detailed logging for troubleshooting

Health Check

Monitor workflow system health:

curl "https://api.sgcarstrends.com/health" \
  -H "Content-Type: application/json"

Response Format

{
  "success": true,
  "data": {
    "status": "healthy",
    "timestamp": "2024-01-15T10:30:00Z",
    "services": {
      "database": "connected",
      "redis": "connected",
      "qstash": "operational"
    },
    "last_update": {
      "cars": "2024-01-15T02:00:00Z",
      "coe": "2024-01-10T02:00:00Z"
    }
  }
}

Best Practices

1. Workflow Timing

Be mindful of workflow timing to avoid conflicts with scheduled operations.

Avoid Peak Hours: Don’t trigger manual workflows during scheduled times
Rate Limiting: Respect API rate limits for external services
Database Load: Consider database performance during bulk operations

2. Error Recovery

async function robustWorkflowTrigger(workflowType, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const result = await triggerWorkflow(workflowType);
      return result;
    } catch (error) {
      console.log(`Attempt ${attempt} failed:`, error.message);
      
      if (attempt === maxRetries) {
        throw new Error(`Failed after ${maxRetries} attempts: ${error.message}`);
      }
      
      // Exponential backoff
      await new Promise(resolve => setTimeout(resolve, 1000 * Math.pow(2, attempt)));
    }
  }
}

3. Data Validation

Always validate workflow responses:

function validateWorkflowResponse(response) {
  if (!response.success) {
    throw new Error(`Workflow failed: ${response.error}`);
  }
  
  if (!response.data) {
    throw new Error('No data returned from workflow');
  }
  
  return true;
}

Development and Testing

Local Development

For local development and testing:

Environment Variables: Set up required environment variables
Mock Data: Use mock data for testing workflows
Dry Run Mode: Test workflows without actual data updates
Staging Environment: Test in staging before production

Testing Workflows

// Test workflow without side effects
async function testWorkflow(workflowType) {
  const testEndpoint = `https://api-staging.sgcarstrends.com/workflows/${workflowType}`;
  
  const response = await fetch(testEndpoint, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-Test-Mode': 'true' // Custom header for test mode
    },
  });
  
  return await response.json();
}

Integration Examples

Custom Workflow Integration

// Custom workflow that combines multiple operations
async function customDataUpdateWorkflow() {
  try {
    console.log('Starting custom data update workflow...');
    
    // Step 1: Trigger car data update
    const carResult = await triggerWorkflow('cars');
    console.log('Car data update completed:', carResult.success);
    
    // Step 2: Trigger COE data update
    const coeResult = await triggerWorkflow('coe');
    console.log('COE data update completed:', coeResult.success);
    
    // Step 3: Post to social media if new data was processed
    if (carResult.data?.processed_new_data || coeResult.data?.processed_new_data) {
      await postToSocialMedia('New data available for January 2024!');
    }
    
    console.log('Custom workflow completed successfully');
    
  } catch (error) {
    console.error('Custom workflow failed:', error);
    
    // Send alert to Discord
    await fetch('https://api.sgcarstrends.com/workflows/discord/post', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        message: `⚠️ Custom workflow failed: ${error.message}`
      })
    });
  }
}

async function postToSocialMedia(message) {
  const platforms = ['linkedin', 'twitter', 'discord', 'telegram'];
  
  const results = await Promise.allSettled(
    platforms.map(platform => 
      fetch(`https://api.sgcarstrends.com/workflows/${platform}/post`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ message })
      })
    )
  );
  
  results.forEach((result, index) => {
    if (result.status === 'fulfilled') {
      console.log(`${platforms[index]} post successful`);
    } else {
      console.error(`${platforms[index]} post failed:`, result.reason);
    }
  });
}

Next Steps

Development Setup

Set up local development environment

API Reference

Explore workflow API endpoints

Examples

See workflow integration examples

Monitoring

Learn about monitoring and testing

Core Concepts

​Overview

​Workflow Architecture

​System Components

Data Workflows

Social Media Integration

​QStash Integration

​Data Update Workflows

​Car Registration Workflow

​COE Bidding Workflow

​Workflow Trigger (Authenticated)

​Workflow Process

​1. Data Download and Validation

​2. Key Features

Checksum Verification

Batch Processing

Error Handling

Conditional Publishing

​3. Data Processing Steps

​Social Media Integration

​LinkedIn Integration

​Twitter Integration

​Discord Integration

​Telegram Integration

​Scheduled Operations

​Cron Schedule

​Manual Triggers

​Error Handling and Monitoring

​Error Scenarios

Download Failures

Processing Failures

Social Media Failures

Infrastructure Failures

​Monitoring and Alerts

​Health Check

​Response Format

​Best Practices

​1. Workflow Timing

​2. Error Recovery

​3. Data Validation

​Development and Testing

​Local Development

​Testing Workflows

​Integration Examples

​Custom Workflow Integration

​Next Steps

Development Setup

API Reference

Examples

Monitoring

Overview

Workflow Architecture

System Components

QStash Integration

Data Update Workflows

Car Registration Workflow

COE Bidding Workflow

Workflow Trigger (Authenticated)

Workflow Process

1. Data Download and Validation

2. Key Features

3. Data Processing Steps

Social Media Integration

LinkedIn Integration

Twitter Integration

Discord Integration

Telegram Integration

Scheduled Operations

Cron Schedule

Manual Triggers

Error Handling and Monitoring

Error Scenarios

Monitoring and Alerts

Health Check

Response Format

Best Practices

1. Workflow Timing

2. Error Recovery

3. Data Validation

Development and Testing

Local Development

Testing Workflows

Integration Examples

Custom Workflow Integration

Next Steps