Skip to main content
Use batch operations for efficient bulk data processing with progress tracking.

Basic Batch Storage

import { BulkBuilder } from '@onchaindb/sdk';

// Build batch of records
const builder = new BulkBuilder()
  .collection('tweets')
  .add({ message: 'Tweet 1', author: 'alice' })
  .add({ message: 'Tweet 2', author: 'bob' })
  .add({ message: 'Tweet 3', author: 'charlie' });

// Execute batch with progress tracking
const batch = client.batch();
const results = await batch.store(builder.build(), {
  concurrency: 5,
  waitForConfirmation: true,
  onProgress: (completed, total) => {
    console.log(`Progress: ${completed}/${total}`);
  }
});

console.log(`Successfully stored ${results.length} records`);

Progress Tracking (TypeScript)

Track batch operation progress in real-time:
const batch = client.batch();

await batch.store(builder.build(), {
  concurrency: 10,
  waitForConfirmation: true,
  onProgress: (completed, total) => {
    const percent = Math.round((completed / total) * 100);
    console.log(`${percent}% complete (${completed}/${total})`);

    // Update UI progress bar
    progressBar.style.width = `${percent}%`;
  }
});

Large Dataset Processing

import { BulkBuilder } from '@onchaindb/sdk';

async function importLargeDataset(records: any[]) {
  const BATCH_SIZE = 100;
  const results = [];

  for (let i = 0; i < records.length; i += BATCH_SIZE) {
    const chunk = records.slice(i, i + BATCH_SIZE);

    const builder = new BulkBuilder().collection('data');
    chunk.forEach(record => builder.add(record));

    const batch = client.batch();
    const batchResults = await batch.store(builder.build(), {
      concurrency: 5,
      waitForConfirmation: true,
      onProgress: (completed, total) => {
        const globalProgress = i + completed;
        console.log(`Overall: ${globalProgress}/${records.length}`);
      }
    });

    results.push(...batchResults);
    console.log(`Batch ${Math.floor(i / BATCH_SIZE) + 1} complete`);
  }

  return results;
}

Error Handling in Batches

const batch = client.batch();

try {
  const results = await batch.store(builder.build(), {
    concurrency: 5,
    waitForConfirmation: true
  });

  // Check for individual failures
  results.forEach((result, index) => {
    if (result.error) {
      console.error(`Record ${index} failed:`, result.error);
    } else {
      console.log(`Record ${index} stored at block:`, result.block_height);
    }
  });
} catch (error) {
  console.error('Batch operation failed:', error);
}

Concurrency Tuning

Adjust concurrency based on your needs:
ConcurrencyUse Case
1-3Rate-limited APIs, careful processing
5-10Standard batch operations
10-20High-throughput scenarios
// Conservative approach
await batch.store(data, { concurrency: 3 });

// Aggressive approach
await batch.store(data, { concurrency: 15 });

Next Steps