Documentation

API Reference

Complete reference for TorScan REST API endpoints for dark web intelligence gathering.

Base URL

https://your-torscan-instance.com/api/v1

Authentication

TorScan uses JWT tokens for API authentication. Obtain a token by logging in.

# Login to get JWT token
curl -X POST "https://your-instance.com/api/v1/auth/login" \
  -H "Content-Type: application/json" \
  -d '{
    "username": "your-username",
    "password": "your-password"
  }'

# Response
{
  "access_token": "eyJ0eXAiOiJKV1QiLCJhbGc...",
  "refresh_token": "eyJ0eXAiOiJKV1QiLCJhbGc...",
  "expires_in": 3600,
  "user": {
    "id": "user_123",
    "username": "your-username",
    "role": "admin"
  }
}

# Use token in requests
curl -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGc..." \
  https://your-instance.com/api/v1/scans

Scan Management

Create New Scan

POST/scans

Initiate a new dark web scan with specified targets and patterns.

Request Body

{
  "name": "Security Breach Monitoring",
  "description": "Monitor for company data breaches",
  "sources": [
    "http://darkwebsite1.onion",
    "http://darkwebsite2.onion"
  ],
  "patterns": [
    "company.com",
    "breach",
    "database dump",
    "credit card"
  ],
  "config": {
    "depth": 3,
    "max_pages": 1000,
    "timeout": 300,
    "circuit_rotation": true,
    "follow_redirects": true,
    "respect_robots": false
  },
  "schedule": null  // Or schedule object for recurring scans
}

Response

{
  "scan_id": "scan_20240115_abc123",
  "status": "queued",
  "created_at": "2024-01-15T10:00:00Z",
  "estimated_duration": "45 minutes",
  "queue_position": 2,
  "links": {
    "self": "/api/v1/scans/scan_20240115_abc123",
    "results": "/api/v1/scans/scan_20240115_abc123/results",
    "logs": "/api/v1/scans/scan_20240115_abc123/logs"
  }
}

Get Scan Status

GET/scans/{scan_id}

Retrieve current status and progress of a specific scan.

{
  "scan_id": "scan_20240115_abc123",
  "name": "Security Breach Monitoring",
  "status": "running",
  "progress": {
    "sites_crawled": 45,
    "total_sites": 100,
    "pages_processed": 3567,
    "matches_found": 23,
    "errors": 2
  },
  "started_at": "2024-01-15T10:00:30Z",
  "elapsed_time": "00:23:45",
  "estimated_remaining": "00:21:15",
  "current_url": "http://example.onion/forum/page23",
  "tor_circuit": {
    "ip": "185.220.101.45",
    "country": "Netherlands",
    "circuit_id": "circuit_789"
  }
}

List Scans

GET/scans

Retrieve a paginated list of all scans.

# Query parameters
?status=running,completed
&from_date=2024-01-01
&to_date=2024-01-31
&page=1
&limit=20
&sort=created_at:desc

# Response
{
  "scans": [
    {
      "scan_id": "scan_20240115_abc123",
      "name": "Security Breach Monitoring",
      "status": "completed",
      "created_at": "2024-01-15T10:00:00Z",
      "completed_at": "2024-01-15T10:45:23Z",
      "stats": {
        "sites_crawled": 100,
        "pages_processed": 8934,
        "matches_found": 47,
        "duration": "00:45:23"
      }
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 20,
    "total_pages": 5,
    "total_items": 89
  }
}

Search & Results

Search Results

GET/search

Full-text search across all scan results with Elasticsearch.

# Query parameters
?q=bitcoin AND wallet
&scan_id=scan_20240115_abc123  # Optional: limit to specific scan
&from_date=2024-01-01
&to_date=2024-01-31
&confidence_min=0.7
&source_domain=*.onion
&highlight=true
&page=1
&limit=50

# Response
{
  "results": [
    {
      "id": "result_456",
      "scan_id": "scan_20240115_abc123",
      "url": "http://darkmarket.onion/forum/thread123",
      "title": "Bitcoin Wallet Dumps",
      "snippet": "...found <mark>bitcoin</mark> <mark>wallets</mark> with balance...",
      "matched_patterns": ["bitcoin", "wallet"],
      "confidence": 0.92,
      "found_at": "2024-01-15T10:23:45Z",
      "metadata": {
        "page_size": 45678,
        "language": "en",
        "tor_exit_node": "185.220.101.45"
      },
      "context": {
        "before": "Previous 100 characters of context",
        "match": "The exact matched text with patterns",
        "after": "Following 100 characters of context"
      }
    }
  ],
  "aggregations": {
    "by_pattern": {
      "bitcoin": 234,
      "wallet": 189,
      "breach": 67
    },
    "by_domain": {
      "darkmarket.onion": 145,
      "forum123.onion": 89
    }
  },
  "total": 423,
  "took_ms": 145
}

Get Scan Results

GET/scans/{scan_id}/results

Retrieve all results from a specific scan.

{
  "scan_id": "scan_20240115_abc123",
  "total_results": 47,
  "results": [
    {
      "id": "result_123",
      "url": "http://example.onion/page",
      "matched_patterns": ["company.com", "breach"],
      "confidence": 0.95,
      "severity": "high",
      "content": {
        "title": "Data Breach Discussion",
        "body_preview": "Found company.com database...",
        "full_content_available": true
      },
      "iocs": {
        "emails": ["user@company.com"],
        "ips": ["192.168.1.1"],
        "domains": ["company.com"],
        "bitcoin_addresses": ["1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa"],
        "hashes": {
          "md5": ["5d41402abc4b2a76b9719d911017c592"],
          "sha256": ["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"]
        }
      }
    }
  ],
  "export_formats": ["json", "csv", "misp", "stix"]
}

Export Results

GET/scans/{scan_id}/export

Export scan results in various formats.

# Export as CSV
GET /scans/scan_20240115_abc123/export?format=csv

# Export as MISP Event
GET /scans/scan_20240115_abc123/export?format=misp

# Export as STIX 2.1
GET /scans/scan_20240115_abc123/export?format=stix

# Response headers for CSV
Content-Type: text/csv
Content-Disposition: attachment; filename="scan_20240115_abc123_results.csv"

Pattern Management

Create Pattern Set

POST/patterns

{
  "name": "Financial Fraud Indicators",
  "description": "Patterns for detecting financial fraud",
  "patterns": [
    {
      "pattern": "credit card",
      "type": "keyword",
      "case_sensitive": false
    },
    {
      "pattern": "\b(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14})\b",
      "type": "regex",
      "description": "Credit card numbers",
      "severity": "critical"
    },
    {
      "pattern": "bitcoin:([13][a-km-zA-HJ-NP-Z1-9]{25,34})",
      "type": "regex",
      "description": "Bitcoin addresses",
      "extract_group": 1
    }
  ],
  "tags": ["financial", "fraud", "pii"],
  "enabled": true
}

Test Pattern

POST/patterns/test

{
  "pattern": "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
  "type": "regex",
  "test_text": "Contact us at admin@example.com or support@test.org"
}

# Response
{
  "matches": [
    {
      "match": "admin@example.com",
      "start": 14,
      "end": 31
    },
    {
      "match": "support@test.org",
      "start": 35,
      "end": 51
    }
  ],
  "match_count": 2,
  "execution_time_ms": 2
}

Schedule Management

Create Scheduled Scan

POST/schedules

{
  "name": "Daily Dark Web Monitor",
  "scan_config": {
    "sources": ["config/daily_sources.yaml"],
    "patterns": ["pattern_set_123", "pattern_set_456"]
  },
  "schedule": {
    "type": "cron",
    "expression": "0 2 * * *",  // 2 AM daily
    "timezone": "UTC"
  },
  "notifications": {
    "email": ["security@company.com"],
    "webhook": "https://slack.com/webhook/xyz",
    "conditions": {
      "min_severity": "medium",
      "min_matches": 1
    }
  },
  "retention": {
    "keep_results_days": 90,
    "keep_logs_days": 30
  },
  "enabled": true
}

Threat Intelligence Integration

Submit to MISP

POST/threat-intel/misp/events

{
  "scan_id": "scan_20240115_abc123",
  "result_ids": ["result_123", "result_456"],
  "event_info": "Dark Web Intelligence - Company Breach",
  "distribution": 3,  // All communities
  "threat_level": 2,  // Medium
  "analysis": 2,      // Ongoing
  "tags": ["dark-web", "data-breach", "tlp:amber"],
  "attributes": {
    "auto_extract": true,
    "include_context": true,
    "correlation": true
  }
}

Query Threat Intelligence

GET/threat-intel/search

# Query parameters
?ioc=1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
&type=bitcoin-address
&sources=misp,opencti,internal

# Response
{
  "matches": [
    {
      "source": "misp",
      "event_id": "12345",
      "event_info": "Ransomware Campaign Q1 2024",
      "first_seen": "2024-01-10T08:00:00Z",
      "last_seen": "2024-01-15T14:30:00Z",
      "tags": ["ransomware", "bitcoin", "tlp:white"],
      "correlation_count": 23,
      "sightings": 145
    }
  ],
  "risk_score": 8.5,
  "recommendations": [
    "Block bitcoin address in monitoring",
    "Check for related indicators",
    "Update threat intelligence feeds"
  ]
}

Plugin System

Upload Plugin

POST/plugins

# Multipart form upload
curl -X POST https://your-instance.com/api/v1/plugins \
  -H "Authorization: Bearer your-token" \
  -F "plugin=@custom_crawler.py" \
  -F 'metadata={
    "name": "Custom Forum Crawler",
    "type": "crawler",
    "version": "1.0.0",
    "description": "Specialized crawler for forum sites",
    "author": "security-team",
    "config": {
      "max_depth": 5,
      "follow_pagination": true
    }
  }'

# Response
{
  "plugin_id": "plugin_custom_forum_crawler",
  "status": "uploaded",
  "validation": "passed",
  "loaded": true,
  "hooks": [
    "pre_crawl",
    "post_page_process",
    "on_match_found"
  ]
}

Available Plugin Types

Crawlers

Custom site crawling logic, authentication, navigation

Matchers

Advanced pattern matching, ML-based detection, custom scoring

Exporters

Custom export formats, integration with external systems

Notifiers

Custom alerts, webhooks, messaging integrations

WebSocket API

Real-time updates for scan progress and results.

// JavaScript WebSocket connection
const ws = new WebSocket('wss://your-instance.com/ws');

ws.on('open', () => {
  // Authenticate
  ws.send(JSON.stringify({
    type: 'auth',
    token: 'your-jwt-token'
  }));
  
  // Subscribe to scan updates
  ws.send(JSON.stringify({
    type: 'subscribe',
    channel: 'scan',
    scan_id: 'scan_20240115_abc123'
  }));
});

// Receive real-time updates
ws.on('message', (data) => {
  const message = JSON.parse(data);
  
  switch(message.type) {
    case 'scan_progress':
      console.log(`Progress: ${message.progress}%`);
      break;
      
    case 'match_found':
      console.log('New match:', message.result);
      break;
      
    case 'scan_complete':
      console.log('Scan completed:', message.summary);
      break;
  }
});

Response Codes

Code	Description
200	Success - Request completed successfully
201	Created - Resource created successfully
202	Accepted - Scan queued for processing
400	Bad Request - Invalid parameters
401	Unauthorized - Invalid or missing token
403	Forbidden - Insufficient permissions
404	Not Found - Resource not found
429	Too Many Requests - Rate limit exceeded
500	Internal Server Error
503	Service Unavailable - Tor circuit issues

Rate Limits

Endpoint	Rate Limit
/auth/*	10 requests/minute
/scans (POST)	20 requests/hour
/search	100 requests/minute
/threat-intel/*	50 requests/minute
WebSocket connections	5 concurrent per user

API Client Libraries

Python Client Library JavaScript/Node.js Client Go Client Library Postman Collection