Skip to main content
The quality of any AI system is bounded by the quality of its data. The SURCHI Data Pipeline is the infrastructure that ensures the sentinels operate on accurate, timely, and comprehensive signals. From raw Solana transactions to normalized feature vectors ready for AI inference, the pipeline handles ingestion, deduplication, normalization, and streaming with a target end-to-end latency of under 100 milliseconds from on-chain event to AI signal input.

​
Data Sources

The pipeline aggregates signals across six major categories, each with dedicated ingestion and normalization logic:
The primary data source is the Solana blockchain itself. SURCHI runs dedicated RPC nodes with the Geyser plugin enabled, which provides a real-time streaming interface for account updates, transaction confirmations, and program events. Unlike polling-based RPC approaches, Geyser delivers events as they are processed by the validator, enabling sub-100ms event ingestion. All significant program interactions — swaps, liquidity deposits and withdrawals, governance actions, large transfers — are captured and tagged at this layer.
SURCHI aggregates order book depth, quote routing data, and swap execution records from the three major Solana DEX venues. Jupiter’s aggregation layer provides cross-venue routing intelligence. Raydium’s concentrated liquidity pools and Orca’s Whirlpools provide granular tick-level liquidity data. Aggregated DEX data enables the Alpha and Liquidity Sentinels to assess true executable liquidity rather than theoretical pool depth.
Pool health is tracked continuously across all major Solana liquidity pools. Health metrics include: current liquidity depth by price range, utilization rate, fee accumulation rate, impermanent loss projections at current price levels, and pool age. These metrics feed directly into the Liquidity Sentinel’s risk models and are surfaced as alerts when health indicators cross user-defined thresholds.
Users configure watchlists of wallet addresses to monitor — their own, known market maker wallets, protocol treasuries, or any other addresses of strategic interest. The pipeline tracks all transactions from watchlisted wallets in real time, tagging movements with asset type, value in USDC terms, counterparty classification, and historical context. Configurable alert thresholds trigger Sentinel Core notifications for movements exceeding defined values.
Off-chain sentiment signals are ingested from X/Twitter and Telegram through aggregation integrations. Sentiment signals are classified by asset, assigned a directional score, and weighted by source authority metrics. Social sentiment is treated as a secondary signal — it informs Alpha Sentinel’s context layer but does not independently trigger execution. Correlation between on-chain flow and social sentiment is a stronger signal than either in isolation.
Perpetual futures funding rates from Solana-native derivatives venues provide a forward-looking sentiment indicator. Elevated positive funding rates signal leveraged long positioning; elevated negative rates signal short pressure. This data feeds Alpha Sentinel’s market regime classification models, helping distinguish trending from mean-reverting conditions.

​
Processing Architecture

Raw data from all sources enters a four-stage processing pipeline before reaching the AI inference layer:
1

Ingestion and Deduplication

Raw events arrive from multiple sources simultaneously. The ingestion layer deduplicates cross-source events (e.g., the same swap appearing in both Geyser streaming data and DEX API feeds), assigns canonical event IDs, and timestamps events with both chain-native slot timestamps and pipeline ingestion timestamps. Latency between the two timestamps is monitored as a pipeline health metric.
2

Normalization

Each event type is mapped to a canonical schema. Token amounts are normalized to USDC terms using real-time price feeds. Wallet addresses are resolved against classification databases (known exchanges, protocols, whale accounts). Pool addresses are resolved to their constituent assets and protocols. The output of normalization is a structured event record with a consistent schema regardless of source.
3

Feature Extraction

Normalized events are aggregated into feature vectors over rolling time windows (1 minute, 5 minutes, 15 minutes, 1 hour). Features include: net DEX flow by asset, liquidity depth changes by pool, whale wallet activity volume, sentiment score deltas, and funding rate changes. Feature extraction produces the dense numerical representations that AI models consume.
4

Streaming to AI Layer

Completed feature vectors are streamed to the Sentinel Core via an internal message bus. The Core routes vectors to the appropriate sentinel models based on feature type. Vectors that are stale beyond their freshness SLA are discarded rather than processed — the pipeline prioritizes accuracy over throughput when the two conflict.
Target end-to-end latency: sub-100ms from on-chain event to AI signal input.

​
Data Freshness Guarantees

Different data categories have different freshness requirements and different achievable latencies. The pipeline’s SLA targets by category:
Data CategoryTarget LatencyUpdate Frequency
On-chain transactions (Geyser)< 50msPer block (~400ms slots)
DEX order book depth< 100msContinuous streaming
Liquidity pool health metrics< 200msPer block
Whale wallet movements< 100msPer transaction
Social sentiment scores< 60 secondsRolling 1-minute aggregation
Funding rates< 30 secondsPer funding interval
These targets represent steady-state operation. During periods of exceptional network congestion, latency SLAs may be temporarily exceeded. The Sentinel Core is notified of SLA breaches and applies conservative signal weighting during degraded pipeline states.

​
Sample Normalized Signal Payload

The following is an example of a normalized signal payload as it exits the feature extraction stage and enters the Sentinel Core: 
{
  "signal_id": "sig_7f3a2c8e9d1b4f0a",
  "timestamp_chain": "2025-01-15T14:23:07.412Z",
  "timestamp_pipeline": "2025-01-15T14:23:07.498Z",
  "pipeline_latency_ms": 86,
  "signal_type": "dex_flow_anomaly",
  "asset_pair": "SOL/USDC",
  "venue": "jupiter_aggregated",
  "features": {
    "net_flow_usdc_1m": -2847523.40,
    "net_flow_usdc_5m": -6124890.15,
    "trade_count_1m": 1847,
    "large_trade_count_1m": 23,
    "largest_single_trade_usdc": 487200.00,
    "price_impact_bps_avg": 4.2,
    "liquidity_depth_change_pct": -3.8,
    "whale_wallet_participation": true,
    "whale_wallet_ids": ["wlt_classified_mm_02", "wlt_classified_cex_hot_01"]
  },
  "sentiment_overlay": {
    "score": -0.34,
    "confidence": 0.71,
    "source_count": 47
  },
  "routing_tags": ["alpha_sentinel", "liquidity_sentinel"],
  "freshness_valid_until": "2025-01-15T14:23:12.000Z"
}
Data signals are informational. Past signal accuracy does not guarantee future performance.