Vault Development · Production Vault Operations · 6/7
Monitoring & Alerts
Why Monitoring Matters
A vault keeper running without monitoring is a ticking time bomb. If the keeper crashes, loses connection, or makes a bad trade, you need to know immediately — not when depositors start complaining.
Key Metrics to Monitor
Health Metrics
- Keeper uptime — Is the process running?
- Last heartbeat — When was the last successful loop iteration?
- WebSocket connectivity — Is the data feed alive?
- API response times — Are requests taking too long?
Trading Metrics
- Account equity — Track equity curve over time
- Drawdown — Current drawdown from peak equity
- Margin ratio — How much margin is being used
- Position count — Number of open positions
- Unrealized PnL — Current PnL across all positions
Risk Metrics
- Max position size — Alert if any position exceeds limits
- Leverage — Alert if effective leverage exceeds threshold
- Funding payments — Track funding received/paid per hour
Implementing Monitoring
import logging
import time
from dataclasses import dataclass, field
from datetime import datetime
@dataclass
class VaultMetrics:
equity: float = 0.0
peak_equity: float = 0.0
drawdown_pct: float = 0.0
margin_ratio: float = 0.0
num_positions: int = 0
total_pnl: float = 0.0
last_update: datetime = field(default_factory=datetime.now)
errors_last_hour: int = 0
class VaultMonitor:
def __init__(self, vault_address: str, alert_callback=None):
self.vault_address = vault_address
self.metrics = VaultMetrics()
self.alert_callback = alert_callback
self.logger = logging.getLogger("vault_monitor")
def update(self, vault_state: dict):
summary = vault_state["marginSummary"]
equity = float(summary["accountValue"])
margin_used = float(summary["totalMarginUsed"])
self.metrics.equity = equity
self.metrics.peak_equity = max(self.metrics.peak_equity, equity)
self.metrics.drawdown_pct = (
(self.metrics.peak_equity - equity) / self.metrics.peak_equity * 100
if self.metrics.peak_equity > 0 else 0
)
self.metrics.margin_ratio = margin_used / equity if equity > 0 else 1.0
self.metrics.num_positions = len(vault_state.get("assetPositions", []))
self.metrics.last_update = datetime.now()
# Check alert conditions
self._check_alerts()
def _check_alerts(self):
if self.metrics.drawdown_pct > 10:
self._alert(f"DRAWDOWN ALERT: {self.metrics.drawdown_pct:.1f}%")
if self.metrics.margin_ratio > 0.7:
self._alert(f"HIGH MARGIN: {self.metrics.margin_ratio:.1%}")
def _alert(self, message: str):
self.logger.warning(message)
if self.alert_callback:
self.alert_callback(message)
Alert Channels
For production vaults, set up alerts via:
- Telegram bot — Most common for crypto bots
- Discord webhook — Good for team notifications
- PagerDuty/OpsGenie — For serious production setups
- SMS — For critical alerts (keeper down, liquidation risk)
Logging Best Practices
# Use structured logging for easy parsing
import json
import logging
from datetime import datetime
logger = logging.getLogger("vault_trades")
def log_trade(vault_address: str, action: str, coin: str, size: float,
price: float, result: dict):
log_entry = {
"timestamp": datetime.now().isoformat(),
"action": action,
"coin": coin,
"size": size,
"price": price,
"result_status": result.get("status"),
"vault": vault_address,
}
logger.info(json.dumps(log_entry))
Store logs externally (CloudWatch, Datadog) so you can analyze trading history even if the server dies.