Data engineering · quant research
Solana Market Data Collection & Analytics Pipeline
Streaming pipeline that records every new Solana token launch, then mines the history for trading rules that survive out-of-sample validation.
- Year
- 2026
- Status
- Active — collecting
- Stack
- Python · asyncio · WebSockets · DuckDB · SQL · Parquet · Streamlit · Plotly
Pump.fun launches tokens on Solana at a rate of thousands per day, and nearly all of them go to zero. Every trading “strategy” you read about the platform is vibes and survivorship bias. I wanted an answer grounded in data I collected myself.
Collect everything, filter later
The collector holds a WebSocket connection to PumpPortal and writes every token creation event, plus the bonding-curve trades that follow, into an append-only event log. Nothing gets filtered at ingest, because any filter would bake in assumptions about what matters, and I don't know that yet.
Mining rules without fooling myself
A separate miner replays the collected history looking for entry rules that would have worked. The split is chronological: train on the first 70% of time, validate on the last 30%. A random split would leak the future into the past, since this is time-series data. A rule only counts if it holds up on data it hasn't seen.
- Ingest
- PumpPortal WebSocket · asyncio
- Storage
- DuckDB and Parquet: append-only raw events, plus normalized, deduplicated datasets
- Validation
- Chronological 70/30 train/test split
- Analysis
- SQL workflows, time-based features, Streamlit and Plotly for inspection
Current status
The collector runs unattended for days at a time. A July 2026 audit recorded more than 135,000 tokens and 5.5 million time-series observations; those are historical audit counts, not live totals. This is data collection and analysis only, with no trading involved. If no rule survives validation, that's still a useful result.