WorkData & intelligence platformdraft · under review
MarketView
A data-intensive intelligence platform that turns fragmented market activity into structured, queryable insight.
Stack as evidence
Laravel · MySQL · ClickHouse · Redis · Queues
Concerns
Ingestion pipelines · Normalisation · Time-series analytics · Aggregation · Reporting APIs
- 01
Sources
Heterogeneous, unreliable
- 02
Ingestion
Queued, replayable
- 03
Normalisation
One schema, provenance kept
- 04
Analytical store
Columnar, time-series
- 05
Insight
Aggregates, trends, APIs
01The problem
The data existed. It just did not agree with itself.
Market information arrives from many sources, in different shapes, at different cadences, and with different levels of trust. Making it useful means ingesting all of it, reconciling it into a consistent model, and answering analytical questions over time ranges that a transactional database was never designed to scan.
02The constraints
What could not simply be changed.
Sources cannot be fixed upstream
The platform had to be tolerant of late, duplicated and malformed input rather than assume clean data.
Questions are analytical, not transactional
Users ask about trends and comparisons across time, which shapes the storage choice more than anything else.
03The architecture
Ingestion is a queue-backed pipeline: raw records are captured with provenance, normalised into a single schema, and loaded into a columnar analytical store suited to time-series aggregation. The transactional database keeps ownership of reference data and configuration; the analytical store owns the volume. Insight surfaces as pre-computed aggregates and query APIs, with Redis caching the expensive, frequently requested views.
04Engineering decisions
01
Keep the raw record
Normalisation rules change. Keeping raw input with provenance means the pipeline can be replayed instead of the data re-collected.
02
Separate the analytical store
Scanning months of records for a trend is a columnar workload. Giving it a columnar home kept the transactional side fast and the analytical side honest.
05The trade-offs
What complexity I introduced, on purpose.
- Two stores mean two models to keep aligned and a pipeline that has to be monitored.
- Pre-computed aggregates trade freshness for speed, so the platform is explicit about how recent each view is.
06The outcome
- A pipeline that turns inconsistent market data into a consistent, queryable analytical model.
- Analytical questions answered from a store built for them rather than from the product database.
07The lesson
Data platforms succeed or fail on ingestion discipline. If you cannot replay, you cannot fix mistakes, and you will make mistakes.