# Homepage Preview
Continuous Market Intelligence
Sibyl helps founders, marketers and analysts follow their markets without rebuilding the same research each week. Teams track a portfolio of topics, compare attention and momentum, investigate signals and related opportunities, and review geographic interest, search results and source evidence. Watchlists, market theses, alerts and briefs turn those observations into a recurring decision workflow; Ask Sibyl answers questions from workspace evidence. I conceived the product, its scoring and signal algorithms, and the asynchronous architecture behind collection, shared computation and historical monitoring.
Market research becomes difficult to maintain when search demand, news, competitors and social activity are scattered across providers. One-off reports age quickly, while short-lived spikes and incomplete observations can distort the interpretation of a market.
Sibyl keeps a portfolio of markets under recurring observation, then turns available search, news, video and social evidence into comparable views and explainable signals. Teams can investigate what changed, follow an opportunity, assess a market thesis and share a brief. Deterministic algorithms compute market scores; AI helps explain persisted evidence through summaries and contextual answers.
Choose niches, brands or topics and follow their attention, momentum and historical movement in a shared workspace. Compare tracked markets without repeating every search manually.
Investigate signals and risk flags, work through a review queue, and configure monitoring rules and alerts to support a recurring review routine.
Explore source-backed subtopics and adjacent markets, review why they relate to your topic, and keep promising candidates in an opportunity watchlist.
Inspect source coverage, geographic interest, search-result composition, actors and saturation indicators. Open the observations behind an interpretation.
Turn a business hypothesis into a monitored thesis, inspect its evidence and timeline, and revisit it as new observations become available.
Ask Sibyl about observed changes, comparisons or risks. Review AI-assisted answers with citations, prepare market and daily-review briefs, and share or export findings with your team.
Combining provider data with different timestamps, formats, coverage and freshness without treating missing observations as zero.
Separating short-lived spikes from structural market movement while preserving the direction and timing of real changes.
Sharing collection and computation across users to control processing costs without mixing topic context or workspace access.
Recovering delayed postbacks and partial collection cycles while keeping published analytical views consistent.
Giving AI answers useful source citations while keeping scores, signals and market decisions tied to the deterministic analytical model.
Keeping contextual analysis relevant and traceable as the available evidence changes.
Sibyl deploys Next.js on managed hosting and the NestJS backend on a separate Docker Compose host behind an edge gateway and Nginx. Redis/BullMQ transports work between dedicated provider I/O, materialization and reliability processes. Managed services supply identity and PostgreSQL; Drizzle persists observations, historical snapshots and read models. Provider postbacks return through the API origin. Redis update events and an SSE proxy refresh the UI without triggering provider collection on dashboard reads.
A managed web runtime connects to the API origin through an edge gateway. Nginx fronts a private Docker network containing the NestJS API, Redis and three dedicated worker roles.
The api process does not consume queues or run schedulers. Collection, projection building and recovery run in separately provisioned containers built from the same NestJS image.
Dashboard reads use PostgreSQL projections and Redis caching. Workers publish update events through Redis; API instances invalidate cached views and relay updates over SSE.
The browser reaches the application through web and API gateways. Data collection, analysis and publication remain backend responsibilities, separate from dashboard reads.
| From | To | Protocol / responsibility | Flow |
|---|---|---|---|
| Analyst browser | Next.js application | HTTPS | Request / response |
| Next.js application | Identity service | Auth / JWT | Request / response |
| Next.js application | Edge gateway | HTTPS · API + SSE | Request / response |
| Edge gateway | Nginx | HTTPS · origin TLS | Request / response |
| Certificate services | Nginx | TLS files / reload | Control / operations |
| Nginx | NestJS API | HTTP · private upstream | Request / response |
| NestJS API | Redis / BullMQ | Jobs / cache / events | Queued work |
| Redis / BullMQ | NestJS workers | BullMQ dispatch | Queued work |
| NestJS API | PostgreSQL | SQL / TLS · read projections | Persistent data |
| NestJS workers | PostgreSQL | Persist observations / projections | Persistent data |
| NestJS workers | Market data providers | Provider tasks / collection | Request / response |
| Market data providers | Nginx | Provider callbacks | Events / callbacks |
| NestJS API | AI inference service | Evidence-grounded inference | Request / response |
| NestJS workers | Commercial services | Durable outbound effects | Request / response |
| Commercial services | Edge gateway | Billing webhooks | Events / callbacks |
Provider task submission, provider callbacks and materialization are separate flows. PostgreSQL records cycles and observations; dedicated consumers publish the analytical read models.
Provider callbacks enter through the public API origin and enqueue processing. The materialization role handles callback processing, late-reaggregation and materialization-recovery.
PostgreSQL stores cycle/task correlation, observations and publication state. Redis is transport and cache; it does not replace the persisted analytical history.
Reliability detects overdue work and enqueues recovery. The materialization owner reloads persisted state, rejects stale recovery commands and repairs the relevant read models.
| From | To | Protocol / responsibility | Flow |
|---|---|---|---|
| NestJS API | Redis / BullMQ | Enqueue | Queued work |
| External providers | NestJS API | Postback → Nginx → API | Events / callbacks |
| Redis / BullMQ | provider-io-worker | Collection queues | Queued work |
| Redis / BullMQ | materialization-worker | Postback / recovery queues | Queued work |
| Redis / BullMQ | reliability-worker | Watchdog queue | Queued work |
| provider-io-worker | External providers | Submit provider tasks | Request / response |
| materialization-worker | PostgreSQL / Drizzle | Aggregate / score / publish | Persistent data |
| provider-io-worker | PostgreSQL / Drizzle | Observations + outboxes | Persistent data |
| reliability-worker | Redis / BullMQ | Enqueue recovery | Control / operations |
| PostgreSQL / Drizzle | NestJS API | Read published projections | Persistent data |
| materialization-worker | Redis / BullMQ | Read-model update events | Events / callbacks |
| NestJS API | Next.js dashboard | Persisted reads + SSE | Request / response |
Dashboard requests consume persisted projections; provider work and expensive recomputation run outside the HTTP request.
Observations retain their source, context and timestamp. Algorithms compute scores from that evidence; AI explains it. Missing history remains unknown rather than becoming an invented growth value.
Provider collection, materialization and recovery have dedicated process roles so scaling one workload does not change the API's responsibilities.
Add the market or opportunity to investigate, with context relevant to the research.
Follow collection activity as Sibyl gathers observations from configured providers.
Review scores, geographic interest and sources; explore related opportunities and ask contextual questions.
Follow signals, watchlists and market theses over time; review alerts and evidence as observations change.
Prepare a cited market brief or report and share findings to support the team's next decision.
Move from a research topic to an ongoing view of the market and its changes.
Discuss this Project