trend-scanner

Scrapes social buzz, news feeds, Google Trends, Reddit, and X to surface trending topics in configurable verticals. Use when scanning for what’s hot, discovering new content opportunities, or feeding the topic scorer.

Model	Source
sonnet	pack: content-pumper

Full Reference

┏━ 🔍 trend-scanner ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Data ingestion layer for the Topic Brain system ┃ ┃ — scrapes the web, scores buzz, feeds memory ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

trend-scanner

Surfaces trending topics from across the web and feeds them into topic-memory. Runs on-demand or on a schedule. Output is a scored, deduplicated array of trend objects ready for topic-scorer and sentiment-mapper.

---

Scan Sources

Source	Tool	Signal
Google Trends	WebSearch (trending [vertical] [date])	Search velocity, breakout queries
Reddit	firecrawl (reddit.com/r/<subreddit>/hot)	Upvotes, comment count, post velocity
X / Twitter	WebSearch (site:x.com [vertical] trending)	Mention clusters, retweet signals
News aggregators	firecrawl (Google News, AP, Reuters RSS)	Recency, syndication breadth
Industry-specific feeds	firecrawl (configured per vertical)	Niche authority signal

---

Verticals Config

Default verticals — configurable per project via content-topics.json config block or env var CONTENT_VERTICALS:

Vertical	Default Sources
sports	NFL, NBA, MLB, NHL, MLS, UFC, college football/basketball
tech	HackerNews, TechCrunch, Verge, Product Hunt
local-news	Local paper RSS, city subreddits, NextDoor signals
business	Bloomberg, WSJ, CNBC, r/investing, r/entrepreneur
entertainment	Billboard, Deadline, r/movies, r/television

Add custom verticals by extending the config block — scanner picks them up automatically.

---

Output Schema

Each scan returns an array of trend objects:

[
  {
    "title": "<string>",
    "source_url": "<string>",
    "source_platform": "google-trends | reddit | x | news | industry",
    "buzz_score": 0,
    "velocity": "rising | stable | falling",
    "category": "<string>",
    "verticals": ["<string>"],
    "discovered_at": "<ISO timestamp>",
    "raw_signals": {
      "mentions": 0,
      "shares": 0,
      "comments": 0
    }
  }
]

buzz_score is 0–100. velocity is derived from signal rate-of-change: rising if mentions doubled in last 6h, falling if halved, stable otherwise.

---

Scan Process

1. Read config — load verticals from content-topics.json config block or CONTENT_VERTICALS env var
2. WebSearch per vertical — run targeted queries with current date to force freshness (e.g., "NFL trending today 2026-03-01")
3. Firecrawl top results — scrape the top 2 URLs per vertical for deeper signal extraction (comment counts, share counts, engagement depth)
4. Deduplicate — fuzzy title match at 80% similarity threshold; merge duplicates, keep highest buzz_score entry
5. Score each trend — calculate buzz_score from raw_signals using the formula below
6. Feed topic-memory — for each trend: if topic exists → update-signals; if new → add-topic then update-signals

Buzz Score Formula

buzz_score = min(100, (
  (mentions × 0.5) +
  (shares × 0.3) +
  (comments × 0.2)
) / normalization_factor)

normalization_factor is the max raw signal value observed in the current scan batch — ensures relative scoring within each run.

---

Rate Limiting

Constraint	Limit
WebSearch queries per scan	Max 20
firecrawl pages per scan	Max 10
Scan frequency	Min 6h between full scans (respect config.checkIntervalHours)
Queries per vertical	Max 4 WebSearch + 2 firecrawl pages

Abort the scan and log a warning if limits are hit mid-run. Partial results are still valid — write what was collected.

---

Deduplication

Fuzzy title matching using normalized Levenshtein distance:

• Lowercase + strip punctuation before comparing
• Topics within 80% similarity are merged
• On merge: keep highest buzz_score, union verticals[], sum raw_signals
• Log merged topic pairs for audit trail

---

Integration

Skill	Relationship
topic-memory	Writes via add-topic + update-signals
topic-scorer	Reads trend-scanner output to calculate composite score
sentiment-mapper	Reads trending topics to map polarization angles
topic-brain-pimp	Orchestrates scan cadence and routes output
content-pumper-pimp	Triggers scans at configured intervals