Research Intelligence
Search, crawl, extract, structure and retain web intelligence as reusable organizational knowledge.
Interactive platform map
Architecture in context
The focused blueprint, its required foundation, and declared recommendations.
Research Intelligence
Search, crawl, extract, structure and retain web intelligence as reusable organizational knowledge.
What this replaces
Web research is repetitive, disappears into individual sessions and often relies on expensive or opaque search APIs.
What your team gains
A private research pipeline that discovers sources, crawls approved targets, structures findings and stores them for semantic/graph retrieval.
What is inside the blueprint
How it fits the platform
Agent -> SearXNG discovery -> scrapy-mcp isolated crawl -> extraction/pipelines -> Postgres/pgvector + Neo4j -> retrieval/analysis.
SearXNG, scrapy-mcp, Scrapy, PostgreSQL/pgvector, Neo4j
- AICORTEX Core PlatformCrawl, search and storage services are Core-managed.
- MCP Agent HubResearch crawls are invoked and governed through the MCP tool layer.
From prerequisites to operation
- Research scope
- Target-domain policy/terms review
- Storage schema
- Crawler templates
- Configure SearXNG API
- Deploy/rebuild pinned Scrapy MCP image
- Create crawl isolation workflow
- Define extraction pipelines
- Connect persistence/indexing
- Search engines/categories
- Crawl depth
- Concurrency and delay
- Robots compliance
- Extraction selectors
- Deduplication
- Export/storage targets
- Target-site changes
- Rate limits/bans
- SDK/image dependency drift
- Crawler resource isolation
- Data freshness
What this unlocks with other layers
Research-to-Knowledge
Discovery becomes a retained, structured research corpus instead of disappearing after one chat.