Marketplace
LIVEAgents / agency
Research Intelligence
Search, crawl, extract, structure and retain web intelligence as reusable organizational knowledge.
Live workload cutawayArchitecture in context
Research Intelligence
Signal path active6 deployment units
What this replaces
Web research is repetitive, disappears into individual sessions and often relies on expensive or opaque search APIs.
What your team gains
A private research pipeline that discovers sources, crawls approved targets, structures findings and stores them for semantic/graph retrieval.
What is inside the blueprint
Private metasearch across many engines/categories
HTTP search API with JSON/CSV/RSS output options
Reduced user profiling and query leakage
Concurrent/fault-tolerant web crawling
CSS/XPath/regex extraction
Spiders and generic sitemap/feed spiders
Scheduler, downloader and middleware extensibility
Cookies, auth, caching, compression, robots and crawl depth features
AutoThrottle/concurrency controls
Item pipelines for cleaning/validation/persistence
JSON/JSONL/CSV/XML feed exports and multiple storage backends
Container-per-crawl isolation in AICORTEX
Embedding and graph enrichment after crawl
How it fits the platform
Agent -> web search discovery -> crawler-mcp isolated crawl -> extraction/pipelines -> relational database/vector store + graph database -> retrieval/analysis.
Included services
Web search, crawler-mcp, Web crawler, Relational database, Vector store, Graph database
Platform requirements
- AICORTEX Core PlatformCrawl, search and storage services are Core-managed.
- MCP Agent HubResearch crawls are invoked and governed through the MCP tool layer.
From prerequisites to operation
Prerequisites
- Research scope
- Target-domain policy/terms review
- Storage schema
- Crawler templates
Deployment
- Configure the web search API
- Deploy/rebuild the pinned crawler MCP image
- Create crawl isolation workflow
- Define extraction pipelines
- Connect persistence/indexing
Configuration
- Search engines/categories
- Crawl depth
- Concurrency and delay
- Robots compliance
- Extraction selectors
- Deduplication
- Export/storage targets
Operations
- Target-site changes
- Rate limits/bans
- SDK/image dependency drift
- Crawler resource isolation
- Data freshness
What this unlocks with other layers
Research-to-Knowledge
Discovery becomes a retained, structured research corpus instead of disappearing after one chat.
Technology behind this capability
Web searchVector storeGraph databaseModel Context Protocol (MCP)Web crawlerCrawler MCP serverAICORTEX native