Unified conversational agent that acts like Cursor for web scraping. Maintains context, learns from patterns, and creates reusable pipelines.
| 41 | |
| 42 | |
| 43 | class UnifiedScrapingAgent: |
| 44 | """ |
| 45 | Unified conversational agent that acts like Cursor for web scraping. |
| 46 | Maintains context, learns from patterns, and creates reusable pipelines. |
| 47 | """ |
| 48 | |
| 49 | def __init__(self): |
| 50 | self.llm = get_llm() |
| 51 | self.scraping_service = ScrapingService(settings.SCRAPEGRAPH_API_KEY) |
| 52 | self.redis_client: Optional[redis.Redis] = None |
| 53 | self.conversation_ttl = 86400 * 7 # 7 days |
| 54 | |
| 55 | # Initialize services (will be imported separately) |
| 56 | self.pipeline_repo = None # Will be initialized with PipelineRepository |
| 57 | self.pattern_learner = None # Will be initialized with PatternLearner |
| 58 | |
| 59 | self.system_prompt = self._build_system_prompt() |
| 60 | |
| 61 | async def initialize(self): |
| 62 | """Initialize async components.""" |
| 63 | # Connect to Redis for conversation memory |
| 64 | self.redis_client = await redis.from_url( |
| 65 | settings.REDIS_URL, |
| 66 | encoding="utf-8", |
| 67 | decode_responses=True |
| 68 | ) |
| 69 | |
| 70 | # Initialize repository and learning services |
| 71 | from app.services.pipeline_repository import PipelineRepository |
| 72 | from app.services.pattern_learner import PatternLearner |
| 73 | |
| 74 | self.pipeline_repo = PipelineRepository() |
| 75 | await self.pipeline_repo.initialize() |
| 76 | |
| 77 | self.pattern_learner = PatternLearner() |
| 78 | await self.pattern_learner.initialize() |
| 79 | |
| 80 | def _build_system_prompt(self) -> str: |
| 81 | """Build the system prompt for the agent.""" |
| 82 | return """You are ScrapeCraft AI, an intelligent conversational assistant for web scraping, similar to how Cursor works for coding. |
| 83 | |
| 84 | Your capabilities: |
| 85 | 1. **Conversational Understanding**: Maintain context across conversations and learn from interactions |
| 86 | 2. **Pipeline Creation**: Help users build reusable scraping pipelines with ScrapeGraphAI |
| 87 | 3. **Pattern Recognition**: Identify and suggest optimizations based on successful patterns |
| 88 | 4. **Code Generation**: Generate production-ready Python code using async ScrapeGraphAI client |
| 89 | 5. **Pipeline Reuse**: Find and adapt existing pipelines for new use cases |
| 90 | |
| 91 | Key behaviors: |
| 92 | - Be conversational and helpful, like a pair programmer |
| 93 | - Proactively suggest improvements and optimizations |
| 94 | - Learn from each interaction to improve future suggestions |
| 95 | - Remember context across sessions |
| 96 | - Offer to save successful pipelines for reuse |
| 97 | |
| 98 | When generating code, ALWAYS use the async ScrapeGraphAI pattern with prompt-based extraction: |
| 99 | ```python |
| 100 | import asyncio |