Wait for decontextualization to complete

Log a warning message asynchronously.

Write a message to the SSE stream in a format compatible with existing handlers. Args: message: Message dictionary to se

Aggregate results from multiple endpoints, merging JSON data for duplicate URLs. When the same URL appears in multiple endpoints, th

For templates with dimension_select_map, update the query's SELECT.

Construct the system and user message sequence enforcing a JSON schema.

Construct the message sequence for JSON-schema enforcement.

Construct the system and user message sequence enforcing a JSON schema.

Check if the retrieved recipes need substitutions based on user requirements.

Remove circular references and clean object for JSON serialization.

Collect result items and the message ids they originated from.

Compute and normalize rubric features for a batch of items.

Convert a single Bing search result to NLWeb format. Args: bing_result: Single result from Bing API site: Si

Extract client parameters from endpoint config.

Extract client parameters from endpoint config.

Extract client parameters from endpoint config.

Create the conversation index with appropriate fields.

Create and return an index definition with specified embedding size

Create the Elasticsearch index with proper vector mapping if it doesn't exist. Args: index_name: Optional index name (de

Create a reverse mapping from DCIDs to human-readable names.

Create a Qdrant filter for site filtering. Args: site: Site or list of sites to filter by Returns:

Extract embedding properties from endpoint config.

Create and return a vector search configuration

Dispatch log message to the appropriate logger method based on level

Process all remaining items in the queue

Return value as list, filtering out falsy entries.

Resolve log directory path (but don't create it yet)

Ensure the required packages for an LLM type are installed. Args: llm_type: The type of LLM provider

Evaluate if an item matches the requested item.

Evaluate if an item matches the requested item. If the score is above 75, also extract the details that the user is looking for.

Evaluate a single tool for the query.

Evaluate tools asynchronously with early termination for high-scoring results. Args: query: The query to evaluate too

Execute retrieval and ranking in parallel for all queries.

Execute retrieval steps. Returns (step_results, sql_queries).

Return the answer payload (if any) from a message.

Extract domain from URL for site identification.

Pull result items from legacy payload structures.

Extract product information from title and snippet text. Args: title: Page title from Bing result snippet: S

Extract just the query text from previous queries. Handles both simple string lists and complex nested structures.

Recursively extract @type values from schema objects.

Fetch a single page and return HTML content and size.

Find the config file by checking multiple possible locations

Find items that match the requested item using parallel LLM calls.

Force all handlers to flush their buffers.

Format Qdrant search results to match expected API: [url, text_json, name, site]. Args: search_result: Qdrant search res

Generate ensemble recommendations using LLM.

Generate finished pottery description with explicit negation.

Generate finished pottery description with contextual phrasing.

Generate finished pottery description with category prefixing.

Generate substitution suggestions using LLM.

Generate supply store description with explicit negation.

Generate supply store description with contextual phrasing.

Generate supply store description with category prefixing.

Get or create the global async log processor

Get the base directory for all output files from the environment variable. Returns None if the environment variable is not set.

Get sites list with caching and background refresh. Uses stale-while-revalidate pattern for better performance. Returns:

Get or create a Cloudflare SDK client. Returns: Cloudflare instance

Get the configuration directory from environment variable or use default. Default is NLWeb/config relative to the code directory.

Parse the PostgreSQL connection string and return a dictionary of configuration parameters. Args: connection_string: Pos

Get or create the connection pool for PostgreSQL. Connection pooling is used for better performance and resource management.

Get default configuration

Get the PostgreSQL endpoint configuration from CONFIG Returns: Tuple of (RetrievalProviderConfig)

Get the Elasticsearch endpoint configuration from CONFIG

Get the OpenSearch endpoint configuration from CONFIG

Get the Qdrant endpoint configuration from CONFIG

Get the Cloudflare SDK endpoint configuration from CONFIG

Get the Milvus endpoint configuration from CONFIG

Get the Azure Search endpoint configuration from CONFIG

Get the Bing Search endpoint configuration from CONFIG

Get item details using URL-based retrieval.

Get or create the NLWebScorer instance (lazy-loaded on first use).

Handle non-streaming response from NLWebServer. Args: session: The aiohttp session endpoint: The endpoint URL params

Handle streaming (SSE) response from NLWebServer. Args: session: The aiohttp session endpoint: The endpoint URL para

Check if HTML file for URL already exists.

Infer a numeric property for SUM/AVG based on alias and type.

Initialize the Elasticsearch client

Join results from two steps by a common key. Returns dict: {entity_key: {"entity": left_row, "associated": [right_rows]}}

Use LLM to map unknown NL values to ontology filter specs.

Load configuration from YAML file

Load configuration from YAML file

Load DCID mappings from the JSON file.

Load crawl status from file or create new one.

Load query templates from the statistics_templates.txt file.