Method postWebPages

solr/core/src/java/org/apache/solr/cli/PostTool.java:587–601

This method takes as input a list of start URL strings for crawling, converts the URL strings to URI strings and adds each one to a backlog and then starts crawling @param args the raw input args from main() @param startIndexInArgs offset for where to start @param out outputStream to write results

(String[] args, int startIndexInArgs, OutputStream out)

   * @return the number of web pages posted
   */
  public int postWebPages(String[] args, int startIndexInArgs, OutputStream out) {
    reset();
    LinkedHashSet<URI> s = new LinkedHashSet<>();
    for (int j = startIndexInArgs; j < args.length; j++) {
      try {
        URI uri = new URI(normalizeUrlEnding(args[j]));
        s.add(uri);
      } catch (URISyntaxException e) {
        warn("Skipping malformed input URL: " + args[j]);
      }
    }
    // Add URIs to level 0 of the backlog and start recursive crawling
    backlog.add(s);
    return webCrawl(0, out);
  }

  /**
   * Normalizes a URL string by removing anchor part and trailing slash