Package org.jweaver.crawler.internal.runner

Class JWeaverBuilderImpl

java.lang.Object

org.jweaver.crawler.internal.runner.JWeaverBuilderImpl

All Implemented Interfaces:

JWeaverCrawler.Builder

public final class JWeaverBuilderImpl
extends Object
implements JWeaverCrawler.Builder

A concrete implementation of the JWeaverCrawler.Builder interface used to configure and build instances of JWeaverCrawler.

Constructor Summary

Constructors

Constructor	Description
JWeaverBuilderImpl()	Constructs a new JWeaverBuilderImpl instance.

Method Summary

Modifier and Type	Method	Description
JWeaverCrawler	build(Set<String> uriSet)	Builds and returns a new instance of JWeaverCrawler with the configured parameters.
JWeaverCrawler.Builder	exportConfiguration(ExportConfig exportConfiguration)	Sets the export configuration for configuring data export options.
JWeaverCrawler.Builder	httpClient(HttpClient httpClient)	Sets the HTTP client to be used by the crawler for making HTTP requests.
JWeaverCrawler.Builder	maxDepth(int maxDepth)	Sets the maximum depth of crawling .
JWeaverCrawler.Builder	parser(DocumentParser documentParser)	Sets the document parser to be used by the crawler for parsing relevant information from HTML body.
JWeaverCrawler.Builder	politenessDelay(Duration politenessDelay)	Sets the politeness delay between consecutive requests made by the crawler to the same host
JWeaverCrawler.Builder	writer(JWeaverWriter writer)	Sets the writer for exporting the crawled data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

JWeaverBuilderImpl

public JWeaverBuilderImpl()

Constructs a new JWeaverBuilderImpl instance.

Method Details

httpClient

public JWeaverCrawler.Builder httpClient(HttpClient httpClient)

Description copied from interface: JWeaverCrawler.Builder

Sets the HTTP client to be used by the crawler for making HTTP requests.

Default Will use a default HttpClient if not provided with connection timeout 5 seconds and follow redirects policy to AlWAYS

Specified by:

httpClient in interface JWeaverCrawler.Builder

Parameters:

httpClient - The HTTP client to be used. (Optional)

Returns:

This builder instance.

parser

public JWeaverCrawler.Builder parser(DocumentParser documentParser)

Description copied from interface: JWeaverCrawler.Builder

Sets the document parser to be used by the crawler for parsing relevant information from HTML body.

Default Will use the JWeaverDocumentParser if not provided.

Specified by:

parser in interface JWeaverCrawler.Builder

Parameters:

documentParser - The document parser to be used. (Optional)

Returns:

This builder instance.

writer

public JWeaverCrawler.Builder writer(JWeaverWriter writer)

Description copied from interface: JWeaverCrawler.Builder

Sets the writer for exporting the crawled data.

Default Will use a JWeaverFileWriter if not provided

Specified by:

writer in interface JWeaverCrawler.Builder

Parameters:

writer - The writer for exporting data. (Optional)

Returns:

This builder instance for method chaining.

exportConfiguration

public JWeaverCrawler.Builder exportConfiguration(ExportConfig exportConfiguration)

Description copied from interface: JWeaverCrawler.Builder

Sets the export configuration for configuring data export options.

Default Will use the default ExportConfig.exportDefault() which uses a Markdown format and the 'output'/ path for files.

Specified by:

exportConfiguration in interface JWeaverCrawler.Builder

Parameters:

exportConfiguration - The export configuration. (Optional)

Returns:

This builder instance for method chaining.

politenessDelay

public JWeaverCrawler.Builder politenessDelay(Duration politenessDelay)

Description copied from interface: JWeaverCrawler.Builder

Sets the politeness delay between consecutive requests made by the crawler to the same host

Default 3 seconds.

Specified by:

politenessDelay in interface JWeaverCrawler.Builder

Parameters:

politenessDelay - The politeness delay duration. (Optional)

Returns:

This builder instance for method chaining.

maxDepth

public JWeaverCrawler.Builder maxDepth(int maxDepth)

Description copied from interface: JWeaverCrawler.Builder

Sets the maximum depth of crawling .

Default 3

Specified by:

maxDepth in interface JWeaverCrawler.Builder

Parameters:

maxDepth - The maximum depth of crawling. (Optional)

Returns:

This builder instance for method chaining.

build

public JWeaverCrawler build(Set<String> uriSet)

Description copied from interface: JWeaverCrawler.Builder

Builds and returns a new instance of JWeaverCrawler with the configured parameters.

Specified by:

build in interface JWeaverCrawler.Builder

Parameters:

uriSet - The required initial set of URIs to start crawling from. Each URI should be from a different host. For each URI, a new JWeaverTask (execution) will be created.

Returns:

A new instance of JWeaverCrawler.