Open Source Spotlight Setting Up a Web Harvesting Pipeline with Heritrix

Heritrix remains a workhorse for large scale web harvesting. This spotlight covers installing configuring and operating Heritrix as part of a harvest pipeline including tips for deduplication scheduling and monitoring to keep the system healthy.

Installation and prerequisites

Heritrix runs on Java so ensure you have a compatible JRE installed. Use a dedicated host with sufficient disk and CPU resources for your planned harvest. For production use consider deploying on a cluster and separating storage and job orchestration components.

Basic configuration

Create a seed list that defines starting URLs and limit policies to scope the crawl. Configure the crawl order and politeness policies to avoid overloading remote sites. Heritrix provides a web based UI for job creation but job definitions can be scripted for reproducibility.

Deduplication and storage optimization

Enable deduplication at the content level to avoid storing multiple copies of identical large assets. Use a signature based approach combining checksums and byte range fingerprinting. Offload large binary assets to object storage and maintain metadata pointers in your index.

Scheduling and job management

Use job queues and cron workflows to schedule different collections with appropriate frequencies. For high priority sources set daily crawls while lower priority domains can be captured monthly or quarterly. Keep a record of job parameters to ensure captures are reproducible.

Monitoring and alerting

Set up monitoring for queue lengths job failures and resource utilization. Capture logs centrally and configure alerts for crawl stalls and storage thresholds. Periodic replay tests help ensure captures are complete and usable.

Integration with indexing and discovery

Export capture metadata to a search index to provide faceted discovery. Consider a pipeline that converts WARCs to searchable text extracts and links extracted entities to authority files for improved researcher workflow.

Community tools and extensions

Heritrix integrates with tools like WCT for quality assessment and Brozzler for improved dynamic content capture. Use community developed modules to extend functionality rather than forking core code for maintainability.

Operational tips

• Start with small pilot crawls to validate seeds and politeness rules
• Automate export of WARCs and metadata to offsite backups
• Document capture policies and retention schedules for stakeholders

Conclusion

Heritrix is a mature tool capable of supporting institution scale harvesting when paired with operational practices for monitoring and storage. With careful planning it can deliver reliable captures as part of a broader preservation strategy.

Author: Alex Chen