Projects Web Crawler

Web Crawler

A custom-built, politeness-compliant web crawler developed for a UCI Information Retrieval course project, built from scratch in Python with multi-threaded crawling and URL frontier management.

University Project Information Retrieval

Tech Stack

  • Python
  • Multi-threading
  • URL Frontier Management
Web Crawler dashboard showing crawl map, live activity log, crawl metrics, politeness settings, and multi-threaded worker status

Overview

Built as part of a UCI Information Retrieval course project: a custom web crawler built from scratch in Python (not using existing frameworks like Scrapy), designed to respect per-domain politeness constraints, avoid duplicate fetches, and coordinate multi-threaded workers through a shared URL frontier.

The crawler collected and structured web pages for downstream indexing and retrieval experiments — a team submission focused on correct concurrent crawling behavior rather than a production-scale deployment.