Sci tech web crawler
No Thumbnail Available
Date
2026-06
Journal Title
Journal ISSN
Volume Title
Publisher
AIKTC
Abstract
The Sci-Tech Web Crawler project is designed to build a domain-specific
search engine focused exclusively on science and technology content. Unlike
general-purpose crawlers that index a wide range of web pages based on popularity
or backlinks, this system prioritizes relevance, credibility, and academic quality.
It gathers and analyzes information from authoritative platforms such as
IEEE, arXiv, and other verified research repositories to ensure that users—students,
researchers, and professionals—gain access to accurate and topic-specific data.
The primary objective of this work is to create a precise, intelligent, and automated
tool that minimizes irrelevant or promotional noise typically present in
mainstream search engines.
To achieve this goal, the crawler leverages Python-based technologies including
BeautifulSoup, Requests, Scikit-learn (TF-IDF), NLTK, and Hugging
Face Transformers. The methodology involves fetching HTML content from
user-specified URLs, extracting and cleaning text, computing TF-IDF scores
to identify high-value keywords, and generating concise summaries using stateof-
the-art language models. The processed information is stored efficiently in
MongoDB, forming a structured knowledge base that can be queried and expanded
dynamically. Additionally, a separate module categorizes hyperlinks by
thematic relevance—such as Artificial Intelligence, Machine Learning, or Quantum
Computing—enabling a more organized and domain-aware exploration of
scientific resources.
Through this implementation, the project successfully demonstrates how artificial
intelligence and natural language processing techniques can be combined
to build an intelligent, research-oriented web crawler. The system not
only achieves its objective of focused data extraction but also offers scalability
for future integration with real-time academic databases and interactive search
interfaces. Overall, Sci-Tech Web Crawler contributes meaningfully toward
improving the accessibility, accuracy, and efficiency of scientific information retrieval
in the modern digital era.