Sci tech web crawler
| dc.contributor.author | Ansari, Mukhtar | |
| dc.contributor.author | Wangde, Amin [21CO62] | |
| dc.contributor.author | Shaikh, Ashraf [22CO44] | |
| dc.contributor.author | Mohammed Ali Sahabuddin [22CO50] | |
| dc.contributor.author | Khan, Mohd. Afzal [20CO30] | |
| dc.date.accessioned | 2026-08-20T09:39:43Z | |
| dc.date.available | 2026-08-20T09:39:43Z | |
| dc.date.issued | 2026-06 | |
| dc.description.abstract | The Sci-Tech Web Crawler project is designed to build a domain-specific search engine focused exclusively on science and technology content. Unlike general-purpose crawlers that index a wide range of web pages based on popularity or backlinks, this system prioritizes relevance, credibility, and academic quality. It gathers and analyzes information from authoritative platforms such as IEEE, arXiv, and other verified research repositories to ensure that users—students, researchers, and professionals—gain access to accurate and topic-specific data. The primary objective of this work is to create a precise, intelligent, and automated tool that minimizes irrelevant or promotional noise typically present in mainstream search engines. To achieve this goal, the crawler leverages Python-based technologies including BeautifulSoup, Requests, Scikit-learn (TF-IDF), NLTK, and Hugging Face Transformers. The methodology involves fetching HTML content from user-specified URLs, extracting and cleaning text, computing TF-IDF scores to identify high-value keywords, and generating concise summaries using stateof- the-art language models. The processed information is stored efficiently in MongoDB, forming a structured knowledge base that can be queried and expanded dynamically. Additionally, a separate module categorizes hyperlinks by thematic relevance—such as Artificial Intelligence, Machine Learning, or Quantum Computing—enabling a more organized and domain-aware exploration of scientific resources. Through this implementation, the project successfully demonstrates how artificial intelligence and natural language processing techniques can be combined to build an intelligent, research-oriented web crawler. The system not only achieves its objective of focused data extraction but also offers scalability for future integration with real-time academic databases and interactive search interfaces. Overall, Sci-Tech Web Crawler contributes meaningfully toward improving the accessibility, accuracy, and efficiency of scientific information retrieval in the modern digital era. | |
| dc.identifier.uri | http://aiktc.ndl.gov.in/handle/123456789/4571 | |
| dc.language.iso | en_US | |
| dc.publisher | AIKTC | |
| dc.title | Sci tech web crawler | |
| dc.type | Project Report |