Multi-agent crawling systems (MACS) architecture for effective web retrieval

Abstract

Recently, many web search engines used for information gathering in World Wide Web (WWW). For instance, Google, Yahoo, AltaVista and others. Web crawler is a program or automated script which browses the WWW in a methodically, automated manner that mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. From the study, we found that web pages crawled by crawlers will slow down the server. Thus, it makes users refuses to allow crawlers exploring web pages and even worst if they block the crawler’s IP address during entering the web pages. In order to achieve higher accuracy rate, we propose the architecture of multi-agent system in web crawling known as Multi-Agent Crawling System (MACS). Since Java Agent Development Framework (JADE) is one of the most used and promising agent development framework, MACS will be model in Java based on JADE architecture. We expected this model will enhance the network interaction between the web agents and servers.