Optimal Allocation Of Heterogeneous Robots In World Wide Web Search Engines

It is a conflicting requirement to reduce network traffic and loads on Web servers caused by the robots in a World Wide Web search engine and to increase the currency of the database in the search engine. This paper addresses the problem of finding optimal allocations of a given number of robots in a search engine to heterogeneous Internet domains such that a cost function which characterizes the combined consideration of network/server loads and database currency can be minimized. The cost function and the optimization problem are formulated based on a queueing model of search engines. Our algorithm to solve the problem uses the depth-first search strategy to traverse an enumeration tree which generates all possible allocations.

Keywords:

• Depth-first Search
• Enumeration Tree
• Indexing Engine
• Internet
• Queueing System
• Robot
• Search Engine
• World Wide Web