SAMA: a real-time Web search architecture

ABSTRACT

With the increasing number of information resources in the World Wide Web, how to retrieve real-time information is becoming very important. The resources include traditional databases, flat files, knowledge bases, topical content and programs. In this paper, we propose a new model for retrieving high-quality data from distributed Web resources over the Internet. Unlike traditional models which are web crawler-based algorithms and collect data prior to indexing, supposedly making the process going offline, the proposed algorithm embeds the crawling process with the searching process to make the process more realistic since data are collected and indexed in real-time fashion. We aim to get high-quality and refreshed results for every user’s query, as well as, for better query expansion and formulation. In our experiments, we found that the proposed approach was highly relevant for most recently searched queries, consisting in its performance and resilient to the drawbacks faced by other techniques.

KEYWORDS:

• Adaptive indexing
• real-time algorithm
• federated search
• real world entity

Acknowledgements

This research does not receive a grant from any funding agency in the public, commercial, or not-for-profit sectors. The author would like to thank Prof. Diana Inkpen at the University of Ottawa for the generous contribution she had made to extend his testing server.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 http://trec.nist.gov/tracks.html

2 https://lemurproject.org/clueweb09.php/

3 http://trec.nist.gov/pubs/trec22/papers/FEDERATED.OVERVIEW.pdf

4 The training data are available freely for research development.

5 The goal of this track is to run a retrieval task similar to standard ad hoc retrieval, but to evaluate large numbers of queries incompletely, rather than a small number more completely. Participants run 10,000 queries and random of 1,000 or so were evaluated.

Additional information

Notes on contributors

Falah Al-akashi

Falah Al-akashi was completed a PhD in Computer Science at the University of Ottawa since 2014. Currently, he is a professor at the Faculty of Engineering, University of Kufa. He is experienced in Web Search Engines, Data mining, NLP, Information Retrieval, and text analysis, with excellent communication and interpersonal skills honed through 6 years of as a PhD student, teaching and research assistant.