Enhancement of a natural language processing(NLP) based search engine

Abstract

Search engine is also known as information retrieval. A Web search engine is a tool designed to search for information on the World Wide Web. As the popularization of internet, search engine has already played an indispensable role in people’s everyday life. “If you don’t know, just Google it.” has become conventional words. However, traditional search engines, such as Google, provide either simple string matching or statistical processing of text for its search items. Because of language ambiguity, search engine users are often shown with a bunch of irrelevant documents and have to spend a lot of time to filter away those links and get links relevant to their query.

To improve this, NLP (Natural Language Processing) technique can be utilized as a design tool. This project’s objective is to develop a powerful and accurate search input parser based on NLP techniques. By analyzing and parsing the search input, ambiguity is reduced, which in response makes the search results more accurate.

Two kinds of parsers are implemented, which are Finite State Machine (FSM) Parser and Phrase Structure Grammar (PSG) Parser. FSM parser implementation is grounded on the assumption that the language is finite. PSG parser implementation involves in three phases. The first phase covers the basic design and skeleton, applying grammar rule. The second phase depicts factors such as unknown words, agreement etc. The third phase is related to interrogative sentence structure, verb category and final improvement on the parser. Besides, a parser user interface is designed to perform user-friendly implementation of the parser.