Building a Semantic Web Search Engine: Challenges and Solutions

Current Web search engines allow users to specify keywords queries and return links to documents. Users have to manually trawl through lists of links and glean the required information from documents. In contrast, semantic search engines allow more expressive queries over information integrated from multiple sources, and return specific information about entities, for example people, locations, news items, or proteins. An entity-centric data model furthermore permits powerful query and browsing techniques such as faceted navigation.

There are a number of challenges in implementing such a system:

• The architecture of a semantic search engine must scale to the Web.
• Dealing with data rather than documents requires a different indexing approach compared to traditional information retrieval systems.
• Data from the Web is messy, which poses challenges for data cleansing and entity consolidation.
• The schema of the data is unknown a priori, which makes building generic user interfaces difficult.

In this presentation, I will give an overview of the architecture of SWSE, a Semantic Web Search Engine that scales to billions of RDF statements, and discuss in detail the necessary adaptations to traditional search engine components, such as crawling, indexing, query processing, ranking, and user interfaces.

With the majority of scaling challenges solved, open questions remain involving trustworthiness of data used in reasoning, and user interaction models over graph-structured data collected from the Web.