Ivannikov Institute for System Programming of the RAS


Extracting Key Terms From Noisy and Multitheme Documents.

Authors

Grineva M., Grinev M., Lizorkin D.

Abstract

We present a novel method for key term extraction from text documents. In our method, document is modeled as a graph of semantic relationships between terms of that document. We exploit the following remarkable feature of the graph: the terms related to the main topics of the document tend to bunch up into densely interconnected subgraphs or communities, while non-important terms fall into weakly interconnected communities, or even become isolated vertices. We apply graph community detection techniques to partition the graph into thematically cohesive groups of terms. We introduce a criterion function to select groups that contain key terms discarding groups with unimportant terms. To weight terms and determine semantic relatedness between them we exploit information extracted from Wikipedia.

Using such an approach gives us the following two advantages. First, it allows effectively processing multi-theme documents. Second, it is good at filtering out noise information in the document, such as, for example, navigational bars or headers in web pages.

Evaluations of the method show that it outperforms existing methods producing key terms with higher precision and recall. Additional experiments on web pages prove that our method appears to be substantially more effective on noisy and multi-theme documents than existing methods.

Full text of the paper in pdf

Keywords

Wikipedia, community detection, graph analysis, keywords extraction, semantic similarity

Edition

WWW2009: 18th International World Wide Web Conference.

Research Group

Information Systems

All publications during 2009 All publications