Abstract
The term summary, of a statement or of an account, comprises the chief points or the sum and substance of the matter. Moreover, a summary is expected to be comprehensive yet brief and readable. Therefore, there is a necessity to verify how close a summary is to the chief or key points in the source text; and verification is closely related to evaluation. This forms the main focus of this thesis: can a summary be evaluated by computers? In other words, we have developed an automatic evaluation procedure based on a metric which could provide summary evaluation without human assistance. The metrics used in the evaluation of summaries are discussed and a novel framework, which includes two metrics, for summary evaluation, is presented. The first metric is based on a known and powerful statistical test, the χ2 goodness-of-fit test, and has been used in several applications. The second metric is derived from three common metrics used to evaluate NLP systems, namely precision, recall and f-measure. The combination of these two metrics is intended to allow one to assess the quality of summaries quickly, cheaply and without the need of human intervention, minimising though, the role of subjective judgment and bias.