任务指标

摘要评分

摘要评分指标衡量摘要（响应）从参考上下文中捕获重要信息的程度。该指标的核心思想是：一个好的摘要应该包含上下文中存在的所有重要信息。

我们首先从上下文中提取一组重要的关键词组。然后，这些关键词组被用于生成一组问题。对于上下文来说，这些问题的答案始终是"是"(1)。接着，我们向摘要询问这些问题，并将摘要评分计算为正确回答的问题数与问题总数的比率。

我们使用答案（由1和0组成的列表）来计算问答分数。问答分数计算为正确回答的问题数（答案=1）与问题总数的比率。

我们还引入了一个选项，通过提供简洁性评分来对较长的摘要进行惩罚。如果启用此选项，最终分数将计算为摘要评分和简洁性评分的加权平均值。此简洁性评分确保只是文本副本的摘要不会获得高分，因为它们显然会正确回答所有问题。

我们还提供一个系数 coeff （默认值为 0.5）来控制分数的权重。

最终摘要评分的计算公式为：

示例

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import SummaryScore

# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

# Create metric
scorer = SummaryScore(llm=llm)

# Evaluate
result = await scorer.ascore(
    reference_contexts=[
        "A company is launching a new product, a smartphone app designed to help users track their fitness goals. The app allows users to set daily exercise targets, log their meals, and track their water intake. It also provides personalized workout recommendations and sends motivational reminders throughout the day."
    ],
    response="A company is launching a fitness tracking app that helps users set exercise goals, log meals, and track water intake, with personalized workout suggestions and motivational reminders."
)
print(f"Summary Score: {result.value}")

输出：

Summary Score: 0.6423387096775146

同步用法
如果您偏好同步代码，可以使用 .score() 方法替代 .ascore() ：

result = scorer.score(
    reference_contexts=[...],
    response="..."
)

旧版指标 API

以下示例使用旧版指标 API 模式。对于新项目，我们建议使用上面显示的基于集合的 API。

弃用时间线
此 API 将在 0.4 版本中弃用，并在 1.0 版本中移除。请迁移到上面显示的基于集合的 API。

使用 SingleTurnSample 的示例

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import SummarizationScore


sample = SingleTurnSample(
    response="A company is launching a fitness tracking app that helps users set exercise goals, log meals, and track water intake, with personalized workout suggestions and motivational reminders.",
    reference_contexts=[
        "A company is launching a new product, a smartphone app designed to help users track their fitness goals. The app allows users to set daily exercise targets, log their meals, and track their water intake. It also provides personalized workout recommendations and sends motivational reminders throughout the day."
    ]
)

scorer = SummarizationScore(llm=evaluator_llm)
await scorer.single_turn_ascore(sample)

输出：

0.6423387096775146