MCPcopy Create free account
hub / github.com/OpenRaiser/PaperFlow / benchmark_one_day

Function benchmark_one_day

experiments/token_cost/token_usage_tracker.py:222–335  ·  view source on GitHub ↗

基准测试:模拟一天的完整流程,统计 token 消耗 流程: 1. Embedding 所有论文 2. 对每个用户进行排序 3. 生成推送内容 4. 模拟用户反馈后生成精读报告

(
    date: str,
    llm_model: str,
    papers: List[Dict],
    user_profiles: List[Dict],
)

Source from the content-addressed store, hash-verified

220
221
222def benchmark_one_day(
223 date: str,
224 llm_model: str,
225 papers: List[Dict],
226 user_profiles: List[Dict],
227) -> Dict[str, Any]:
228 """
229 基准测试:模拟一天的完整流程,统计 token 消耗
230
231 流程:
232 1. Embedding 所有论文
233 2. 对每个用户进行排序
234 3. 生成推送内容
235 4. 模拟用户反馈后生成精读报告
236 """
237 result = {
238 "date": date,
239 "llm_model": llm_model,
240 "embedding_model": os.environ.get("EMBEDDING_MODEL", "Qwen/Qwen3-Embedding-8B"),
241 "paper_count": len(papers),
242 "user_count": len(user_profiles),
243 "breakdown": {},
244 "total": {
245 "embedding_tokens": 0,
246 "llm_tokens": 0,
247 "estimated_cost_usd": 0,
248 }
249 }
250
251 # 1. Embedding 所有论文
252 embedding_tokens = 0
253 for paper in papers:
254 text = f"{paper.get('title', '')} {paper.get('abstract', '')}"
255 tokens = estimate_tokens(text)
256 embedding_tokens += tokens
257
258 # 记录
259 log_token_usage(
260 task_type="embedding",
261 model=result["embedding_model"],
262 input_tokens=tokens,
263 output_tokens=0,
264 date=date,
265 )
266
267 result["breakdown"]["embedding"] = {
268 "papers": len(papers),
269 "tokens_per_paper": embedding_tokens // max(1, len(papers)),
270 "total_tokens": embedding_tokens,
271 }
272 result["total"]["embedding_tokens"] = embedding_tokens
273
274 # 2. LLM 排序(对每个用户)
275 llm_ranking_tokens = 0
276 for profile in user_profiles:
277 # 每个用户排序:输入 = 论文摘要 + 用户画像,输出 = 排序结果
278 user_text = json.dumps(profile.get("core_directions", {}), ensure_ascii=False)
279 paper_text = " ".join([f"{p.get('title')} {p.get('abstract')}" for p in papers[:20]]) # 假设每次推 20 篇

Callers 2

compare_llm_modelsFunction · 0.85
mainFunction · 0.85

Calls 3

estimate_tokensFunction · 0.85
log_token_usageFunction · 0.85
getMethod · 0.80

Tested by

no test coverage detected