MCPcopy Create free account
hub / github.com/OpenRaiser/PaperFlow / benchmark_one_day

Function benchmark_one_day

experiments/token_cost/token_usage_tracker.py:222–335  ·  view source on GitHub ↗

基准测试:模拟一天的完整流程,统计 token 消耗 流程: 1. Embedding 所有论文 2. 对每个用户进行排序 3. 生成推送内容 4. 模拟用户反馈后生成精读报告

(
    date: str,
    llm_model: str,
    papers: List[Dict],
    user_profiles: List[Dict],
)

Source from the content-addressed store, hash-verified

220
221def benchmark_one_day(
222 date: str,
223 llm_model: str,
224 papers: List[Dict],
225 user_profiles: List[Dict],
226) -> Dict[str, Any]:
227 """
228 基准测试:模拟一天的完整流程,统计 token 消耗
229
230 流程:
231 1. Embedding 所有论文
232 2. 对每个用户进行排序
233 3. 生成推送内容
234 4. 模拟用户反馈后生成精读报告
235 """
236 result = {
237 "date": date,
238 "llm_model": llm_model,
239 "embedding_model": os.environ.get("EMBEDDING_MODEL", "Qwen/Qwen3-Embedding-8B"),
240 "paper_count": len(papers),
241 "user_count": len(user_profiles),
242 "breakdown": {},
243 "total": {
244 "embedding_tokens": 0,
245 "llm_tokens": 0,
246 "estimated_cost_usd": 0,
247 }
248 }
249
250 # 1. Embedding 所有论文
251 embedding_tokens = 0
252 for paper in papers:
253 text = f"{paper.get('title', '')} {paper.get('abstract', '')}"
254 tokens = estimate_tokens(text)
255 embedding_tokens += tokens
256
257 # 记录
258 log_token_usage(
259 task_type="embedding",
260 model=result["embedding_model"],
261 input_tokens=tokens,
262 output_tokens=0,
263 date=date,
264 )
265
266 result["breakdown"]["embedding"] = {
267 "papers": len(papers),
268 "tokens_per_paper": embedding_tokens // max(1, len(papers)),
269 "total_tokens": embedding_tokens,
270 }
271 result["total"]["embedding_tokens"] = embedding_tokens
272
273 # 2. LLM 排序(对每个用户)
274 llm_ranking_tokens = 0
275 for profile in user_profiles:
276 # 每个用户排序:输入 = 论文摘要 + 用户画像,输出 = 排序结果
277 user_text = json.dumps(profile.get("core_directions", {}), ensure_ascii=False)
278 paper_text = " ".join([f"{p.get('title')} {p.get('abstract')}" for p in papers[:20]]) # 假设每次推 20 篇
279

Callers 2

compare_llm_modelsFunction · 0.85
mainFunction · 0.85

Calls 3

estimate_tokensFunction · 0.85
log_token_usageFunction · 0.85
getMethod · 0.80

Tested by

no test coverage detected