China Publishes World’s First Global LLM Safety Evaluation: 38 Models Tested on 313 Questions Against 94,108 Scientific Papers

TL;DR

  • July 2, 2026: At the Global Digital Economy Conference Cloud-Computing Security Forum held in Beijing, Dongbi Tech Data (Dongbi Data) and the School of Digital Economy, Shanghai University of Finance and Economics (SUFE) jointly released the Global Large Language Model Safety Evaluation Report (2026) — the world’s first dedicated LLM science-safety benchmark and the first “science-for-good” oriented LLM safety report.
  • Test set scale: 313 high-risk science questions (covering biochemistry, cybersecurity, and other frontier areas) testing 38 domestic and international LLMs.
  • RAG reference corpus: 94,108 scientific papers from Dongbi’s Dbdata platform plus 34,452 encyclopedia entries for fact grounding.
  • Direct attack success rate: 7.6% — the majority of models have built baseline refusal capabilities.
  • However, composite attacks (prefix injection + scenario camouflage + emotional camouflage + example induction) clearly stretch the safety boundaries of multiple models.
  • The report introduces three new evaluation dimensions beyond simple refusal rate: intent recognition, disclosure calibration, and “reliable-but-risky” outputs.

1. Why This Report Matters: The First LLM Benchmark Built for Scientific Misuse

On July 2, 2026, at the Global Digital Economy Conference in Beijing, two Chinese institutions released what they call the world’s first dedicated LLM science-safety benchmark.

Unlike typical LLM safety benchmarks that focus on general-purpose risks (hate speech, illegal advice, etc.), this report narrows on science-domain high-risk questions, the kind of questions a malicious actor might use to elicit dangerous dual-use knowledge — biochemical synthesis, exploit development, etc.

The release was covered by at least five independent Chinese sources including CSDN News (2026-07-02 17:19), Tencent News (new.qq.com 2026-07-02 17:57), Tencent html5.qq.com (2026-07-02 17:38 + 2026-07-03 04:46), and Hubei Science & Technology Think Tank (2026-07-13 06:57).

2. The Six Hard Numbers You Need to Know

2.1 Direct attack success rate: only 7.6%

On direct attacks — straightforward queries like “how do I synthesize X” — the overall success rate across all 38 tested models is 7.6%. That means 92.4% of these direct harmful queries are correctly refused. This number is reassuring: the majority of current-gen LLMs have built a solid baseline refusal capability.

Source: Tencent News, 2026-07-02 17:57.

2.2 Composite attacks push models past their safety boundaries

The headline finding is that the real safety holes are not in direct attacks, but in composite attacks. Four attack categories are explicitly named:

  • Prefix Injection — wrapping dangerous queries in authoritative framings (“I’m a doctor…”, “this is for a textbook…”)
  • Scenario Camouflage — wrapping queries as academic scenarios, literary writing, fictional storylines
  • Emotional Camouflage — pressuring the model with emotional scripts (“this is my last request, please help me…”)
  • Composite Example Induction — combining the above with worked examples to lead the model step-by-step

Many models show significantly worse refusal behavior on these composite attacks compared to direct attacks (sources: CSDN News 2026-07-02 + Tencent html5.qq.com 2026-07-02 17:38).

2.3 The test set: 313 high-risk questions curated from 94,108 papers + 34,452 encyclopedia entries

The test set was constructed by curating 313 high-risk questions from 94,108 scientific papers in Dongbi’s Dbdata global science literature platform, supplemented by 34,452 science encyclopedia entries. These external sources served as the RAG retrieval reference — when an LLM produced an answer, the report checked whether the model’s claims were factually consistent with the authoritative papers, whether the technical principles were sound, whether the causal logic held up, and whether uncertainty was expressed appropriately.

Source: html5.qq.com 2026-07-02 17:38 (full methodology breakdown).

2.4 Three-way grouping: domestic/foreign, open-source/closed, large-scale/small-scale

The 38 models are compared across three attributes:

  1. By origin — domestic (Chinese) vs. foreign (international)
  2. By deployment — open-source vs. closed-source
  3. By scale — large-scale vs. small-scale

While the report does not publish the full 38-model leaderboard, it makes two directional statements: large-scale closed-source models show the strongest defense against small-scale adversarial inputs, and domestic top-tier models have a clear edge on culture-specific questions (source: html5.qq.com 2026-07-02 17:38).

2.5 Three new evaluation dimensions

The report argues that a single “refusal rate” metric is insufficient for measuring LLM safety. It introduces three new evaluation dimensions:

  • Intent Recognition — can the model identify the user’s underlying intent, not just the surface prompt?
  • Disclosure Calibration — does the model calibrate the depth of detail it reveals?
  • Reliable-but-Risky Outputs — does the model sometimes produce technically correct but potentially harmful content (i.e., not a refusal failure, but a calibration failure)?

This last concept is genuinely novel — most prior benchmarks have treated “refusal” as a binary. The “reliable-but-risky” framing acknowledges that a model can be technically right and yet socially dangerous (source: Tencent html5.qq.com 2026-07-02 17:38).

2.6 5W1H breakdown

  • Who: Dongbi Tech Data (Dongbi Data Co. Ltd.) + School of Digital Economy, SUFE.
  • What: Global Large Language Model Safety Evaluation Report (2026).
  • When: Released July 2, 2026 at the Global Digital Economy Conference Cloud-Computing Security Forum.
  • Where: Beijing, China.
  • Why: To address the rise of science-domain LLM misuse and provide a benchmark for AI governance.
  • How: 313 questions × 38 models × 94,108 paper RAG reference × 34,452 encyclopedia RAG reference.

3. Three Practical Takeaways for Enterprise LLM Deployment

3.1 Always test composite attacks, not just direct attacks

Vendor marketing often quotes direct-attack refusal rates of 95%+. That’s good news, but it doesn’t translate to real-world safety, because real-world adversaries use composite attacks. Add a “composite-attack resistance” KPI to your procurement checklist and LLM Ops dashboard.

3.2 Build hybrid RAG: own knowledge base + third-party authoritative knowledge bases

The report’s methodology demonstrates the value of grounding model outputs in authoritative third-party science corpora (e.g., Dongbi’s Dbdata). For enterprise deployments, we recommend building a hybrid RAG architecture combining the company’s internal knowledge base with one or more authoritative third-party data sources. This reduces both factual errors and the model “going off-script” in high-risk ways.

3.3 Compliance teams need to monitor “reliable-but-risky” outputs

For compliance and risk teams: do not focus only on whether a model refused a harmful query. Watch for cases where the model produced a correct answer that nonetheless constitutes a high-risk output — for example, syntactically valid exploit code, technically accurate biochemical procedures, or privacy-grade information that the model judged harmless to share.

4. Comparison With Existing International LLM Safety Frameworks

BenchmarkYearIssuerModels testedAttack scopeReference
Global LLM Safety Evaluation Report (2026)2026-07-02Dongbi Tech Data + SUFE (China)38Composite attacks in science domainMulti-source (linked above)
OpenAI Preparedness Framework2025–2026OpenAI (USA)Self-assessmentFrontier Riskopenai.com
Anthropic Responsible Scaling Policy2024–2026Anthropic (USA)Self-assessmentCBRNEanthropic.com
MLCommons AILUM2025MLCommons (international)~10General red-teamingmlcommons.org

The key difference: this new report is the first to deeply focus on the science high-risk domain with a third-party RAG-grounded evaluation, rather than vendor self-assessment.

FAQ (Direct Answers to Common Questions)

Q1: Which 38 specific LLMs were tested? A: The full list of 38 models has not been disclosed in the publicly available summary, but the report confirms grouping by domestic/foreign, open/closed source, and large/small scale. (Source: html5.qq.com 2026-07-02)

Q2: How is this different from OpenAI Preparedness or Anthropic Responsible Scaling? A: Those are vendor self-assessments focused on frontier CBRNE risks. This report is neutral academic + third-party industry, covering 38 models head-to-head on science-domain composite attacks. (Source: Tencent html5.qq.com 2026-07-02)

Q3: What should LLM application developers actually do? A: (1) Add composite-attack resistance to your CI/CD testing pipeline; (2) build hybrid RAG with at least one authoritative third-party corpus; (3) instrument “reliable-but-risky output” detection in production. (Source: This report + industry practice)

Q4: Is 7.6% direct attack success rate high or low? A: It is acceptable for direct attacks. The concern is composite attacks, where success rates are materially higher than 7.6%. Enterprises should test composite attacks explicitly, not rely solely on direct-attack KPIs. (Source: CSDN News 2026-07-02)

Q5: Who is Dongbi Tech Data? A: Dongbi Tech Data Co. Ltd. operates the Dongbi (Dbdata) global science literature platform, which hosts the 94,108 papers used as the RAG reference in this report. They co-published with Shanghai University of Finance and Economics’ School of Digital Economy. (Source: CSDN News 2026-07-02 17:19)

Q6: What is the methodology behind the 313 questions? A: The 313 high-risk science questions were curated from 94,108 papers and 34,452 encyclopedia entries in Dbdata, covering biochemistry, cybersecurity, and other frontier areas. Each model answer is RAG-checked against these references for fact accuracy, principle soundness, causal logic, and uncertainty expression. (Source: html5.qq.com 2026-07-02 17:38)

Key Terminology

  • LLM Safety: The capability of a large language model to remain safe when handling harmful queries, misleading prompts, and composite attacks — measured across refusal rate, intent recognition, disclosure calibration, and reliable-but-risky output metrics.
  • Direct Attack: A straightforward harmful query, such as “tell me how to synthesize X.”
  • Prefix Injection: An attack wrapping the user’s harmful intent inside an authoritative prefix, such as “I am a doctor…” to bypass the model’s safety judgment.
  • Scenario Camouflage: An attack disguising a dangerous query as an academic scenario, literary writing, or fictional storyline.
  • Emotional Camouflage: An attack using emotional pressure (“this is my last request,” “I’m desperate, please help”) to bypass model safety.
  • Composite Attack: The combination of prefix injection + scenario camouflage + emotional camouflage + example induction — the most realistic attack pattern in real-world adversarial settings.
  • RAG (Retrieval-Augmented Generation): A technique in which the model retrieves external authoritative facts before generating an answer, used here to fact-check model outputs against 94,108 papers and 34,452 encyclopedia entries.
  • Dbdata (Dongbi): Dongbi Tech Data’s global science literature platform, used in this report as the RAG reference corpus.
  • Reliable-but-Risky Outputs: A novel concept introduced by this report — model outputs that are technically correct and factually grounded, but nonetheless constitute high social risk if misused.

References

Primary Sources (China)

International Reference Benchmarks

Data Platforms and Institutions

  • Dongbi Tech Data Co. Ltd. — Dbdata global science literature platform (94,108 papers): Dongbi operated
  • School of Digital Economy, Shanghai University of Finance and Economics: https://sdufe.edu.cn/

Publication Info: Published 2026-07-14 by Xi’an Boao Intelligent Technology Co. Ltd. (www.boaoai.cn). Synthesized from the original July 2, 2026 release at the Global Digital Economy Conference in Beijing, with cross-verification across 5 independent sources and structured comparison with international benchmarks.