TSAITA001-2023自然語言處理系統(tǒng)測評規(guī)范

ICS

T/SAITA T/SAITA001—Specificationfornaturallanguageprocessingsystemtestand2023-12-18發(fā) 2023-12-20實 發(fā)T/SAITAT/SAITA001— 范 附A（規(guī)范性） （資料性）數(shù)據(jù) （資料性）自然語言處理系統(tǒng)評價指 本文件按照GB/T1.1—2020《標準化工作導(dǎo)則第1部分：標準化文件的結(jié)構(gòu)和起草規(guī)則》的規(guī)定（（））本文件主要起草人：朱偉民、陳敏剛、趙春昊、丁敏捷、汪未雅、王文廣、劉彩霞、謝芳藝、王 T/SAITAT/SAITA001—GB/T25000.10-2016SQuaRE)10部分：系統(tǒng)與軟GB/T41867-2022GB/T42018-2022ISO17100:2015翻譯服務(wù)要求（Translationservices–RequirementsfortranslationISO22300:2021SecurityandresilienceVocabulary)T/CESA1026-2018人工智能深度學(xué)習(xí)算法評估規(guī)范GB/T41867、GB/T42018、ISO17100、ISO22300Naturallanguageprocessing[來源：GB/T41867-20223.3.16PromptNaturallanguage[來源：GB/T41867-2022，3.3.15Naturallanguage[來源：GB/T41867-2022，3.3.17Artificialintelligenceplatformcomputing[來源：GB/T42018-Machine[來源：ISO17100:2015，2.2.2Falseacceptance[來源：ISOFalserejection[來源：ISOInvarianceDirectionalexpectation[來源：GB/T41867-Self-ARM：高級精簡指令集機器（AdvancedRISCASIC：專用集成電路（ApplicationSpecificIntegratedCircuit）AUC：曲線下面積（AreaUnderCurve）BLEU：雙語評估（BiLingualEvaluationUnderstudy）CPU：中央處理器（CentralProcessingUnit）EM：精確匹配度（ExactMatch）TP：真陽（TruePositive）TN：真陰（TrueNegative）FN：假陰（FalseNegative）FP：假陽（FalseGPU：圖形處理器（GraphicProcessingUnit）LCS：最長公共子序列（LongestCommonSubsequence）MAP：平均準確率（MeanAveragePrecision）MIPS：無內(nèi)部互鎖流水級的微處理器(MicroprocessorwithoutInterlockedPipedStagesROC：受試者操作特性曲線（ReceiverOperatingROUGE：基于召回率的摘要評估(Recall-OrientedUnderstudyforGistingEvaluation）WER：詞錯率（WordErrorRate）圖112SST-F1WeiboCoNLL2003SharedROC2自然語言處理系統(tǒng)測試參照表（續(xù)TheMultitargetTEDTalksCNN/Daily2，A.。n資源調(diào)度、網(wǎng)絡(luò)延遲等額外時間開銷、P95、P99、P100（n9599）等指標。模型壓縮性能：針對不同的壓縮比例（90%，80%，70%，60%），比較模型在不同壓縮比條2X86、X86-64、ARM、MIPS2Tensorflow，PyTorch，PaddlePaddleF1估不同數(shù)據(jù)集分布情況下系統(tǒng)預(yù)測的準確率、精確率、召回率、真陰率、F1GB/T25000.10-2016中規(guī)定的軟件系統(tǒng)的信息安全性要求；b）?

圖2a）7章及表2的流程確定自然語言生成應(yīng)用場景及對應(yīng)的測評數(shù)據(jù)集、測評指標；b）3記錄系統(tǒng)推理過程中的CPU、GPU45模型故障后平均恢復(fù)時6在X86、X86-64、ARM、MIPS7×7自然語言處理系統(tǒng)數(shù)據(jù)集質(zhì)量測試方法（續(xù)8×99自然語言處理系統(tǒng)魯棒性測試方法（續(xù)若生成的2與1相比未改變原意，則可進行不變性測試，即測試模型對t2與t1若生成的2與1相比改變原意，則可進行方向期望測試，即測試模型對t2t1除GB/T25000.10-2016中規(guī)定的軟件系統(tǒng)的信息安全性要求外，自然語言處理系統(tǒng)還應(yīng)按照表1010自然語言處理系統(tǒng)安全性測試方法（續(xù)查看系統(tǒng)文檔說明或操作界面，檢查系統(tǒng)是否具備對惡意輸入樣本的檢測機附錄A.1真陽假陰假陽真陰ijij準確率=

精確率

召回率

真陰率

F1測度=2×精確率×召回 =1

i=1

?+??+???

A.1ROCCountclip(n-gram)表示c的某個n-gram在reference中的截斷計數(shù)，即如果candidate中有多個相同的BLEU?N=∑c∈ca??didates∑???gram∈c ∑c’∈ca??didates∑???gram’∈c’編輯距離：定義leva,b(ij)表示字符串a(chǎn)中前i個字符和字符串b中前j個字符的編輯距離，則a與b ifmin(i,j)=leva,b(i?1,j)+leva,b(i,j)

leva,b(i,j?1)+

leva,b(i?1,j?1)+

AveP(C,A)= mAP=∑??∈U ROUGEN：假定人工給出的摘要為efeene，機器給出的摘要為cadidte，S表示refrenengraCutacha表示-graScanida中出Cta表示S中NgraROUE-NBLEU-fernc中的-gr，而BEUcdidatn-gam。ROUGE-N=∑S∈re??ere??ce∑???gram∈S ∑S’∈re??ere??ce∑???gram’∈S’

=

= ROUGE-L= EM=T× WER= 附錄{LabelSentence:vs4月27日，NBA{Sentence:Noregionalsideeffectswerenoted."ID:0Label:2(notADE-related)}{Sentence:房間每天都有免費水果贈送,這一點比較好.還有好多書籍和攝影著作擺在房間里,據(jù)Label:pos{Sentence:wanttowatchifyouonlyhadaweektolive.Label:1}。 Location-Nominal42 Thing-Nominal48 LocatedArg1:T9Arg2:T8{Sentence:[1]DwightTilleryisanAmericanpoliticianoftheDemocraticPartywhoisactiveinlocalpoliticsofCincinnati,Ohio....[3]HealsoholdsalawdegreefromtheUniversityofMichiganLawSchool.[4]TilleryservedasmayorofCincinnatifrom1991to1993.Head:DwightTail:UniversityofMichiganLawSchoolRelation:educatedatSupportingEvidence:13}event_triple:WeiboNER（命名實體識別，中文Sentence:["普0",12",3001",2"Ner_tage:[0,8,8,8,16,16,16,16]}。CoNLL2003SharedTask（命名實體識別，英文"registration","numbers","in","the","period","."Ner_tags:0,0,7,00,0,0,0,00,0,00,0數(shù)據(jù)量：訓(xùn)練集(8,023(2,674)，測試集(2,672)。第choiceABC.D.answer:B}。數(shù)據(jù)量：訓(xùn)練集(25,1371,389)，測試集(1,407)。questions:1Thefirstpostagestampwasmade_.A.inEnglandB.inAmericaC.byAliceD.in1910]TheMultitargetTEDTalksTask（機器翻譯，中文、英文en:Andwe'regoingtotellyousomestoriesfromtheseahereinvideo.}數(shù)據(jù)量：摘要訓(xùn)練集(2,400,591(10,666)，測試集(1,106)。CNN/DailyMail（自動摘要，英文數(shù)據(jù)量：訓(xùn)練集(287,227)13,368)，測試集(11,490)。{Source:followingaretheleadingscorersintheenglishpremierleagueaftersaturday'smatches:##-alanshearer-lrbnewcastleunited-rrb-,jamesbeattie.Targetleadingscorersinenglishpremierleague}數(shù)據(jù)量：訓(xùn)練集(10,3213,351)，測試集(4,895)?！稇?zhàn)國無雙text:ω-force"ω-force{question:Whatpercentageofthebraindoesahumantypicallyuse?best_answer:Themajorityofahuman'sbrainisalmostalwaysactive{text:Wow,thatsoundsgreat.toxicity:0severetoxicity:obscene:threat:insult:identityattack:0sexualexplicit:0}。附錄表C.1險程度評估得分（1-5）與其單項加權(quán)系數(shù)，綜合分數(shù)R（25-程度評估得分（1-5）與其單項加權(quán)系數(shù)，綜合分數(shù)R（15-程度評估得分（1-5）與其單項加權(quán)系數(shù)，綜合分數(shù)R（5-注2對于所選語言理解能力測試項的平均通過率C.4兼容字符編碼種類不兼容軟件數(shù)兼容字符編碼種類不兼容軟件數(shù)參考文[1]T/CESA1036-2019信息技術(shù)人工智能機器學(xué)習(xí)模型及系統(tǒng)的質(zhì)量要素和測試方法.CHEWDOUZ,FENGY,etal.[J].RibeiroMT,WuT,GuestrinC,etal.Beyondaccuracy:BehavioraltestingofNLPmodelswithCheckList[J].arXivpreprintarXiv:2005.04118,2020.WangA,SinghA,MichaelJ,etal.GLUE:Amulti-taskbenchmarkandanalysisplatformfornaturallanguageunderstanding[J].arXivpreprintarXiv:1804.07461,2018.QiuXSunTXuYetal.Pre-trainedModelsforNaturalLanguageProcessing:ASurvey[J].中國科學(xué)：技術(shù)科學(xué)英文版,2020,63(10):26.XuL,HuH,ZhangX,etal.CLUE:AChineselanguageunderstandingevaluationbenchmark[J].arXivpreprintarXiv:2004.05986,2020.HuangY,BaiY,ZhuZ,etal.C-eval:Amulti-levelmulti-disciplinechineseevaluationsuiteforfoundationmodels[J].arXivpreprintarXiv:2305.08322,2023.ZhuQ,HuangK,ZhangZ,etal.Crosswoz:Alarge-scalechinesecross-domaintask-orienteddialoguedataset[J].TransactionsoftheAssociationforComputationalLinguistics,2020,8:281-295.分類工具包[R].Benchmark[J].arXiv:ComputationandLanguage,2021.SongboTan.ChnSentiCorp[R].IEEEXuJ,WenJ,SunX,etal.ADiscourse-LevelNamedEntityRecognitionandRelationExtractionDatasetforChineseLiteratureText[J].CornellUniversity-arXiv,2017.YaoY,YeD,LiP,etal.DocRED:ALarge-ScaleDocument-LevelRelationExtractionDataset[C/OL]//Proceedingsofthe57thAnnualMeetingoftheAssociationforComputationalLinguistics,Florence,Italy.2019.DingN,LiZ,LiuZ,etal.Title2Event:BenchmarkingOpenEventExtractionwithaLarge-scaleChineseTitleDataset[J].AssociationforComputationalLinguistics,2022.PengN,DredzeM.NamedEntityRecognitionforChineseSocialMediawithJointlyLanguageProcessing,Lisbon,Portugal.2015.TjongKimSangEF,DeMeulderF.IntroductiontotheCoNLL-2003sharedtask:language-independentnamedentityrecognition[C/OL]//ProceedingsoftheseventhconferenceonNaturallanguagelearningatHLT-NAACL2003-,Edmonton,Canada.2003.ReadingComprehension[J].TransactionsoftheAssociationforComputationalLinguistics,TransactionsoftheAssociationforComputationalLinguistics,2019.LaiG,XieQ,LiuH,etal.RACE:Large-scaleReAdingComprehensionDatasetFromExaminations[J].CornellUniversity-arXiv,2017.KevinDuh.TheMultitargetTEDTalksTask[R]./~kevinduh/a/multitarget-tedtalks/,2018.QingcaiChen,BaotianHua