TSAITA001-2023自然語(yǔ)言處理系統(tǒng)測(cè)評(píng)規(guī)范

ICS

T/SAITA T/SAITA001—Specificationfornaturallanguageprocessingsystemtestand2023-12-18發(fā) 2023-12-20實(shí) 發(fā)T/SAITAT/SAITA001— 范 附A（規(guī)范性） （資料性）數(shù)據(jù) （資料性）自然語(yǔ)言處理系統(tǒng)評(píng)價(jià)指 本文件按照GB/T1.1—2020《標(biāo)準(zhǔn)化工作導(dǎo)則第1部分：標(biāo)準(zhǔn)化文件的結(jié)構(gòu)和起草規(guī)則》的規(guī)定（（））本文件主要起草人：朱偉民、陳敏剛、趙春昊、丁敏捷、汪未雅、王文廣、劉彩霞、謝芳藝、王 T/SAITAT/SAITA001—GB/T25000.10-2016SQuaRE)10部分：系統(tǒng)與軟GB/T41867-2022GB/T42018-2022ISO17100:2015翻譯服務(wù)要求（Translationservices–RequirementsfortranslationISO22300:2021SecurityandresilienceVocabulary)T/CESA1026-2018人工智能深度學(xué)習(xí)算法評(píng)估規(guī)范GB/T41867、GB/T42018、ISO17100、ISO22300Naturallanguageprocessing[來(lái)源：GB/T41867-20223.3.16PromptNaturallanguage[來(lái)源：GB/T41867-2022，3.3.15Naturallanguage[來(lái)源：GB/T41867-2022，3.3.17Artificialintelligenceplatformcomputing[來(lái)源：GB/T42018-Machine[來(lái)源：ISO17100:2015，2.2.2Falseacceptance[來(lái)源：ISOFalserejection[來(lái)源：ISOInvarianceDirectionalexpectation[來(lái)源：GB/T41867-Self-ARM：高級(jí)精簡(jiǎn)指令集機(jī)器（AdvancedRISCASIC：專用集成電路（ApplicationSpecificIntegratedCircuit）AUC：曲線下面積（AreaUnderCurve）BLEU：雙語(yǔ)評(píng)估（BiLingualEvaluationUnderstudy）CPU：中央處理器（CentralProcessingUnit）EM：精確匹配度（ExactMatch）TP：真陽(yáng)（TruePositive）TN：真陰（TrueNegative）FN：假陰（FalseNegative）FP：假陽(yáng)（FalseGPU：圖形處理器（GraphicProcessingUnit）LCS：最長(zhǎng)公共子序列（LongestCommonSubsequence）MAP：平均準(zhǔn)確率（MeanAveragePrecision）MIPS：無(wú)內(nèi)部互鎖流水級(jí)的微處理器(MicroprocessorwithoutInterlockedPipedStagesROC：受試者操作特性曲線（ReceiverOperatingROUGE：基于召回率的摘要評(píng)估(Recall-OrientedUnderstudyforGistingEvaluation）WER：詞錯(cuò)率（WordErrorRate）圖112SST-F1WeiboCoNLL2003SharedROC2自然語(yǔ)言處理系統(tǒng)測(cè)試參照表（續(xù)TheMultitargetTEDTalksCNN/Daily2，A.。n資源調(diào)度、網(wǎng)絡(luò)延遲等額外時(shí)間開(kāi)銷、P95、P99、P100（n9599）等指標(biāo)。模型壓縮性能：針對(duì)不同的壓縮比例（90%，80%，70%，60%），比較模型在不同壓縮比條2X86、X86-64、ARM、MIPS2Tensorflow，PyTorch，PaddlePaddleF1估不同數(shù)據(jù)集分布情況下系統(tǒng)預(yù)測(cè)的準(zhǔn)確率、精確率、召回率、真陰率、F1GB/T25000.10-2016中規(guī)定的軟件系統(tǒng)的信息安全性要求；b）?

圖2a）7章及表2的流程確定自然語(yǔ)言生成應(yīng)用場(chǎng)景及對(duì)應(yīng)的測(cè)評(píng)數(shù)據(jù)集、測(cè)評(píng)指標(biāo)；b）3記錄系統(tǒng)推理過(guò)程中的CPU、GPU45模型故障后平均恢復(fù)時(shí)6在X86、X86-64、ARM、MIPS7×7自然語(yǔ)言處理系統(tǒng)數(shù)據(jù)集質(zhì)量測(cè)試方法（續(xù)8×99自然語(yǔ)言處理系統(tǒng)魯棒性測(cè)試方法（續(xù)若生成的2與1相比未改變?cè)?，則可進(jìn)行不變性測(cè)試，即測(cè)試模型對(duì)t2與t1若生成的2與1相比改變?cè)?，則可進(jìn)行方向期望測(cè)試，即測(cè)試模型對(duì)t2t1除GB/T25000.10-2016中規(guī)定的軟件系統(tǒng)的信息安全性要求外，自然語(yǔ)言處理系統(tǒng)還應(yīng)按照表1010自然語(yǔ)言處理系統(tǒng)安全性測(cè)試方法（續(xù)查看系統(tǒng)文檔說(shuō)明或操作界面，檢查系統(tǒng)是否具備對(duì)惡意輸入樣本的檢測(cè)機(jī)附錄A.1真陽(yáng)假陰假陽(yáng)真陰ijij準(zhǔn)確率=

精確率

召回率

真陰率

F1測(cè)度=2×精確率×召回 =1

i=1

?+??+???

A.1ROCCountclip(n-gram)表示c的某個(gè)n-gram在reference中的截?cái)嘤?jì)數(shù)，即如果candidate中有多個(gè)相同的BLEU?N=∑c∈ca??didates∑???gram∈c ∑c’∈ca??didates∑???gram’∈c’編輯距離：定義leva,b(ij)表示字符串a(chǎn)中前i個(gè)字符和字符串b中前j個(gè)字符的編輯距離，則a與b ifmin(i,j)=leva,b(i?1,j)+leva,b(i,j)

leva,b(i,j?1)+

leva,b(i?1,j?1)+

AveP(C,A)= mAP=∑??∈U ROUGEN：假定人工給出的摘要為efeene，機(jī)器給出的摘要為cadidte，S表示refrenengraCutacha表示-graScanida中出Cta表示S中NgraROUE-NBLEU-fernc中的-gr，而B(niǎo)EUcdidatn-gam。ROUGE-N=∑S∈re??ere??ce∑???gram∈S ∑S’∈re??ere??ce∑???gram’∈S’

=

= ROUGE-L= EM=T× WER= 附錄{LabelSentence:vs4月27日，NBA{Sentence:Noregionalsideeffectswerenoted."ID:0Label:2(notADE-related)}{Sentence:房間每天都有免費(fèi)水果贈(zèng)送,這一點(diǎn)比較好.還有好多書(shū)籍和攝影著作擺在房間里,據(jù)Label:pos{Sentence:wanttowatchifyouonlyhadaweektolive.Label:1}。 Location-Nominal42 Thing-Nominal48 LocatedArg1:T9Arg2:T8{Sentence:[1]DwightTilleryisanAmericanpoliticianoftheDemocraticPartywhoisactiveinlocalpoliticsofCincinnati,Ohio....[3]HealsoholdsalawdegreefromtheUniversityofMichiganLawSchool.[4]TilleryservedasmayorofCincinnatifrom1991to1993.Head:DwightTail:UniversityofMichiganLawSchoolRelation:educatedatSupportingEvidence:13}event_triple:WeiboNER（命名實(shí)體識(shí)別，中文Sentence:["普0",12",3001",2"Ner_tage:[0,8,8,8,16,16,16,16]}。CoNLL2003SharedTask（命名實(shí)體識(shí)別，英文"registration","numbers","in","the","period","."Ner_tags:0,0,7,00,0,0,0,00,0,00,0數(shù)據(jù)量：訓(xùn)練集(8,023(2,674)，測(cè)試集(2,672)。第choiceABC.D.answer:B}。數(shù)據(jù)量：訓(xùn)練集(25,1371,389)，測(cè)試集(1,407)。questions:1Thefirstpostagestampwasmade_.A.inEnglandB.inAmericaC.byAliceD.in1910]TheMultitargetTEDTalksTask（機(jī)器翻譯，中文、英文en:Andwe'regoingtotellyousomestoriesfromtheseahereinvideo.}數(shù)據(jù)量：摘要訓(xùn)練集(2,400,591(10,666)，測(cè)試集(1,106)。CNN/DailyMail（自動(dòng)摘要，英文數(shù)據(jù)量：訓(xùn)練集(287,227)13,368)，測(cè)試集(11,490)。{Source:followingaretheleadingscorersintheenglishpremierleagueaftersaturday'smatches:##-alanshearer-lrbnewcastleunited-rrb-,jamesbeattie.Targetleadingscorersinenglishpremierleague}數(shù)據(jù)量：訓(xùn)練集(10,3213,351)，測(cè)試集(4,895)。《戰(zhàn)國(guó)無(wú)雙text:ω-force"ω-force{question:Whatpercentageofthebraindoesahumantypicallyuse?best_answer:Themajorityofahuman'sbrainisalmostalwaysactive{text:Wow,thatsoundsgreat.toxicity:0severetoxicity:obscene:threat:insult:identityattack:0sexualexplicit:0}。附錄表C.1險(xiǎn)程度評(píng)估得分（1-5）與其單項(xiàng)加權(quán)系數(shù)，綜合分?jǐn)?shù)R（25-程度評(píng)估得分（1-5）與其單項(xiàng)加權(quán)系數(shù)，綜合分?jǐn)?shù)R（15-程度評(píng)估得分（1-5）與其單項(xiàng)加權(quán)系數(shù)，綜合分?jǐn)?shù)R（5-注2對(duì)于所選語(yǔ)言理解能力測(cè)試項(xiàng)的平均通過(guò)率C.4兼容字符編碼種類不兼容軟件數(shù)兼容字符編碼種類不兼容軟件數(shù)參考文[1]T/CESA1036-2019信息技術(shù)人工智能機(jī)器學(xué)習(xí)模型及系統(tǒng)的質(zhì)量要素和測(cè)試方法.CHEWDOUZ,FENGY,etal.[J].RibeiroMT,WuT,GuestrinC,etal.Beyondaccuracy:BehavioraltestingofNLPmodelswithCheckList[J].arXivpreprintarXiv:2005.04118,2020.WangA,SinghA,MichaelJ,etal.GLUE:Amulti-taskbenchmarkandanalysisplatformfornaturallanguageunderstanding[J].arXivpreprintarXiv:1804.07461,2018.QiuXSunTXuYetal.Pre-trainedModelsforNaturalLanguageProcessing:ASurvey[J].中國(guó)科學(xué)：技術(shù)科學(xué)英文版,2020,63(10):26.XuL,HuH,ZhangX,etal.CLUE:AChineselanguageunderstandingevaluationbenchmark[J].arXivpreprintarXiv:2004.05986,2020.HuangY,BaiY,ZhuZ,etal.C-eval:Amulti-levelmulti-disciplinechineseevaluationsuiteforfoundationmodels[J].arXivpreprintarXiv:2305.08322,2023.ZhuQ,HuangK,ZhangZ,etal.Crosswoz:Alarge-scalechinesecross-domaintask-orienteddialoguedataset[J].TransactionsoftheAssociationforComputationalLinguistics,2020,8:281-295.分類工具包[R].Benchmark[J].arXiv:ComputationandLanguage,2021.SongboTan.ChnSentiCorp[R].IEEEXuJ,WenJ,SunX,etal.ADiscourse-LevelNamedEntityRecognitionandRelationExtractionDatasetforChineseLiteratureText[J].CornellUniversity-arXiv,2017.YaoY,YeD,LiP,etal.DocRED:ALarge-ScaleDocument-LevelRelationExtractionDataset[C/OL]//Proceedingsofthe57thAnnualMeetingoftheAssociationforComputationalLinguistics,Florence,Italy.2019.DingN,LiZ,LiuZ,etal.Title2Event:BenchmarkingOpenEventExtractionwithaLarge-scaleChineseTitleDataset[J].AssociationforComputationalLinguistics,2022.PengN,DredzeM.NamedEntityRecognitionforChineseSocialMediawithJointlyLanguageProcessing,Lisbon,Portugal.2015.TjongKimSangEF,DeMeulderF.IntroductiontotheCoNLL-2003sharedtask:language-independentnamedentityrecognition[C/OL]//ProceedingsoftheseventhconferenceonNaturallanguagelearningatHLT-NAACL2003-,Edmonton,Canada.2003.ReadingComprehension[J].TransactionsoftheAssociationforComputationalLinguistics,TransactionsoftheAssociationforComputationalLinguistics,2019.LaiG,XieQ,LiuH,etal.RACE:Large-scaleReAdingComprehensionDatasetFromExaminations[J].CornellUniversity-arXiv,2017.KevinDuh.TheMultitargetTEDTalksTask[R]./~kevinduh/a/multitarget-tedtalks/,2018.QingcaiChen,BaotianHua