挖掘技術(shù)教程

半結(jié)構(gòu)化文半結(jié)構(gòu)化文本挖掘方楊建北京大學(xué)計(jì)算機(jī)科學(xué)技術(shù)研究1Text-centricXMLDocumentsText-centricXMLDocumentsmarkedupasE.g.,assemblymanuals,journalQueriesareuserinformationE.g.,givemetheSection(element)ofedocumentthattellsmehowtochangeabrakelightDifferentfromwell-structuredXMLquerieswhereyoutightlyspecifywhatyou’relooking2VectorspacesandVectorspacesandVectorspaces–tried+testedframeworkforkeywordretrievalOther“bagofwords”applicationsinclassification,clustering…Fortext-centricXMLretrieval,canwemakeuseofvectorspaceideas?Challenge:capturethestructureofanXMLdocumentinthevectorspace.3VectorspacesandForinstance,distinguishbetweenVectorspacesandForinstance,distinguishbetweenfollowingtwoThePearlyMicrosoftBillBill4Content-richXML:MicrosoftTheContent-richXML:MicrosoftTheLexicon5EncodingtheGatesEncodingtheGatesWhataretheaxesofthevectorIntextretrieval,therewouldbeasingleaxisforGatesHerewemustseparateoutthetwooccurrences,underAuthorandTitleThus,axesmustrepresentnotonlyterms,butsomethingabouttheirpositioninanXMLtree6Beforeaddressingthis,letustheBeforeaddressingthis,letusthekindsofquerieswewanttoMicrosoft7QueryTheprecedingQueryTheprecedingexamplescanbeviewedassubtreesofthedocumentButwhat(GatessomewhereunderneathThisisharderandwewillreturntoit8SubtreesandConsiderallsubtreesofthethatincludeatleastonelexiconMicrosoftSubtreesandConsiderallsubtreesofthethatincludeatleastonelexiconMicrosoftMicrosoft…MicrosoftMicrosoft9StructuralCalleachoftheresulting(8+,inpreviousStructuralCalleachoftheresulting(8+,inpreviousslide)subtreesastructuralNotethatstructuraltermsmightoccurmultipletimesinadocumentCreateoneaxisinthevectorspaceforeachdistinctstructuraltermWeightsbasedonfrequenciesfornumberofoccurrences(justaswehadtf)Alltheusualissueswithterms(stemming?Casefolding?)remainExampleoftfToExampleoftfTobeortoExercise:HowmanyaxesarethereinthisHerethestructuraltermscontainingtoorbewouldhavemoreweightthanthosethatForthedocontheleft:inastructuraltermrootedatthenodePlay,shouldn’tHamlethaveForthedocontheleft:inastructuraltermrootedatthenodePlay,shouldn’tHamlethaveahighertfweightIdea:multiplytfcontributionofatermtoanodeklevelsupbyk,forsomeg<Hamlet=0.8Forthedoconthepreviousslide,theHamletismultipliedbyYorickismultipliedbyinanystructuraltermrootedatThenumberofThenumberofstructuralCanbeAlright,howhuge,ImpracticaltobuildavectorindexwithsomanyWillexaminepragmaticsolutionstothisshortly;fornow,continuetobelieve…Structuralterms:Structuralterms:Thenotionofstructuraltermsisindependentofanyschema/DTDfortheXMLdocumentsWell-suitedtoaheterogeneouscollectionofXMLEachdocumentbecomesavectorinthespaceofstructuraltermsAquerytreecanlikewisebefactoredintostructuraltermsAndrepresentedasaAllowsweightingportionsoftheExample…Example…WeightTheWeightTheassignmentoftheweights0.6and0.4inthepreviousexampletosubtreeswasCanbemoreThinkofitasgeneratedbyanapplication,notnecessarilyanend-userQueries,documentsbecomenormalizedRetrievalscorecomputation“just”amatterofcosinesimilaritycomputationRestrictstructuralRestrictstructuralDependingontheapplication,wemayrestrictthestructuraltermsE.g.,mayneverwanttoreturnaTitlenode,onlyBookorPlaynodesSodon’tenumerate/index/retrieve/scorestructuraltermsrootedatsomenodesThecatchThisisThecatchThisisallverypromising,butHowbigisthisvectorCanbeexponentiallylargeinthesizeoftheCannothopetobuildsuchanAndinanycase,stillfailstoanswerqueriesTwoQuery-timeTwoQuery-timematerializationofRestrictthekindsofsubtreestoamanageablesetQuery-timeInsteadofenumeratingallstructuraltermsofalldocsQuery-timeInsteadofenumeratingallstructuraltermsofalldocs(andthequery),enumerateonlyforthequeryThelatterishopefullyasmallNow,we’rereducedtocheckingwhichstructuralterm(s)fromthequerymatchasubtreeofanyThisistreepatternmatching:givenatexttreeandapatterntree,findmatchesExceptwehavemanytextOurtreesarelabeledandTextHereweseekadocwithHamletintheTextHereweseekadocwithHamletintheOnfindingthematchwecomputethecosinesimilarityscoreAfterallmatchesarefound,rankbysortingHamletQueryHamlet(StillAdoc(StillAdocwithYoricksomewhereinQueryWillgettoitRestrictingtheRestrictingtheEnumeratingallstructuralterms(subtrees)isprohibitive,forindexingMostsubtreesmayneverbeusedinprocessinganyqueryCanwegetawaywithindexingarestrictedclassofsubtreesIdeally–focusonsubtreeslikelytoariseinJuruXML(IBMOnlypathsincludingalexicontermInthisJuruXML(IBMOnlypathsincludingalexicontermInthisexamplethereareonly14(why?)suchpathsThuswehave14structuraltermsintheHamletTobeortoWhyisthisfarmoreHowbigcantheindexbeasafunctionoftheCouldhaveusedothersubtrees–e.g.,allsubtreeswithtwosiblingsunderanodeWhichsubtreesgetused:dependsonthelikelyqueriesintheCouldbespecifiedatindexCouldhaveusedothersubtrees–e.g.,allsubtreeswithtwosiblingsunderanodeWhichsubtreesgetused:dependsonthelikelyqueriesintheCouldbespecifiedatindextime–areawithlittleresearchsofarMicrosoft2MicrosoftWhywouldthisbeanydifferentfromjustBecausewepreservemoreofthestructurethataquerymayWhywouldthisbeanydifferentfromjustBecausewepreservemoreofthestructurethataquerymayMicrosoftReturntothedescendantReturntothedescendantNoknownQueryseeksGatesunderinthevectorDeviseamatchfunctioninthevectorDeviseamatchfunctionthatyieldsascorein[0,1]betweenstructuraltermsE.g.,whenthestructuraltermsarepaths,measureThegreatertheoverlap,thehigherthematchCanadjustmatchforwheretheoverlapHowdoweHowdoweusethisinFirstenumeratestructuraltermsintheMeasureeachformatchagainstthedictionaryofstructuraltermsJustlikeapostingslookup,exceptnotBoolean(doesthetermexist)Instead,produceascorethatsays“80%closetothisstructuralterm”,etc.Then,retrievedocswiththatstructuralterm,computecosinesimilarities,etc.ExampleofaretrievalMatchST=ExampleofaretrievalMatchST=StructuralNowranktheDoc’sbycosinesimilarity;e.g.,Doc9scores0.578.ClosingButwhatexactlyisaClosingButwhatexactlyisaInasense,anentirecorpuscanbeviewedasanXMLdocumentWhatareWhataretheDoc’sintheAnythingwearepreparedtoreturnasanCouldbenodes,someoftheirchildrenWhatareWhatarequerieswecan’thandleusingvectorspaces?FindfiguresthatdescribetheCorbaarchitectureandtheparagraphsthatrefertothosefiguresRequiresJOINbetween2RetrievethetitlesofarticlespublishedintheSpecialFeaturesectionofthejournalIEEEMicroDependsonorderofsiblingCanwedoCanwedoYes,butdoesn’tmakesensetodoitcorpus-Candoit,forinstance,withinalltextunderacertainelementnamesayChapterYieldsatf-idfweightforeachlexicontermunderanelementIssues:howdowepropagatecontributionstohigherlevelnodes.SayGateshashighIDFundertheAuthorHowSayGateshashighIDFundertheAuthorHowshoulditbetf-idfweightedfortheBookShouldweusetheidfforGatesinAuthororthatinBook?SQLforSQLforUsageHuman-readableData-orientedMixeddocuments(e.g.,patientReliesXMLSchemaTuringXQueryisstillaworkingTheprincipalTheprincipalformsofXQueryexpressionspathelementFLWR("flower")listdatatypeEvaluatedwithrespecttoaFOR$pINdocument("bib.xml")//publisherLETFOR$pINdocument("bib.xml")//publisherLET$b:=document("bib.xml”)//book[publisher=$p]WHEREcount($b)>100RETURN$pFORgeneratesanorderedlistofbindingsofpublishernamesto$pLETassociatestoeachbindingafurtherbindingofthelistofbookelementswiththatpublisherto$batthisstage,wehaveanorderedlistoftuplesofbindings:WHEREfiltersthatlisttoretainonlythedesiredRETURNconstructsforeachtuplearesultingQueriesSupportedbyQueriesSupportedbyLocation/position(“chapterSimple/play/titlecontainsPathtitlecontains/play//titlecontainsComplexEmployeeswithtwoSubsumes:WhataboutrelevanceHowXQueryHowXQuerymakesAlldocumentsinsetAmustberankedabovealldocumentsinsetB.Fragmentsmustbeorderedindepth-first,left-to-rightorder.XQuery:OrderByXQuery:OrderByfor$dinlet$e:=document("emps.xml")//emp[deptno=$d]wherecount($e)>=10orderbyavg($e/salary)descendingreturn<big-dept>{$d,XQuery:OrderXQuery:OrderByOrderbyclauseonlyallowsorderingbySaybyanattributeRelevanceIsoftenCan’tbeexpressedeasilyasfunctionofsettobeIsbetterabstractedoutofqueryformulation(cf.UniversityofUniversityofGoal:opensourceXMLsearch“Returnable”fragmentsareE.g.,don’treturna<bold>sometext</bold>StructuredDocumentRetrievalEmpoweruserswhodon’tknowtheEnablesearchforanypersonnomatterhowschemaencodesthedataDon’tworryaboutAtomicSpecifiedAtomicSpecifiedinOnlyatomicunitscanbereturnedasresultofsearch(unlessunitspecified)Tf.idfweightingisappliedtoatomicProbabilisticcombinationof“evidence”fromatomicunitsXIRQLXIRQLAsystemshouldalwaysretrievethemostspecificpartofadocumentansweringaquery.Examplequery:<chapter>0.3<section>0.8XQL0.7syntaxReturnsection,notAugmentationEnsureAugmentationEnsurethatStructuredDocumentRetrievalPrincipleisrespected.Assumedifferentqueryconditionsaredisjointevents->independence.er)*P(XQL|section)–n)=0.3+0.6*0.8-0.3*0.6*0.8=0.636SectionrankedaheadofExample:AssignExample:AssignallelementsandattributeswithpersonsemanticstothisdatatypeAllowusertosearchforwithoutspecifyingXIRQL:RelevanceXIRQL:RelevanceFragment/contextDatatypesSemanticXMLXMLNativeXMLNativeXMLUsesXMLdocumentaslogicalShouldPCDATA(parsedcharacterDocumentContrastDBmodifiedforGenericIRsystemmodifiedforXMLIndexingandMostnativeXMLIndexingandMostnativeXMLdatabasestakenaDBNoIRtyperelevanceOnlyafewthatfocusonrelevanceDatavs.Text-centricDatavs.Text-centricData-centricXML:usedformessagingbetweenenterpriseapplicationsMainlyarecastingofrelationalContent-centricXML:usedforannotatingRichinDemandsgoodintegrationoftextretrievalE.g.,findmetheISBN#sofBookswithatleastthreeChaptersdiscussingcocoaproduction,rankedbyPriceDatastructuresDatastructuresforXMLAverybasicDatastructuresforDatastructuresforXMLWhataretheprimitivesweInvertedindex:givemeallelementsmatchingtextqueryQWeknowhowtodothis–treateachelementasadocumentGivemeallelements(immediately)belowanyinstanceoftheBookelementCombinationoftheParent/childNumbereachParent/childNumbereachMaintainalistofparent-childE.g.,Chapter:21EnablesimmediateButwhatabout“thewordHamletunderSceneelementunderaPlayGeneralpositionalViewtheXMLdocumentasatextGeneralpositionalViewtheXMLdocumentasatextBuildapositionalindexforeachMarkthebeginningandendforeachelement,PositionaldroppethunderVersePositionaldroppethunderVerseunderPl6y.SummaryofdataSummaryofdataPathcontainmentetc.canessentiallybesolvedbypositionalinvertedindexesRetrievalconsistsof“merging”Allthecompressiontricksetc.from276AarestillComplicationsarisefrominsertion/deletionofelements,textwithinelementsBeyondthescopeofthisINEX:aINEX:abenchmarkfortext-XMLBenchmarkforBenchmarkfortheevaluationofXMLAnalogofTREC(recallConsistsSetofXMLCollectionofretrievalEachengineindexesEachengineindexesEngineteamconvertsretrievaltasksintoInXMLquerylanguageunderstoodbyInresponse,theengineretrievesnotdocs,butelementswithindocsEngineranksretrievedINEXForINEXForeachquery,eachretrievedelementishuman-assessedontwomeasures:Relevance–howrelevantistheretrievedCoverage–istheretrievedelementtoospecific,toogeneral,orjustrightE.g.,ifthequeryseeksadefinitionoftheFastFourierTransform,doIgettheequation(toospecific),thechaptercontainingthedefinition(toogeneral)orthedefinitionitselfTheseassessmentsareturnedintocompositeprecision/recallmeasuresINEX12,107INEX12,107articlesfromIEEESociety494Averagearticle:1,532XMLAveragenodedepth=INEXEachINEXEachtopicisaninformationneed,oneoftwokinds:ContentOnly(CO)–freetextContentandStructure(CAS)–structuralconstraints,e.g.,containmentSampleINEXCOSampleINEXCO<Title>computationalbiology<Keywords>computationalbiology,bioinformatics,genome,genomics,proteomics,sequencing,proteinfolding<Description>Challengesthatarise,andapproachesbeingexplored,intheinterdisciplinaryfieldofcomputational<Narrative>Toberelevant,adocument/componentmusteithe