大數(shù)據(jù)科學(xué)與機(jī)器學(xué)習(xí)平臺(tái)介紹

1、大數(shù)據(jù)科學(xué)與機(jī)器學(xué)習(xí)平臺(tái)介紹技術(shù)創(chuàng)新，變革未來Agenda 大綱v 數(shù)據(jù)科學(xué)和機(jī)器學(xué)習(xí)概要Data Science 101Machine Learning 101Data Science and ML Challengesv IBM 數(shù)據(jù)科學(xué)平臺(tái)介紹IBM Data Science ExperienceIBM Machine Learningv 數(shù)據(jù)科學(xué)和機(jī)器學(xué)習(xí)案例演示W(wǎng)hat is Data Science?Data science, also known as data-driven science, is an interdisciplinary field about scienti

2、fic methods, processes and systems to extract knowledge or insights from data in various forms, either structured or unstructured,12 similar to Knowledge Discovery in Databases (KDD).Data science is a concept to unify statistics, data analysis and their related methods in order to understand and ana

3、lyze actual phenomena with data.3 It employs techniques and theories drawn from many fields within the broad areas of mathematics, statistics, information science, and computer science, in particular from the subdomains of machine learning, classification, cluster analysis, data mining, databases, a

4、nd visualization.From Wikipedia4Data Scientist: The Sexiest Job of the 21st CenturyWhat abilities make a data scientist successful?Think of him or her as a hybrid ofdata hackeranalystcommunicatortrusted adviserThe combination is extremely powerfuland rare.-Harvard Business Review Oct 2012 Issue數(shù)據(jù)科學(xué)家

5、的硬技能/thoughts/becoming-a-data-scien tist/機(jī)器學(xué)習(xí) 第三次浪潮What is Machine Learning?Computers that Learn without being explicitly programmed Grow and change when exposed to new dataDeliver personalized and optimized customer interactionsIdentify Patternsnot readily foreseen byhumansBuild Modelsof behavior f

6、rom those patternsAchieving Business Value through Watson Machine LearningChurn analysis helps identify the cause of the churn and implement effective strategies for retention.Detect and understand life- threatening medicalconditions and design ever more effective treatment programsLearn, predict we

7、ather patterns and energy production from renewable sources and integrate into grid more effectivelyProduct recommendation, nextpurchase prediction, targeted offers individual tailored shopping experience.Identify suspicious behavior, predict and prevent threats / fraud continually reduce business r

8、isks and costsCapabilitiesMachine Learning helpsConstantly learns and adaptsAvoids making the same mistakesFaster, deeper, improved insightsResulting inSmarter business outcomesLower business risks and costsNew business opportunities8Machine Learning 101 : Types of machine learningClassificationData

9、 points are labeled and are being used to predict a categoryTwo-class vs multi-classExample:Fraud detection (fraud vs non-fraud)Spam email detection (spam vs non-spam)RegressionWhen a value is being predictedExample:Stock prices predictionClusteringData points are not labeled.Goal is to group data i

10、nto clusters to better organize the data9Machine Learning 101 : feature engineeringA feature is a piece of information that might be useful for predictionExample, predict the churn probability of a customerLabeled data is the desired output dataExample, CHURN_LABEL false representing a churn sampleN

11、OT a featureFeatureFeatureFeatureTraining a modelFeature EngineeringFeature EngineeringScoringLabeled examplesTrainingScoringNew dataModelModelPredicted dataa TrainOps (DevOps) storyDeploy11Data ScientistOperational systemDevOpsWhat is Machine Learning (機(jī)器學(xué)習(xí)概要)The (incomplete) machine learning proce

12、ssTakes significant development, deployment and management effortsIngest DataExtract FeaturesTrain ModelDeploy ModelMake PredictionsHuman Intervention1212Choose Best ModelIdentify Model DegradationPrediction And ScoringManage Deployments數(shù)據(jù)科學(xué)及機(jī)器學(xué)習(xí)新挑戰(zhàn)降低數(shù)據(jù)科學(xué)入門門檻 (Citizen Data Scientist)管控機(jī)器學(xué)習(xí)全生命周期提高持續(xù)交

13、付能力數(shù)據(jù)科學(xué)的可重復(fù)性1133IngestionFeedbackMonitorDataTrainEvalDeployPredict/ActIngestionPrepScoreHistory dataTraining & Validation dataTest dataNew dataAgenda 大綱v 數(shù)據(jù)科學(xué)和機(jī)器學(xué)習(xí)概要Data Science 101Machine Learning 101Data Science and ML Challengesv IBM 數(shù)據(jù)科學(xué)平臺(tái)介紹IBM Data Science ExperienceIBM Machine Learningv 數(shù)據(jù)科學(xué)和機(jī)

14、器學(xué)習(xí)案例演示14IBM 數(shù)據(jù)科學(xué)工具箱v IBM SPSSv IBM Data Science Experiencev IBM Machine Learning1516IBM Data Science Experience社區(qū)教材與數(shù)據(jù)集連接數(shù)據(jù)科學(xué)家提問文章與論文開源Scala/Python/R/SQLJupyter and Zeppelin* NotebooksRStudio IDE and Shiny appsApache Spark復(fù)制與分享項(xiàng)目 Your favorite librariesIBM 提供的能力數(shù)據(jù)預(yù)處理/Pipeline UI *自動(dòng)數(shù)據(jù)準(zhǔn)備與建模*高級(jí)可視化*模型

15、管理與部署模型API文檔*Spark云服務(wù)/Packaged SparkData Science Experience (DSx) 主要特性v DSx Cloud Service v DSx Local EditionIBM Machine Learning for z/OS 組件Notebook 和可視化建立模型Cognitive Assistant for Data Scientists (CADS)模型部署模型管理持續(xù)監(jiān)控和反饋IBM Machine Learning for z/OS 企業(yè)級(jí)機(jī)器學(xué)習(xí)平臺(tái)Feature Highlights CADS 數(shù)據(jù)科學(xué)認(rèn)知助手18What is

16、CADS?Cognitive Assistant for Data Scientist which helps select the best fit algorithm for trainingWhy Data Scientists need CADS?Many algorithms for classification/regression tasks: SVM, Decision Trees/Forests, Nave Bayes, Logistic Regression, etc.Substantial cost in user and compute time to select t

17、he best algorithmUser spends time on trying various learnersComputational cost for training a single SVM can exceed 24hSelection commonly based on data scientist bias and experienceFeature Highlights CADS/HPOTraining DataLogistic RegressionRandom ForestDecision Tree500500Minimize amount of data to b

18、e considered to make an informed selection of most suitable learnerGiven a data set try to select best approach by directly considering part of actual data19Feature Highlights Integrated Notebook Interface with flexible APIsIngest data from DB2z tableData transformation and training20Feature Highlig

19、hts Data Visualization with Brunel (/Brunel-Visualization/Brunel)212122Feature Highlights Visual Model Builder, the guided Machine Learning InterfaceIngest data and transformTraining and evaluationFeature Highlights Model ManagementManage model, create deploymentManage deploymentFeature Highlights Easily cons