Big Data Revolution

Prologue 1 Berkeley, 1930s 1 Pattern Recognition 2 Nelson Peltz 3 Committing to One Percent 5 The Big Data Revolution 6 Introduction 7 Storytelling 7 Objective 7 Outline 8 Part I “The Revolution Starts Now: 9 Industries Transforming with Data” 8 Part II “Learning from Patterns in Big Data” 11 Part III “Leading the Revolution” 11 Storytelling (Continued) 13 Part I: the Revolution Starts Now: 9 Industries Transforming With Data 15 Chapter 1: Transforming Farms with Data 17 California, 2013 17 Brief History of Farming 18 The Data Era 19 Potato Farming 20 Precision Farming 21 Capturing Farm Data 22 Deere & Company Versus Monsanto 24 Integrated Farming Systems 25 Data Prevails 26 The Climate Corporation 26 Growsafe Systems 27 Farm of the Future 27 California, 2013 (Continued) 29 Chapter 2: Why Doctors Will Have Math Degrees 31 United States, 2014 31 The History of Medical Education 32 Scientific Method 32 Rise of Specialists 33 We Have a Problem 34 Ben Goldacre 35 Vinod Khosla 35 The Data Era 36 Collecting Data 36 Telemedicine 38 Innovating with Data 40 Implications of a Data-Driven Medical World 42 The Future of Medical School 42 A Typical Medical School 42 A Medical School for the Data Era 43 United States, 2030 44 Chapter 3: Revolutionizing Insurance: Why Actuaries Will Become Data Scientists 45 Middle of Somewhere, 2012 45 Short History of Property & Casualty Insurance and Underwriting 46 Actuarial Science In Insurance 47 Pensions, Insurance, Leases 49 Compound Interest 50 Probability 50 Mortality Data 50 Modern-Day Insurance 51 Eight Weeks to Eight Days 51 Online Policies 52 The Data Era 52 Dynamic Risk Management 52 Catastrophe Risk 54 Open Access Modeling 55 Opportunities 56 Middle of Somewhere, 2012 (Continued) 58 Chapter 4: Personalizing Retail and Fashion 59 Karolina 59 A Brief History of Retail 60 Retail Eras 60 Aristide Boucicaut 61 The Shift 62 The Data Era 63 Stitch Fix 63 Keaton Row 65 Zara 66 Karolina (Continued) 67 Chapter 5: Transforming Customer Relationships with Data 69 Buying a House 69 Brief History of Customer Service 70 Customer Service Over Time 70 Boeing 72 Financial Services 74 The Data Era 75 An Automobile Manufacturer 76 Zendesk 76 Buying a House (Continued) 77 Chapter 6: Intelligent Machines 79 Denmark 79 Intelligent Machines 80 Machine Data 81 The Data Era 82 General Electric 82 Drones 84 Tesla 86 Networks of Data 87 Denmark (Continued) 88 Chapter 7: Government and Society 89 Egypt, 2011 89 Social Media 90 Intelligence 90 Snowden Effect 91 Privacy Risk Versus Reward 91 Observation or Surveillance 93 Development Targets 93 Open Data 95 Hackathons 95 Open Access 95 Ensuring Personal Protection 96 Private Clouds 97 Sanitizing Data 97 Evidence-Based Policy 97 Public-Private Partnerships 98 Impact Bonds 101 Social Impact Bond 102 Development Impact Bonds 103 The Role of Big Data 104 Egypt, 2011 (Continued) 105 Chapter 8: Corporate Sustainability 107 City of London 107 Global Megaforces 109 Population 109 Carbon Footprint 110 Water Scarcity 110 Environmental Risk 111 BP and Exxon Mobile 111 Early Warning Systems 112 Social Media 113 Risk and Resilience 114 Measuring Sustainability 115 Long-Term Decision Making 116 Stranded Assets 117 City of London (Continued) 118 Chapter 9: Weather and Energy 119 India, 2012 119 The Weather 120 Forecasting the Weather 120 When are Weather Forecasts Wrong? 121 Chaos 122 Ensemble Forecasts 122 Communication 123 Renewable Energy 124 Solar, Hydro, and Wind Power 124 Volatile or Intermittent Supply 125 Energy Consumption 126 Smart Meters 127 Intelligent Demand-Side Management 128 India, 2012 (Continued) 129 Part II: Learning From Patterns in Big Data 131 Chapter 10: Pattern Recognition 133 Elements of Success Rhyme 133 Pattern Recognition: A Gift or Trap? 134 What Fish Teach Us About Pattern Recognition 135 Bayes’ Theorem 135 Tsukiji Market 135 Pattern Recognition 137 Rochester Institute of Technology 137 A Method for Recognizing Patterns 137 Elements of Success Rhyme (Continued) 140 Chapter 11: Why Patterns in Big Data Have Emerged 141 Meatpacking District 141 Business Models in the Data Era 142 Data as a Competitive Advantage 143 Data Improves Existing Products or Services 145 Data as the Product 145 Dun & Bradstreet 146 CoStar 148 Ihs 149 Meatpacking District (Continued) 151 Chapter 12: Patterns in Big Data 153 The Data Factor 154 Summary of Big Data Patterns 155 Redefining a Skilled Worker 155 Creating and Utilizing New Sources of Data 156 Building New Data Applications 157 Transforming and Creating New Business Processes 157 Data Collection for Competitive Advantage 158 Exposing Opinion-Based Biases 159 Real-Time Monitoring and Decision Making 159 Social Networks Leveraging and Creating Data 160 Deconstructing the Value Chain 161 New Product Offerings 161 Building for Customers Instead of Markets 162 Tradeoff Between Privacy and Insight 163 Changing the Definition of a Product 163 Inverting the Search Paradigm for Data Discovery 164 Data Security 165 New Partnerships Founded on Data 165 Shortening the Innovation Lifecycle 166 Defining New Channels to Market 166 New Economic Models 167 Forecasting and Predicting Future Events 168 Changing Incentives 168 New Partnerships (Public/Private) 169 Real-Time Monitoring and Decision Making (Early Warning Systems) 169 A Framework for Big Data Patterns 170 Part III: Leading the Revolution 171 Chapter 13: The Data Opportunity 173 What Oil Teaches Us About Data 173 Bain Study 175 Seizing the Opportunity 176 Chapter 14: Porsche 177 Rome 177 Ferdinand Porsche 178 The Birth of Porsche 178 The Porsche Sports Car 179 Porsche Today 180 Rome (Continued) 180 Chapter 15: Puma 181 Herzogenaurach 181 Advertising Wars 182 Jochen Zeitz 182 Environmental Profit and Loss 183 Herzogenaurach (Continued) 184 Chapter 16: A Methodology for Applying Big Data Patterns 185 Introduction 185 The Method 186 Step 1: Understand Data Assets 187 The Patterns 188 Step 2: Explore Data 191 Challenges 192 Questions 192 Hypotheses 193 Data 193 Models 193 Statistical Significance 194 Step 3: Design the Future 194 The Patterns 195 Step 4: Design a Data-Driven Business Model 197 The Patterns 197 Step 5: Transform Business Processes for the Data Era 199 The Patterns 199 Step 6: Design for Governance and Security 201 The Patterns 201 Step 7: Share Metrics and Incentives 202 Chapter 17: Big Data Architecture 205 Introduction 205 Architect for the Future 206 Lessons from Stuttgart 207 Big Data Reference Architectures 207 Leveraging Investments in Architecture 208 Big Data Reference Architectures 211 Business View 212 Logical View 213 Chapter 18: Business View Reference Architecture 215 Introduction 215 Men’s Trunk: A Retailer in the Data Era 216 The Business View Reference Architecture 217 Answer Fabric 218 Data Virtualization 219 Data Engines 220 Management 221 Data Governance 221 User Interface, Applications, and Business Processes 222 Summary 222 Chapter 19: Logical View Reference Architecture 223 Introduction 223 Men’s Trunk: A Retailer in the Data Era (Continued) 224 The Logical View Reference Architecture 226 Data Ingest 227 Analytics 227 Discovery 228 Landing 228 Operational Warehouse 229 Information Insight 230 Operational Data 231 Governance 231 Men’s Trunk: A Retailer in the Data Era (Continued) 232 Chapter 20: The Architecture of the Future 233 Men’s Trunk: A Retailer in the Data Era (Continued) 233 Men’s Trunk: Applying the Methodology 235 Step 1: Understand Data Assets 235 Step 2: Explore the Data 236 Step 3: Design the Future 237 Step 4: Design a Data-Driven Business Model 237 Step 5: Transform Business Processes for the Data Era 237 Step 6: Design for Governance and Security 237 Step 7: Share Metrics and Incentives 238 Men’s Trunk: The Business View Reference Architecture 239 Answer Fabric 240 Data Virtualization 241 Data Engines 241 Management 242 Data Governance 242 User Interface, Applications, and Business Processes 243 Men’s Trunk: The Logical View Reference Architecture 244 Approach 244 Men’s Trunk: A Retailer in the Data Era (Continued) 248 Epilogue 249 The Time is Now 249 Taking Action 250 Fear not Usual Competitors 251 The Future 252 Index 255