Big Data For Dummies

Introduction 1 About This Book 2 Foolish Assumptions 2 How This Book Is Organized 3 Part I: Getting Started with Big Data 3 Part II: Technology Foundations for Big Data 3 Part III: Big Data Management 3 Part IV: Analytics and Big Data 4 Part V: Big Data Implementation 4 Part VI: Big Data Solutions in the Real World 4 Part VII: The Part of Tens 4 Glossary 4 Icons Used in This Book 5 Where to Go from Here 5 Part I: Getting Started with Big Data 7 Chapter 1: Grasping the Fundamentals of Big Data 9 The Evolution of Data Management 10 Understanding the Waves of Managing Data 11 Wave 1: Creating manageable data structures 11 Wave 2: Web and content management 13 Wave 3: Managing big data 14 Defining Big Data 15 Building a Successful Big Data Management Architecture 16 Beginning with capture, organize, integrate, analyze, and act 16 Setting the architectural foundation 17 Performance matters 20 Traditional and advanced analytics 22 The Big Data Journey 23 Chapter 2: Examining Big Data Types 25 Defining Structured Data 26 Exploring sources of big structured data 26 Understanding the role of relational databases in big data 27 Defining Unstructured Data 29 Exploring sources of unstructured data 29 Understanding the role of a CMS in big data management 31 Looking at Real-Time and Non-Real-Time Requirements 32 Putting Big Data Together 33 Managing different data types 33 Integrating data types into a big data environment 34 Chapter 3: Old Meets New: Distributed Computing 37 A Brief History of Distributed Computing 37 Giving thanks to DARPA 38 The value of a consistent model 39 Understanding the Basics of Distributed Computing 40 Why we need distributed computing for big data 40 The changing economics of computing 40 The problem with latency 41 Demand meets solutions 41 Getting Performance Right 42 Part II: Technology Foundations for Big Data 45 Chapter 4: Digging into Big Data Technology Components 47 Exploring the Big Data Stack 48 Layer 0: Redundant Physical Infrastructure 49 Physical redundant networks 51 Managing hardware: Storage and servers 51 Infrastructure operations 51 Layer 1: Security Infrastructure 52 Interfaces and Feeds to and from Applications and the Internet 53 Layer 2: Operational Databases 54 Layer 3: Organizing Data Services and Tools 56 Layer 4: Analytical Data Warehouses 56 Big Data Analytics 58 Big Data Applications 58 Chapter 5: Virtualization and How It Supports Distributed Computing 61 Understanding the Basics of Virtualization 61 The importance of virtualization to big data 63 Server virtualization 64 Application virtualization 65 Network virtualization 66 Processor and memory virtualization 66 Data and storage virtualization 67 Managing Virtualization with the Hypervisor 68 Abstraction and Virtualization 69 Implementing Virtualization to Work with Big Data 69 Chapter 6: Examining the Cloud and Big Data 71 Defining the Cloud in the Context of Big Data 71 Understanding Cloud Deployment and Delivery Models 72 Cloud deployment models 73 Cloud delivery models 74 The Cloud as an Imperative for Big Data 75 Making Use of the Cloud for Big Data 77 Providers in the Big Data Cloud Market 78 Amazon’s Public Elastic Compute Cloud 78 Google big data services 79 Microsoft Azure 80 OpenStack 80 Where to be careful when using cloud services 81 Part III: Big Data Management 83 Chapter 7: Operational Databases 85 RDBMSs Are Important in a Big Data Environment 87 PostgreSQL relational database 87 Nonrelational Databases 88 Key-Value Pair Databases 89 Riak key-value database 90 Document Databases 91 MongoDB 92 CouchDB 93 Columnar Databases 94 HBase columnar database 94 Graph Databases 95 Neo4J graph database 96 Spatial Databases 97 PostGIS/OpenGEO Suite 98 Polyglot Persistence 99 Chapter 8: MapReduce Fundamentals 101 Tracing the Origins of MapReduce 101 Understanding the map Function 103 Adding the reduce Function 104 Putting map and reduce Together 105 Optimizing MapReduce Tasks 108 Hardware/network topology 108 Synchronization 108 File system 108 Chapter 9: Exploring the World of Hadoop 111 Explaining Hadoop 111 Understanding the Hadoop Distributed File System (HDFS) 112 NameNodes 113 Data nodes 114 Under the covers of HDFS 115 Hadoop MapReduce 116 Getting the data ready 117 Let the mapping begin 118 Reduce and combine 118 Chapter 10: The Hadoop Foundation and Ecosystem 121 Building a Big Data Foundation with the Hadoop Ecosystem 121 Managing Resources and Applications with Hadoop YARN 122 Storing Big Data with HBase 123 Mining Big Data with Hive 124 Interacting with the Hadoop Ecosystem 125 Pig and Pig Latin 125 Sqoop 126 Zookeeper 127 Chapter 11: Appliances and Big Data Warehouses 129 Integrating Big Data with the Traditional Data Warehouse 129 Optimizing the data warehouse 130 Differentiating big data structures from data warehouse data 130 Examining a hybrid process case study 131 Big Data Analysis and the Data Warehouse 133 The integration lynchpin 134 Rethinking extraction, transformation, and loading 134 Changing the Role of the Data Warehouse 135 Changing Deployment Models in the Big Data Era 136 The appliance model 136 The cloud model 137 Examining the Future of Data Warehouses 137 Part IV: Analytics and Big Data 139 Chapter 12: Defining Big Data Analytics 141 Using Big Data to Get Results 142 Basic analytics 142 Advanced analytics 143 Operationalized analytics 146 Monetizing analytics 146 Modifying Business Intelligence Products to Handle Big Data 147 Data 147 Analytical algorithms 148 Infrastructure support 148 Studying Big Data Analytics Examples 149 Orbitz 149 Nokia 150 NASA 150 Big Data Analytics Solutions 151 Chapter 13: Understanding Text Analytics and Big Data 153 Exploring Unstructured Data 154 Understanding Text Analytics 155 The difference between text analytics and search 156 Analysis and Extraction Techniques 157 Understanding the extracted information 159 Taxonomies 160 Putting Your Results Together with Structured Data 160 Putting Big Data to Use 161 Voice of the customer 161 Social media analytics 162 Text Analytics Tools for Big Data 164 Attensity 164 Clarabridge 165 IBM 165 OpenText 165 SAS 166 Chapter 14: Customized Approaches for Analysis of Big Data 167 Building New Models and Approaches to Support Big Data 168 Characteristics of big data analysis 168 Understanding Different Approaches to Big Data Analysis 170 Custom applications for big data analysis 171 Semi-custom applications for big data analysis 173 Characteristics of a Big Data Analysis Framework 174 Big to Small: A Big Data Paradox 177 Part V: Big Data Implementation 179 Chapter 15: Integrating Data Sources 181 Identifying the Data You Need 181 Exploratory stage 182 Codifying stage 184 Integration and incorporation stage 184 Understanding the Fundamentals of Big Data Integration 186 Defining Traditional ETL 187 Data transformation 188 Understanding ELT — Extract, Load, and Transform 189 Prioritizing Big Data Quality 189 Using Hadoop as ETL 191 Best Practices for Data Integration in a Big Data World 191 Chapter 16: Dealing with Real-Time Data Streams and Complex Event Processing 193 Explaining Streaming Data and Complex Event Processing 194 Using Streaming Data 194 Data streaming 195 The need for metadata in streams 196 Using Complex Event Processing 198 Differentiating CEP from Streams 199 Understanding the Impact of Streaming Data and CEP on Business 200 Chapter 17: Operationalizing Big Data 201 Making Big Data a Part of Your Operational Process 201 Integrating big data 202 Incorporating big data into the diagnosis of diseases 203 Understanding Big Data Workflows 205 Workload in context to the business problem 206 Ensuring the Validity, Veracity, and Volatility of Big Data 207 Data validity 207 Data volatility 208 Chapter 18: Applying Big Data within Your Organization 211 Figuring the Economics of Big Data 212 Identification of data types and sources 212 Business process modifications or new process creation 215 The technology impact of big data workflows 215 Finding the talent to support big data projects 216 Calculating the return on investment (ROI) from big data investments 216 Enterprise Data Management and Big Data 217 Defining Enterprise Data Management 217 Creating a Big Data Implementation Road Map 218 Understanding business urgency 218 Projecting the right amount of capacity 219 Selecting the right software development methodology 219 Balancing budgets and skill sets 219 Determining your appetite for risk 220 Starting Your Big Data Road Map 220 Chapter 19: Security and Governance for Big Data Environments 225 Security in Context with Big Data 225 Assessing the risk for the business 226 Risks lurking inside big data 226 Understanding Data Protection Options 227 The Data Governance Challenge 228 Auditing your big data process 230 Identifying the key stakeholders 231 Putting the Right Organizational Structure in Place 231 Preparing for stewardship and management of risk 232 Setting the right governance and quality policies 232 Developing a Well-Governed and Secure Big Data Environment 233 Part VI: Big Data Solutions in the Real World 235 Chapter 20: The Importance of Big Data to Business 237 Big Data as a Business Planning Tool 238 Stage 1: Planning with data 238 Stage 2: Doing the analysis 239 Stage 3: Checking the results 239 Stage 4: Acting on the plan 240 Adding New Dimensions to the Planning Cycle 240 Stage 5: Monitoring in real time 240 Stage 6: Adjusting the impact 241 Stage 7: Enabling experimentation 241 Keeping Data Analytics in Perspective 241 Getting Started with the Right Foundation 242 Getting your big data strategy started 242 Planning for Big Data 243 Transforming Business Processes with Big Data 244 Chapter 21: Analyzing Data in Motion: A Real-World View 245 Understanding Companies’ Needs for Data in Motion 246 The value of streaming data 247 Streaming Data with an Environmental Impact 247 Using sensors to provide real-time information about rivers and oceans 248 The benefits of real-time data 249 Streaming Data with a Public Policy Impact 249 Streaming Data in the Healthcare Industry 251 Capturing the data stream 251 Streaming Data in the Energy Industry 252 Using streaming data to increase energy efficiency 252 Using streaming data to advance the production of alternative sources of energy 252 Connecting Streaming Data to Historical and Other Real-Time Data Sources 253 Chapter 22: Improving Business Processes with Big Data Analytics: A Real-World View 255 Understanding Companies’ Needs for Big Data Analytics 256 Improving the Customer Experience with Text Analytics 256 The business value to the big data analytics implementation 257 Using Big Data Analytics to Determine Next Best Action 257 Preventing Fraud with Big Data Analytics 260 The Business Benefit of Integrating New Sources of Data 262 Part VII: The Part of Tens 263 Chapter 23: Ten Big Data Best Practices 265 Understand Your Goals 265 Establish a Road Map 266 Discover Your Data 266 Figure Out What Data You Don’t Have 267 Understand the Technology Options 267 Plan for Security in Context with Big Data 268 Plan a Data Governance Strategy 268 Plan for Data Stewardship 268 Continually Test Your Assumptions 269 Study Best Practices and Leverage Patterns 269 Chapter 24: Ten Great Big Data Resources 271 Hurwitz & Associates 271 Standards Organizations 271 The Open Data Foundation 272 The Cloud Security Alliance 272 National Institute of Standards and Technology 272 Apache Software Foundation 273 Oasis 273 Vendor Sites 273 Online Collaborative Sites 274 Big Data Conferences 274 Chapter 25: Ten Big Data Do’s and Don’ts 275 Do Involve All Business Units in Your Big Data Strategy 275 Do Evaluate All Delivery Models for Big Data 276 Do Think about Your Traditional Data Sources as Part of Your Big Data Strategy 276 Do Plan for Consistent Metadata 276 Do Distribute Your Data 277 Don’t Rely on a Single Approach to Big Data Analytics 277 Don’t Go Big Before You Are Ready 277 Don’t Overlook the Need to Integrate Data 277 Don’t Forget to Manage Data Securely 278 Don’t Overlook the Need to Manage the Performance of Your Data 278 Glossary 279 Index 295