Unit 3.4
Understanding Big Data and the Five V's
Learning Objectives
By the end of this chapter, you will be able to:
- ✅ Define Big Data and understand why it requires specialized technologies.
- ✅ Describe the Five V's that characterize Big Data (Volume, Velocity, Variety, Veracity, Value).
- ✅ Differentiate between structured, unstructured, and semi-structured data.
- ✅ Identify common sources and technologies associated with Big Data.
What is Big Data?
Big Data refers to vast, complex datasets that are too large to be managed or analyzed using traditional data processing tools.
The challenge isn't just storage. It's about...
- ⚡ Capturing massive datasets.
- ⚙️ Processing them in a timely manner.
- 📊 Analyzing them for meaningful insights.
Quick Check: Big Data or Traditional?
Click each scenario card to reveal whether it requires Big Data technologies or not.
Scenario A: A streaming service tracks 80 million users playing songs simultaneously and updates playlists in real time.
Scenario B: A local pharmacy records 150 prescriptions per day in a spreadsheet.
Scenario C: Smart traffic cameras across Kathmandu stream HD video 24/7 for city-wide congestion analysis.
Scenario D: A school exports quarterly exam results from its student information system.
Click each card to reveal the answer.
The Five V's of Big Data
Big Data is commonly defined by five key characteristics:
1. Volume
The scale of data
2. Velocity
The speed of data
3. Variety
The different forms of data
4. Veracity
The trustworthiness of data
5. Value
The business outcome from data
Five V's: Identify the V!
Read the scenario then select which V of Big Data it best illustrates. (Stats are approximate.)
📊 Quick Poll: Which V Is the Problem?
Hands up for the V that is the main challenge in each case.
- An airline's engines send sensor readings every few seconds during every flight. → Velocity (and Volume over time)
- A city council has spreadsheets, scanned letters and drone photos about the same roads. → Variety
- A firm stores ten years of data but no one can say what decision it has improved. → Value
🎬 Watch: The Five V's in 90 Seconds
1. Volume (Scale) 📊
Refers to the sheer quantity of data being generated and stored.
We've moved beyond Gigabytes (GB) and Terabytes (TB) to...
Petabytes (PB) & Exabytes (EB)
Example: Facebook stores hundreds of petabytes of user photos and videos. The Large Hadron Collider generates ~1 petabyte of data per second.
2. Velocity (Speed) ⚡
The speed at which new data is created and must be processed.
Often, insights are needed in real-time or near-real-time to be useful.
Example: Real-time stock market analysis, live social media trend monitoring, or data from IoT sensors on a factory floor require immediate processing.
3. Variety (Forms) 🧩
Refers to the different forms that data can take. Big Data is rarely neat and tidy.
Structured
Highly organized, like a spreadsheet or SQL database.
Unstructured
No predefined format, like text, images, or video.
Semi-structured
Has tags/markers, like XML or JSON files.
The vast majority of Big Data is unstructured.
Data Type Sorter
Click each item to cycle through data types, then press Check to score yourself.
4. Veracity (Quality) 🔍
Refers to the trustworthiness, accuracy, and quality of the data.
With data from so many sources, uncertainty and "noise" are major challenges.
"Garbage In, Garbage Out"
Example: Analyzing social media sentiment is difficult due to sarcasm, slang, and fake accounts. This affects the data's veracity and can lead to wrong conclusions.
Veracity: Risk or Not?
Click each data source to label it — then Check to see your score.
🤧 Real Case: Google Flu Trends
- 2008: Google launched Flu Trends, estimating flu levels from what people searched for (e.g., "fever", "cough").
- The promise: estimates days ahead of official doctor-visit reports from the US CDC.
- 2012–13: it predicted more than double the flu level the CDC later measured.
- Why: many searches came from worried or curious people, not sick ones; media coverage drove even more searches.
- 2015: Google stopped publishing the estimates.
Huge volume can't fix poor veracity. Big Data works best combined with trusted data.
🎬 Watch: Why Google Flu Trends Failed
5. Value (Outcome) 🎯
Arguably the most important V. Does the data lead to a tangible business outcome?
If you cannot turn your data into value, it's not an asset—it's a costly storage problem.
The goal is to derive insights that lead to:
- Better business decisions
- Improved operational efficiency
- Competitive advantages
🎯 How Data Becomes Value: Four Types of Analytics
Illustrative: a hypothetical online grocery in Kathmandu.
| Type | Question | Example |
|---|---|---|
| Descriptive | What happened? | Vegetable orders fell last month |
| Diagnostic | Why did it happen? | Late deliveries in two areas led to bad reviews |
| Predictive | What will happen? | Orders will spike before Tihar |
| Prescriptive | What should we do? | Add riders in those areas and stock up two weeks early |
Value comes from the last two steps: acting on the data, not just reporting it.
Sources & Technologies
Common Sources
- Social Media Feeds
- Web & Server Logs
- Internet of Things (IoT) Sensors
- GPS & Location Data
- Multimedia (Images, Video)
Specialized Technologies
- Apache Hadoop: For distributed storage (HDFS) and processing (MapReduce).
- NoSQL Databases: (e.g., MongoDB, Cassandra) Designed for unstructured & semi-structured data at scale.
- Cloud Platforms: (AWS, Azure, GCP) Provide scalable infrastructure for Big Data.
Technology Matcher (1/3)
Click the best technology for each Big Data use case, then Check.
Technology Matcher (2/3)
Technology Matcher (3/3)
🐘 Worked Example: MapReduce Counts Words
Task: count how often each word appears in millions of customer reviews.
- Split: HDFS stores the reviews in blocks spread across many cheap servers.
- Map: each server reads its own block and writes pairs like (fast, 1), (delivery, 1), (fast, 1).
- Shuffle: all pairs for the same word are sent to the same server.
- Reduce: that server adds them up: (fast, 2), (delivery, 1).
The work moves to the data, so 100 servers can finish roughly 100× faster than one. That's why Hadoop scales.
🎬 Watch: MapReduce in a Minute
Application: Deriving Value
Global Example: Netflix
Analyzes massive volumes of viewing data (what you watch, when you pause, what you search for) to power its recommendation engine and decide which new shows to produce.
Nepal Context: Tourism Sector
The Nepal Tourism Board could analyze unstructured data from social media (Instagram geotags, travel blogs, TripAdvisor reviews) to identify emerging tourist destinations, understand visitor sentiment, and plan marketing campaigns more effectively.
💬 Discussion: Big Data, Big Responsibility
- A ride app knows where you go every day. What could it learn about you that you never told it?
- Should a bank use your social media activity when deciding on a loan? Why or why not?
- Which V would you add as a sixth: Vulnerability, Variability or Visualization?
Key Takeaways
- Big Data is defined by the Five V's: Volume, Velocity, Variety, Veracity, and Value.
- Most Big Data is unstructured (text, images, video), which traditional databases cannot handle well.
- Ensuring data Veracity (quality) is a major challenge before analysis can begin.
- The ultimate goal is to extract Value to drive better business decisions and outcomes.
- Specialized tools like Hadoop and NoSQL databases are required to manage Big Data.
Exam Practice 📝
Short answer
- What is big data?
- Explain the five Vs of big data.
- Give three sources of big data.
Exam Practice 📝 (continued)
Long answer
- Explain the characteristics of big data and how organizations can create value from it, with an example.
- Discuss the sources of big data and the technologies used to store and analyse it.
Revise the answers in the full note.
Image & Video Credits
Images
- Four types of analytics diagram: made for this course
- MapReduce word count diagram: made for this course
Videos (YouTube)
- JuniorAnalytics, "Exploring the Power of Big Data 5 V's" (10–95 s)
- PredictionX (Ben Shneiderman), "When Google Failed" (74–153 s); he says the site was removed "by 2013", but it closed in 2015
- Computerphile, "MapReduce" (0–68 s)
Questions & Discussion
Let's discuss the chapter questions.