Unit 3.2
The Challenges of Managing Data
Learning Objectives
By the end of this chapter, you will be able to...
- ✅ Identify the main difficulties organizations face when managing data.
- ✅ Describe the traditional file-based approach to data management.
- ✅ Explain the major problems caused by the file-based approach.
The Core Challenge
Organizations collect more data than ever before, but managing it is a massive challenge.
Fundamental Principle: The value of data depends entirely on its quality.
"Garbage In, Garbage Out" (GIGO)
Effective management is crucial for turning raw data into a valuable business asset.
Interactive: Is It "Garbage In"? (1/2)
For each data entry, decide whether it is Good Data or Garbage. Poor inputs always produce poor outputs!
Interactive: Is It "Garbage In"? (2/2)
For each data entry, decide whether it is Good Data or Garbage. Poor inputs always produce poor outputs!
Challenge 1: The Data Deluge 🌊
Organizations are overwhelmed by the sheer volume of data from diverse sources.
- 📊 Internal Transactions: Sales, inventory, HR records
- 🌐 Web & Social Media: Clickstreams, user comments, likes
- 📡 IoT Devices: Sensor data, smart appliances
This creates significant technical and financial challenges for storage and processing.
📉 Real Case: The Spreadsheet That Lost COVID Cases (2020)
- Setting: Public Health England collected test results from labs as CSV files and loaded them into Excel templates.
- The limit: the old
.xlsformat holds only 65,536 rows. Extra rows were silently cut off. - The result: in early October 2020, about 16,000 positive cases were missed from daily reports, delaying contact tracing.
- The fix: split files into smaller batches, then move to a proper database pipeline.
Volume outgrew a file-based tool. No error message appeared; the data just disappeared.
Challenge 2: Data Quality 🎯
Poor data quality leads to flawed analysis and bad business decisions.
Common Quality Issues:
- Inaccurate: Incorrect customer names or numbers
- Incomplete: Missing contact information
- Inconsistent: "Kathmandu" vs. "KTM"
- Out of Date: Old addresses or phone numbers
Interactive: Spot the Data Quality Problem (1/3)
Read each scenario and click the quality issue it best illustrates.
Interactive: Spot the Data Quality Problem (2/3)
Interactive: Spot the Data Quality Problem (3/3)
Read each scenario and click the quality issue it best illustrates.
🎬 Watch: Garbage In, Garbage Out
Challenge 3: Data Security 🔒
Data is a valuable asset that must be protected from threats.
Ensuring data security and privacy is a complex and continuous effort.
- Protecting against unauthorized internal access.
- Defending against external cyberattacks and data breaches.
- Complying with data privacy regulations.
Interactive: Classify the Security Threat (1/2)
Each scenario describes a data security incident. Choose the correct category for each threat.
Interactive: Classify the Security Threat (2/2)
🔓 Real Case: The Equifax Breach (2017)
- Who: Equifax, one of the three big US credit-reporting agencies, holding financial data on millions of people.
- How: attackers used a known flaw in the Apache Struts web software. A patch had been available for about two months, but it wasn't applied.
- What: names, birth dates, addresses and US Social Security numbers of about 147 million people were stolen.
- Lesson: security is a continuous effort. Patching, monitoring and governance matter as much as firewalls.
🎬 Watch: How Equifax Was Breached
Challenge 4: Data Governance 🏛️
Without clear rules, data becomes chaotic and loses its value.
Data Governance: A collection of policies and procedures to manage the availability, usability, integrity, and security of data.
Lack of governance leads to:
- Inconsistent data definitions
- Unclear data ownership
- Diminished trust in data across the organization
🏛️ Worked Example: Governance in Practice
Illustrative: a hypothetical Nepali insurance company.
Before governance
- Sales, Claims and Finance each "own" the customer address
- "Active customer" means three different things
- No one is responsible for fixing errors
After governance
- Data owner: Head of Customer Service decides who may change it
- Data steward: checks and corrects records weekly
- Data dictionary: one agreed definition of "active customer"
Governance = clear rules, clear owners, shared definitions.
The Traditional Solution: File-Based Approach
Before modern databases, organizations managed data in separate, application-specific files.
Sales Dept.
customer_sales.dat
orders.dat
Accounting Dept.
customer_billing.dat
invoices.dat
Marketing Dept.
customer_prospects.dat
campaigns.dat
This created "Information Silos," where departments could not easily share data.
Problem #1: Data Redundancy 💾💾
Data Redundancy: The same piece of data is stored in multiple, separate files.
Example: A customer's address exists in the Sales file, the Accounting file, AND the Marketing file.
Consequences:
- Wasted storage space.
- Increased complexity in data management.
- Leads directly to the next, more serious problem...
Interactive: Redundancy Simulator
Each department stores its own copy of the customer's address. Click the buttons to see what happens when data is stored redundantly.
Sales Dept.
Accounting
Marketing
Problem #2: Data Inconsistency ⚡
A direct result of data redundancy. When data is duplicated, it often becomes inconsistent.
Scenario: Address Change
- A customer calls Sales to update their address.
- The Sales clerk updates
customer_sales.dat. - The address in
customer_billing.datandcustomer_prospects.datremains unchanged.
Result: The organization now has multiple, conflicting "versions of the truth."
Interactive: Match the File-Based Problem
Click a term on the left, then click its matching definition on the right.
🧪 Quick Activity: Redundancy or Inconsistency?
Call out R (redundancy) or I (inconsistency) for each.
- A college stores each student's date of birth in the admissions, library and exam files. → R: same data stored in several places
- The exam file shows a student's surname as "Gurung"; the library file shows "Gurunng". → I: the copies no longer agree
- A hotel keeps a guest's passport number in both the booking and the billing system. → R, and it becomes I the moment only one copy is updated
Problem #3: Data Isolation 🧱
Data Isolation: Data is stored in different files with different formats, making it difficult to access and integrate.
Example: A manager needs a report on "total sales per customer and their outstanding balance."
- Sales data is in the Sales system.
- Billing data is in the Accounting system.
- Creating this report requires a complex, often manual, process of extracting and combining data.
📂 More Limitations of the File-Based Approach
- Program–data dependence: each program is written for one file's exact layout. Add a column, and every program that reads the file must be rewritten.
- Weak integrity: rules (e.g., "age must be positive") live inside each program, so they're easy to miss or apply inconsistently.
- No ad hoc queries: a new question needs a new program, so managers wait days for simple reports.
- Concurrency problems: two staff editing the same file at once can overwrite each other's changes.
- Poor security: access is all-or-nothing at the file level.
Exam tip: Long Q2 usually expects 4–5 limitations, each with a short example.
🎬 Watch: Seven Limits of File Systems
Real-World Impact: A Nepali Context
Scenario: A Bank in Nepal Using a File-Based System
Imagine you open a savings account at a branch in Pokhara. Later, you apply for a loan at a different branch in Kathmandu.
What problems might you face?
- You have to provide all your KYC ("Know Your Customer") details again.
- If you update your phone number for your loan, your savings account record might not be updated, causing you to miss important alerts.
- The bank struggles to get a single view of you as a customer, making it hard to offer you relevant products.
Summary & Key Takeaways 🔍
- Modern data management faces challenges of volume, quality, security, and governance.
- The traditional file-based approach created isolated "information silos."
- Data redundancy (duplicate data) wastes resources and directly causes data inconsistency (conflicting data).
- Data isolation makes it nearly impossible to get a holistic, unified view of the business.
- These problems highlight the need for a modern database approach.
Exam Practice 📝
Short answer
- Why is managing data difficult for organizations?
- What is data redundancy?
- Distinguish between data redundancy and data inconsistency.
Exam Practice 📝 (continued)
Long answer
- Explain the main difficulties of managing data in organizations.
- Describe the traditional file-based approach to data management and its limitations, with an example.
Revise the answers in the full note.
Image & Video Credits
Images
- Torbay COVID-19 Walk-through Testing Site: Partonez, CC BY-SA 4.0, via Wikimedia Commons
- Equifax HQ: Tyler Lahti, CC BY-SA 4.0, via Wikimedia Commons
Videos (YouTube)
- SMA Marketing, "The Importance of Data Quality (Garbage in, Garbage Out)" (58–95 s)
- SciShow, "How the Massive Equifax Data Breach Happened" (21–63 s)
- Neso Academy, "File System vs. Database Management System" (48–92 s)
Thank You!
Any Questions?
Next Topic: Unit 3.3: The Database Approach and Relational Model | IT 233
Creating a "Single Source of Truth"