What Makes Data “Big”?

ECON250 - Big Data Analytics | Week 1

Oleh Omelchenko

2026-01-01

You already work with data

Let’s start with what you already know. Throughout your studies, you’ve worked with data in various ways — cleaning datasets in Python with pandas, running statistical analyses in R, building pivot tables in Excel, maybe using BI tools like Tableau or Power BI.

You’ve loaded CSV files, queried databases, transformed and cleaned the data, and produced some derived information — insights. This is valid work with data.

So here’s a question I want you to think about…

Is the data you’ve worked with “big”?

What’s the largest dataset you’ve personally worked with?

Pause and think for a minute.

I’d like you to pause the video and think for a minute or two about this question.

What’s the largest dataset you’ve personally worked with? Was it a few thousand rows? A hundred thousand? Maybe millions?

And for you personally — did it feel large? Was it big? Was it the file size? Was it maybe the time it took to process? Or was it something else?

[Pause — give students time to think]

Maybe “big” means…

A CSV file that’s several gigabytes?

A database query that takes minutes to run?

Data that crashes your laptop?

These are all reasonable answers. A 5 GB CSV file certainly feels big when Excel refuses to open it. A query that makes you wait definitely feels significant.

But here’s the thing — these definitions are relative to our tools. Let’s think about what happens when we push beyond our familiar boundaries.

Your laptop’s limits

RAM — 8–16 GB

Storage — 256 GB – 1 TB

CPUs — 4–8 cores

Here’s what you’re working with. A typical laptop has maybe 16 gigabytes of RAM — that’s your working memory, what pandas loads data into. Storage is bigger, maybe a terabyte, but that’s where files sit, not where analysis happens.

Even if you use something like Google Colab — yes, it’s cloud-based, but it’s still limited in RAM and storage. And importantly, it’s temporary — you have no persistence, your session can disconnect.

This sets hard limits on what we can analyze. If your dataset is larger than RAM, pandas will struggle or crash. There are tools that can bypass this limitation, but the trade-off is you spend much more time processing. You’re always trading something when working locally — either you need more space, more time, or more power from your notebook. Which, by the way, is quite relevant in our times with power outages — battery life matters for productivity.

Now let’s think about what happens when we scale up…

From KSE HUB…

How many rows? What tables? Think about it.

Consider KSE HUB — our internal portal that you all use. It has information about courses, enrollments, grades, schedules.

If you took a databases course, you have some idea how data is stored in the backend. So think about it — what kind of tables would there be? How many rows would each have?

[Pause — let students think]

You’d have a table for students — maybe a few thousand, including alumni. A table for teachers. A table for courses. And a table for enrollments — this is the big one. Each student enrolls in maybe 50 courses over their studies. Multiply students by courses, and you get your upper limit for enrollment rows.

Even counting all of this, we’re talking maybe hundreds of thousands of rows at most. This is data you could realistically load into pandas or R and analyze locally. It fits. It’s manageable.

But what about something bigger?

…to Rozetka

Millions of products × millions of users × every interaction = ?

Now imagine you’re analyzing Rozetka — Ukraine’s largest online retailer. Think about the scale at which Rozetka operates. Millions of products. Millions of active users.

And here’s the key insight: if you’re analyzing how well things sell, you’re not just interested in transactions — the sales records. That’s just the tip of the iceberg. You want to know what LED to each transaction. What products did people click on? What tabs did they open? What did they search for? What’s happening on the website in general?

And importantly — in big data analysis, you’re NOT interested in one particular person’s behavior. That’s not the goal. What you focus on is the large scale: thousands of customers, millions of interactions, over large periods of time.

So very easily, you’re thinking about billions of rows of data generated daily. Could you load a year of this into your laptop and analyze it in a BI tool? No. The data physically won’t fit.

This is our first clue about what “big” really means.

“Just buy a bigger computer?”

Machine	RAM	Cost
Your laptop	16 GB	~$1,000
Powerful workstation	128 GB	~$5,000
High-end server	1 TB	~$50,000
Rozetka’s data	10+ TB	???

OK, you might think — if my laptop can’t handle it, I’ll just get a bigger computer. And that works, but it has limits and it has costs.

You can buy a workstation with lots of RAM — 128 GB. Expensive, but doable. You can buy a server with a terabyte of RAM for around $50,000.

But Rozetka’s data is much more than 10 terabytes. I would guess it’s measured in petabytes if we’re talking about all historical data. It’s not realistic to just scale up vertically.

At some point, buying bigger machines hits a wall. You need a different approach entirely.

The first dimension of “big”

VOLUME

When data doesn’t fit on one machine

This is Volume — the first V, the first dimension of big data. When data exceeds what a single machine can store or process, you need a fundamentally different approach.

The solution is horizontal scaling: instead of one big computer, use many smaller computers working together. This is typically called “cloud” — but not all cloud is designed for big data. Google Colab is cloud, but it’s not built for this. What we need is infrastructure specifically designed to work with big data in terms of volume.

Systems like BigQuery do exactly this — they spread your data across thousands of servers automatically. You write SQL, they coordinate the army of machines. That’s what we’ll focus on in this course.

Orders of magnitude

MB → Course grades spreadsheet ✓

GB → KSE HUB (all tabular data) ✓

TB → Rozetka (transactions + behavior) ✗

PB → Netflix, Spotify ✗✗

1 PB = 1,000 TB = 1,000,000 GB :::

Let’s put this in perspective with our examples.

When I finish teaching a course and want to analyze student performance, I export from Moodle — that’s several megabytes at most. Easy.

KSE HUB — all of it, the tabular data (not images, not PDF syllabuses, just the database tables) — maybe up to a gigabyte. Still manageable locally.

Rozetka — we’re talking terabytes, possibly petabytes for all historical data. Not realistic to handle locally.

And global players like Netflix, Spotify — you’re thinking petabytes and more. There’s no laptop solution. You need distributed systems.

Now consider a different problem

[Image: Weather station, calm sky]

Daily measurements → Monthly analysis → Quarterly report

Volume is one dimension of “bigness.” But it’s not the only one. Let me pose a different problem.

Imagine you’re analyzing weather patterns. You collect daily temperature, humidity, precipitation. Once a month, you run your analysis, produce your report, update your models.

This is comfortable. Data arrives slowly, you process it in batches, there’s time between collection and analysis. But what if the timing constraints were radically different?

Taxi pricing at rush hour

[Image: Uklon app showing surge pricing]

Is demand being met right now?

Consider Uklon — Ukraine’s taxi service. During rush hour, thousands of ride requests pour in every minute. Pricing needs to adjust in real-time to balance supply and demand. If prices are wrong for even five minutes, either drivers sit idle or passengers can’t get rides.

You can’t collect today’s data and analyze it tomorrow. By then, rush hour is over. You need to process events as they stream in — thousands per second — and react immediately.

Same problem, different domain

[Image: Monobank app notification]

Is this transaction fraudulent?

You have milliseconds to decide.

Same logic applies to fraud detection at Monobank. When a transaction comes in, you can’t flag it as suspicious after the money is gone. You need to analyze the pattern — is this normal behavior for this user? Does it match known fraud patterns? — and decide in milliseconds whether to approve or block.

This is a fundamentally different challenge than volume.

The second dimension of “big”

VELOCITY

When you can’t wait for batch processing

This is Velocity — the speed at which data arrives and must be processed. Some problems require real-time or near-real-time responses.

There’s a spectrum here: batch processing handles data in chunks — hourly, daily, weekly. Near-real-time means minutes to hours. True real-time means milliseconds.

Most business analytics — including what you’ll learn in this course — uses batch processing. But knowing that velocity is a dimension helps you understand when different tools are needed.

The velocity spectrum

Batch (hours–days)

↓

Near-real-time (minutes)

↓

Real-time (milliseconds)

Batch processing is what you’re familiar with — collect data, run analysis, produce output. Most business intelligence works this way, and it’s simpler and more reliable.

Near-real-time is an intermediate zone — you’re processing quickly, but not instantly. Good for dashboards that update every few minutes.

True real-time requires specialized systems like Apache Kafka that can process millions of events per second. That’s a different engineering problem — one we won’t cover in this course, but you should know it exists.

One more dimension to consider

customer_id	name	purchase_date	amount
1001	Anna	2024-03-15	150.00
1002	Oleh	2024-03-16	89.50

This is comfortable. Rows, columns, clear types.

We’ve covered size and speed. But there’s a third dimension that changes everything about how we approach data.

This is what we’re used to. Rows and columns. Clear schema. Each field has a defined type and meaning. SQL was designed for exactly this kind of data, and it works beautifully.

But what if your data doesn’t look like this?

What if your data looks like this?

Scientific papers

Web pages to scrape

Imagine you’re tasked with analyzing scientific research. Your data is thousands of PDF files — text, figures, tables, references all mixed together. How do you put that in a spreadsheet?

Or you need to extract pricing data from competitor websites. Raw HTML, JavaScript-rendered content, constantly changing structure. Where’s your clean CSV?

Or like this?

Nested API responses

Text on image for OCR

JSON from APIs — semi-structured, nested, sometimes inconsistent between records. Social media posts mixing text, emojis, links, and attached media.

Or pure unstructured data: images that need computer vision, audio that needs transcription, video that needs frame-by-frame analysis.

The challenge isn’t size or speed here — it’s that the data doesn’t fit our familiar row-and-column mental model.

The third dimension of “big”

VARIETY

When data isn’t rows and columns

This is Variety — the diversity of data types and structures. Modern data comes in many forms: structured tables, semi-structured JSON, unstructured text and media.

The hardest analytics problems often require combining multiple types: matching transaction records to customer support chat logs, linking product data to image catalogs, connecting sensor readings to maintenance reports.

The Three Vs

📦

VOLUME

Too much for one machine

⚡

VELOCITY

Too fast for batch

🔀

VARIETY

Too diverse for tables

So we’ve discovered the classic framework for understanding big data: the Three Vs.

Data is “big” when it challenges us on at least one of these dimensions. Volume — sheer size. Velocity — speed requirements. Variety — structural complexity.

Most real-world big data problems involve multiple Vs. Rozetka has volume AND velocity — massive historical data PLUS real-time click streams.

Some add more Vs

Veracity — Can you trust the data? Is it accurate?

Value — Is there actual insight worth extracting?

The 3 Vs remain the core framework.

You’ll sometimes see people add more Vs to the framework. Veracity — is the data accurate and trustworthy? Garbage in, garbage out.

Value — just because data is big doesn’t mean it’s useful. Is there actual insight worth the effort of processing it?

These are valid considerations, but the original three Vs capture the fundamental technical challenges. The others are more about data quality and business strategy.

Real-world scale

Who	Scale	Challenge
Spotify	100+ PB, 600M users	Real-time recommendations
Rozetka	Millions of daily events	Inventory, personalization
Monobank	Millions of txns/day	Fraud detection
Nova Poshta	Every package, every scan	Logistics optimization

Let’s ground this with real examples. Spotify has over 100 petabytes and serves 600 million users — when you open the app, it has milliseconds to recommend what you should listen to.

Closer to home: Rozetka processes millions of events daily. Monobank handles millions of transactions, each needing real-time fraud screening. Nova Poshta tracks every package through their network.

These aren’t abstract numbers. These are real systems, running right now.

Economic research is changing

[Satellite imagery → economic activity]

[Mobile data → migration patterns]

This matters for you specifically because economic research is changing. Economists now use satellite imagery to track economic activity — counting cars in parking lots, measuring nighttime light intensity as a proxy for development.

Mobile phone location data reveals migration patterns and urban mobility. Transaction records from banks analyze consumption in real-time. Social media sentiment predicts market movements.

What you’ll learn

This course

• Analytical SQL at scale
• BigQuery handles the Volume
• Patterns that work on billions of rows

Why it transfers

• Same SQL everywhere
• Foundation for Velocity tools
• Foundation for Variety tools

In this course, you’ll learn analytical SQL using BigQuery. BigQuery is Google’s data warehouse — it handles the Volume problem automatically. You write SQL, it distributes your query across thousands of machines.

The patterns you learn will work whether you’re querying a thousand rows or a billion. The SQL skills transfer directly to other platforms — Snowflake, Databricks, anything that uses SQL.

And this foundation makes learning streaming and unstructured data tools much easier later.

“Big data” isn’t a buzzword

It’s a threshold where your tools and techniques

must fundamentally change.

This course teaches you to work beyond that threshold.

Let me leave you with this thought. “Big data” gets thrown around as a marketing term, but it represents something real: a threshold where the approaches that work for small data fail.

You can’t just scale up Excel. You can’t just add more RAM. At some point, you need fundamentally different tools — distributed systems, cloud warehouses, streaming architectures.

This course teaches you to work beyond that threshold. You’ll write SQL that BigQuery translates into operations across a fleet of servers. You’ll query datasets that would crash any laptop.

That’s what makes this valuable. Not the buzzword, but the capability.

Next up

The Big Data Technology Landscape

What tools exist, why they exist,

and why we’re focusing on one part.

In the next video, we’ll zoom out and look at the broader technology landscape. What tools exist for handling big data? Why are there so many? And why are we focusing specifically on BigQuery and analytical SQL in this course?

Understanding where SQL fits in the larger ecosystem will help you see both its power and its limits — and point you toward what to learn next after this course.