Datasets and data structures

0:00- [Instructor] The world generates an incomprehensible amount of data every day, and that growth curve is exponential. So the amount of data we generate only gets bigger. That data can come from social media, security cameras, even telescopes. There's power in having that data.

0:17But the real power comes from our ability to analyze it using programming. A dataset is just a structured collection of data typically used for analysis. For example, we could write a program to process a dataset of millions of social media posts made in the last year.

0:37Our analysis could identify the peak topics for the year, the growth in popularity of certain slang words or trends in public sentiment towards specific people or products. Individuals or companies may then use that analysis to inform their actions, like a streaming service might look at public sentiment to decide which shows to renew, or an aid organization might look at trending topics to help identify areas of need in the aftermath of a natural disaster.

1:06But when there's so much data, how can our programs possibly make sense of it? As our data gets more complex, a list might not always fit our needs. If we have a hundred different data fields, trying to keep track of which information is at which index is going to become quite annoying.

1:25And sometimes there are relationships we need to express in our data, like a social media post and the location it was made from, or names and emails, or a school district might have schools which have classes which have students. Organizing that data in a useful way calls for more complex data structures.

1:44The data structure we choose for a particular program depends entirely on our use case. Certain data structures are better at certain things. Imagine you have a bunch of T-shirts that you need to store. One organizational strategy might be to throw them all on the ground in your room.

2:02Now this is highly optimized for the add a new T-shirt operation. When I'm done using a shirt, I could just throw it on the ground. But if I need to find a specific shirt, that could take me a while. This T-shirt organization isn't optimized for the find operation.

2:22Now, let's say instead I go for the strategy of folding my shirts and putting them away in a drawer, adding a new shirt takes a bit longer now because I need to fold it first. Finding a shirt on average is a bit faster. I might have to dig through the pile, but at least all the shirts are in one place.

2:41I am, however, highly optimized for the find the most recently used shirt operation. If I wear the same few shirts over and over again, they'll always be back on the top of the pile. Which organizational strategy I choose depends on the operations that I think will be the most useful to me.

3:01If I really don't care about which T-shirt I wear, then maybe the find operation isn't all that useful to me. So I'd optimize for the most efficient add a T-shirt operation. Okay, but back to programming. If our T-shirts are our data and our organizational strategy is our data structure, then we need to choose our data structure based on how we think we'll use the data in our program.

3:25And that can be half the battle when we design programs. Choosing the right organization for our data can make our code a lot easier to write, and our program a lot more efficient to run. So let's get into exploring some different datasets, the problems they can be used to solve, and the choice of data structures we use along the way.