Data modeling and APIs

0:00- [Instructor] We have a bunch of data and we stored it in a dictionary, but now we have 20 fields, a hundred fields, and all of this data is getting kind of hard to read. I have to scan through all of these fields just to find the piece of data that I need. Now remember when this same thing happened with our code, when our programs got to be too long at the top level, we started breaking them down into functions.

0:25We can apply that same idea to our data. When our data gets to be too big at the top level, we can break it down into smaller data structures, which we can then combine together. By organizing our data into meaningful chunks, we make it easier to understand for all of the programmers out there trying to use it.

0:47Let's take a look at a slightly smaller example. Here we have data for a student record. Right now, all of the fields are stored at the top level, but there are relationships in this data that aren't being clearly represented like all of these fields are for an address and all of these fields are for different classes.

1:07Wouldn't it make more sense if all of the data for a single class was grouped into its own dictionary? We call this process data modeling. We design a meaningful structure for our data that makes it easier to understand and use. So I might group all of these fields for an address into a smaller address dictionary, and then I might group all of the fields related to the student's guardian into its own guardian dictionary and so on and so on.

1:39Adding structure to my data by nesting smaller data structures inside the larger dictionary. You can think of this just like adding structure to a document. You take all of those words and then you group them into sections, and within those sections you have subheadings and maybe those subheadings have other subheadings until finally you have it organized so someone can come in and use that structure to easily find the information they're looking for.

2:09Okay, but who's making data structures that are this complicated in the first place? It turns out pretty much the entire internet. The websites and apps that you use have both a backend and a front end component. The backend is all of the main program logic. It might retrieve some information from a database, process or filter it in some way and then perform some calculations on it.

2:33The front end requests specific data from the backend, and then it displays it on the screen in an appealing and useful way. In most cases, the end user only interacts with the front end, but what's happening in the middle? This layer is called an API or Application Programming Interface.

2:53An API is essentially a special set of functions that a backend program has made available over the internet. For example, when you do navigation in a Maps app, the front end needs to first request all of that directions data from the backend, so maybe it calls a specific API endpoint called get_directions, and then the front end passes in some information about what directions the user is looking for.

3:23The backend then processes that request. It likely retrieves some data from a database and then does some calculations with it. Returning the final result back through the API to the front end, which the front end then displays as this nice little map overlay.

3:40The raw data that came through that API endpoint however, was formatted just like a dictionary. This dictionary format is called JSON, and it's the format most APIs on the web use to transmit data. In most of the programs we write, we won't be the ones creating the data, we'll just be using it.

4:02So we can only hope that whoever did create it modeled the data well. Our job as a programmer then is to learn how to access the fields in a nested data structure and then to do something interesting with it.