The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation, or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
JEREMY KEPNER: All right, welcome everybody.
I'm really pleased to see you all here today
for what will be the first of a nine lecture course called
signal processing on databases.
And this is a really special day for me
because this sort of represents a big step
in a set of technology that I've been working
on for a couple of decades, and really
been working intensely on for the last three or four years.
And this is the first course on what
we hope will be a technology that
will become very widely used.
It is a completely unique course on a completely unique
technology, so I want to thank you
for being the guinea pigs of this course.
It's hard enough just doing a new course
on an existing topic.
Doing a new course on a novel topic is quite a challenge,
so we're going to be trying a few different things here.
There are going to be certain topics covered
from more than one angle, and we'll
see which ones sort of make the most sense for you.
The first thing I'd like to do is
talk a little bit about the title of the course, which
is a little bit different.
The title is signal processing on databases,
which is not two phrases, signal processing and databases, that
are usually connected together very often.
And so, signal processing, as we all know,
is, here at Lincoln Labs, one of the most core things
that we do.
In terms of the technology, it's almost difficult
to imagine a technology that's more core to what we do.
The only thing I could think of was systems engineering, maybe.
But signal processing is really at the heart of what we do.
Mathematically, signal processing really,
the heart of that is detection theory.
How do you find things?
Given sets of data, how do you eliminate
things you're not looking for and find the things
that you are looking for?
And then, mathematically, the heart of detection theory
is linear algebra, and so, just a show
of hands how many people have had a linear algebra
course in there?
See, that's really awesome.
See at Lincoln, that's probably--
this one a few places in the world
where it was pretty unanimous.
You know, there's not too many places where that's the case.
And so this technology is really designed for you,
because it really assumes a foundation in linear algebra,
and that's not as common as you might think.
And so people who haven't had linear algebra,
it's much harder for them to take advantage of the ideas
that we're going to talk about.
So signal processing, detection theory, linear algebra,
that's sort of the root of the first phrase.
The second phrase, databases, when
we think of databases we think of typically, maybe
things that you do, searching on the web, texts, maybe
other things like that, which is not necessarily
consistent with the mathematics that we think about when we
think about signal processing, detection
theory, and linear algebra, we naturally go onto real numbers,
matrices, those types of things.
So what we're really doing here is
we're connecting that mathematics with sort
of a whole new branch of data.
Words and strings and relationships.
And so that's kind of the really novel piece of it,
and it's really the core idea that we're bringing here
is we're going to bring all that machinery that we know so well,
and we're going to try and apply it
to this new area which is becoming increasingly
important for us.
So that's the explanation of the title.
That's how we picked the title.
There's other titles, and so I just
wanted to spend a little time on that.
An enormous number of people have contributed to this work.
This is a list of the people that I could remember
at the time that I typed the slide,
but no doubt I have forgotten somebody important,
and so I apologize for that.
But certainly these people, and the whole team
have been very instrumental in allowing this technology
to move forward here.
So this is going to be the outline
for this particular lecture.
And there's going to be a rhythm to most
of these lectures, which is there's
going to be a lecture, which is going to be PowerPoint--
I apologize, we all see plenty of PowerPoint--
mainly to introduce ideas.
And then there will be sort of a demo, which
will go through code that is actually
available in your LLGrid accounts now,
and some of that code will even be
part of homework assignments, if you so choose
to do the homework assignments.
And so that's going to be the flavor.
We'll do the slides, we'll take a short break,
and then we'll do the examples and talk about them.
So let me talk about some of the kinds of--
and I stole this slide-- let me talk about some
of the kinds of data that the technology that we're
going to be talking about today really, really enables.
It allows us to deal with data that
has to do with relationships, relationships
particularly as those represented by graphs.
So for example, you might have vehicle tracks.
Those represent relationships between locations.
A car goes from one place to another.
That's a relationship between those locations.
Another is social networks.
You know, everyone wants to know who their friends are,
how many friends they are, who they're friends of friends are,
those types of things.
So that's another type of set of relationship.
Again, very textural and oriented, very much oriented
towards words and not numbers.
Another area is cyber networks, a very apropros topic
that people are, at least everyone is impacted by,
and many people are doing work on, which is relationships
between computers, communication between web servers,
those types of things.
And again, that is a very important area for us.
Just to give you sort of a big picture here,
like well, why am I even here?
Why do I care about this?
What are the things that we can do?
I'm going to kind of walk through an example of some
of the things that we have used this technology for that
have been very exciting for us.
And so, trying-- and we'll have examples
like that throughout the course, you
know, how you can actually use this,
how this has had really a real impact.
And so the first one we want to talk about
is some work we've done in the cyber domain.
So here's an example of a data set,
which is a graph that is web traffic.
And so, if we see here, this is the source IPs.
So, you can imagine these are the computers that
are on the inside of some domain,
and then these are the destination IPs,
all web servers that they hit are the green ones around here.
This is sometimes referred to in graph theory
as a bipartite graph.
It's basically two separate sets,
a vertices that talk to each other
but have no, at least in this data set,
we are not recording any connections within the sets.
And then, hopefully you can see here these faint blue lines
here that are the actual connections that show you
which of these computers are connecting
to which of these computers.
There's obviously a lot of data, web traffic data.
There's enormous amount of it.
It shows 90 minutes of data, and in this particular case
there was actually malicious activity
that we were looking to detect in this data set.
So one of the biggest challenges that you have in any data
set like this is data representation.
And sometimes in database parlance
this is called the schema.
So how you represent your data is usually
if you can figure that out, you're
almost halfway to solving the problem.
And so, given that, in this case,
we wanted to use graph techniques,
we discover that our data almost always
doesn't come to us in a nicely formed graph,
and so we have to go through a lot of steps here.
And so one thing you'll see is with this technology
is that there's almost always possible part of a pipeline
from raw data to conditioning the raw data to inserting it
into a database to doing analytics to doing queries
and other types of things.
You're always really going to be using a pipeline.
And you're going to be wanting to use different technology
tools for different stages in the pipeline.
We're not in any way trying to suggest
that the technologies we're talking about today
are the only pieces.
It's an important piece, the technologies
we're talking about, but they're not the only pieces.
And you're always really functioning
in the context of a pipeline.
So this just shows the technology
stack that we will be leveraging,
that we typically leverage in these types of things.
Ultimately, our goal is to get up here
to these graph analytics.
One of the things we'll be talking about most
in this course is the high level languages
that allow you to essentially do graph analytics very easily
and bridge between the databases themselves,
and the high performance computing that you will need.
So again, not only do you have a pipeline of software and data,
you have a pipeline or stack of technology.
So when you're addressing problems like this,
usually you're starting at one place
but you will almost always, when you build a system,
work out and deal with these other types of things.
So that's sort of just a quick overview and example
and I don't expect any of you-- it's probably just kind of,
the purpose of the example's probably
to raise more questions.
Well, how do you really do this?
So now I'm going to sort of take a bit of a right turn
here and really talk about what the course is about, OK.
And I'm going to actually talk about the course
a little bit of a big picture.
And I always felt that every course should tell you
why that course is the most important course you'll ever
take in your career.
You know, I think every single course
you can come up with an argument for that,
but people don't say it.
So I'm going to tell you why this course is
the most important course you will ever take.
Of course, every course is the most important course
you'll ever take, but I'm going to tell you why this one is.
So stepping way back, we're all part of MIT here
and MIT has a formula for success.
MIT is the number one science and engineering
university on earth.
We can actually, by any reasonable measure
that is true.
And in fact, by the people who track this, it's true by a lot.
And so you know, it's a real honor
to be associated with this organization,
to be at an organization that's at its zenith.
And MIT has had this incredible run of success over the last 60
years because fundamentally they recognize the formula
for making discoveries is a combination of theory
and experiment.
Bring those two together, and as an organization,
we've actually organized ourselves,
MIT is organized around this formula in that we have theory
and experiment, and theory is the academic part
of the mission.
That's run out of departments.
Typically, it involves mathematics.
This is a place-- I mean, everyone here
raised their hands to linear algebra,
and I can go to any part of MIT and that will be true.
We probably have the highest mathematical literacy
of any organization on earth, and that's a very powerful,
powerful capability.
The mathematics that we use, they become manifest or real,
typically in the forms of algorithms,
and our algorithms become real in the form of software.
Likewise, on the experimental side, you know,
that's the research that we conduct.
MIT actually has large laboratories.
Laboratories that are fully peered with the departments.
That is actually quite a unique thing
for an academic institution to have laboratories
and laboratory heads that are right up
there with the department heads in how they're positioned.
Lincoln being the largest of them, CSAIL, the Media Lab,
there's a number of them.
So we have this thing.
And the focus of research, of course,
is measurement, which is data, which
often gets reduced to bytes.
And so, often implementing this MIT formula
comes down to marrying software and bytes together
in a computer.
That's sort of where we bring these two ideas together.
And so we're going to be talking a lot about that.
Implementing software bringing bytes together
analyzing together that's where these two things come together,
and I think that's really, really important.
Now what's the tool we use?
What's the machine that we use to bring these together?
It's a computer, all right.
And nowadays, computers have gotten a little bit
more complicated than the way we used to think about them.
We used to think about them typically
as a von Neumann machine, which means
you had data and operations, you know, bytes and software.
And the bytes go into a part of the computer,
they get operated on, they come out.
And that model is still true at a high level,
but the computers we tend to deal with now
are much, much, much more complicated.
So on the left is a standard parallel computer.
Almost every computer used today looks like this.
You have processors, you have memory,
you have some kind of persistent storage,
you have a network connecting them together,
and this form a very complicated what we call memory hierarchy.
It begins with registers here at the top, caches,
local memory, remote memory, and then storage.
And these are the units that go between them.
And there's a lot of implications of this hierarchy,
in terms of what things are-- you know,
bandwidth goes up as you go this way, latency goes down.
Programmability generally goes up the closer you are.
You don't have to worry about things.
And likewise, the capacity of data
goes down as you go traverse this way in a hierarchy.
So all modern computers that you use
are these von Neumann architectures
with multi-level memory hierarchy.
And fundamentally the thing you have to know
is that this architecture selects the algorithms you use.
If algorithms are not going to run well in this architecture,
you probably don't implement them.
So one has to be very much aware of one's algorithms
with respect to this architecture,
because if you pick algorithms that aren't going to run well
in this architecture, it's going to be very, very difficult.
The goal of this course is to teach you
techniques that will allow you to get your work done faster.
And so this is a rather complicated drawing here,
but this shows, given a problem that you want to solve,
if you implement it in software using
a variety of techniques, what the performance
of those techniques is.
So in a certain sense, you could view this as the volume of code
that you have to write to solve a problem.
And then we have implementing the program and C
as a reference point.
So as you move over here and you do
things like in assembly or Java, MapReduce,
you end up writing more code.
If you move over here and do things in C++ or Java
or MATLAB, you write less code, and this shows you the relative
performance.
Again, as you do things in lower level
environments like assembly, you can get more performance.
And as you do things in higher level environments like MATLAB,
you get less performance.
And this is a fairly general trade off space here.
If we add parallel programming models here,
of which there is many, we have direct memory access,
we have message passing, we have a manager worker style
or map reduce style, we have a whole different styles
of programming here.
We're going to want to try and be here.
We're going to want to do things that take less effort
and deliver you more performance.
And so, we are going to be mostly focusing
on MATLAB in this class and a technology
we call [INAUDIBLE] to describe that more later.
But this is a software combination
that allows you to get relatively good performance
at a significantly reduced effort,
and that's really the only thing we're doing in this course.
We're just saying, if you're doing these types of problems,
there's a right tool for the job or there
are right tools for the job, and if you
use the right tool for the job, it will take you
less time to get the job done.
And we're going to try and teach you about these tools
and how they actually work.
Now, a little bit of bait and switch here.
We call the course signal processing databases,
but we're going to get to databases at the very end,
unfortunately, to disappoint you all.
And the reason is because databases
are good for one type of problem,
but it's often at the very end of the game
where you really need to work with databases.
So to understand, we have a variety of different ways
we can solve problems with algorithms, OK.
We can solve our problems just using a single computer's
memory.
We can solve our problems using the memory
on multiple computers.
We can solve our problems using disks or storage, or even
with many disks, OK, and as we go this way,
we can do more and more and more data.
But one of the things that really matters, and it kind of
goes back to that other chart there is well,
when I'm working with this data, what is the chunk size?
What are the blocks that I need to get the data in?
And if I need to get the data in very, very small chunks,
well then obviously having the data in memory
is going to be the most efficient thing I can do.
If I want to just get one record out of many records,
then that's the most efficient, having the data in memory.
Likewise-- so that's where we typically start, right?
If our problem is small when we almost always try and start
with our problems with small, we're going to be working here.
We're going to be writing what we call serial programs on one
computer.
And then, the great thing about serial memory
is that even as our requests size gets larger,
they're really not going to do anything better than just
having the data in memory.
So we'll be able to grow that model up to data sets
that are fairly small, maybe gigabytes, and we can continue
to use that model just fine.
And then, if we decide, well, we're
really moving beyond that, we need to do more data,
we can go to a parallel program to get more memory,
or we can use data in files.
We can write our data to files, we
can read them in, process them, and read them out,
and that model works very, very well.
And in fact, particularly if we're
going to be accessing the data in large chunks.
If I have a lot of data and like I
want to do some processing on the whole piece of it.
Likewise, if we go to a very large data sets,
we find that we can even do parallel programs writing files
in parallel if we want to do extreme large problems that we
want to traverse all the data.
But finally, we're going to come back to this case
where we need a database.
And a database is when we have very large amounts of data
but we want to access-- database,
data that won't fit in our memory,
but we want to access in little bits.
We want to look up something.
And that's really the only time we really need a database,
unless of course someone says this is your data.
It is available to you on a database.
That's the only way you're going to access it.
But if you have control-- so you always
want to follow this path.
You want to start with your serial program memory,
maybe go to a parallel program using files, maybe
parallel files, and then the database is almost
the two of last resort because it
adds a great deal of complexity, and while it can look up
things much quicker, if you actually end up doing something
where you're going to be traversing
a significant fraction of the data,
you will find it's actually significantly slower
than if you have the data on files.
So we're going to show you techniques,
we're going to follow this path, and then hopefully by the end,
you'll be like, oh, well yes, now I
know why we use a database for these instances,
but I realize that I can get most
of my work done using these other techniques that
work just fine.
So hopefully you're going to be learning
what we call the fast path.
So if you want to get a certain level of performance,
OK, if you have no training or minimal training,
this is the path that you follow.
You began using a technology.
Your program gets slower for a while.
You beat your head against the wall and the web for a while.
Maybe eventually, after days, you
get back to where you were, and then slowly you
climb up here to get it to some level of performance weeks
later.
This is something we observe time and time again,
and typically what happens is that right around here
people just give up with the technology
and they say the technology didn't work for them.
The point of this class is to give you the techniques
that an expert would have.
An expert could use the technology,
know exactly how to use it, and then can go on this path
where basically within hours you're getting good performance
and if they really want to take it to the next level
they can spend days doing that.
And so that's kind of our whole philosophy here.
So we're going to actually have some new concepts
in this course.
I think they're talked about in the course
outline a little bit, some of the really new ideas
that you probably wouldn't have run across in other courses.
So we're going to be talking about graphs a lot
in this course.
The standard graph that-- how many people
have taken a graph theory course or a computer science course
that talked about graphs?
Most.
I'm glad to see linear algebra is still the stronger one here.
That's more important.
Between the two, having the linear algebra background
is the more important.
And I can, if you know linear algebra,
I can teach you graph theory pretty easily.
The traditional definitions of graphs,
though, that are taught in most computer science courses,
they tend to focus on what we call
undirected, unweighted graphs.
So these are a graph is a set of vertices and edges,
and so what we're saying when we say
it's an undirected and unweighted graph
is that all the edges are the same between any two vertices.
There's no real difference between them.
Either an edge exists or it doesn't.
It's kind of like a zero or a one.
And then it doesn't really-- when
we say two vertices connected, there's
no directionality of that.
The majority of graph theory and what's taught in class
is based on that.
Excuse me.
Unfortunately, when we get into the real world--
unfortunately, when we get into the real world,
we find that I've never run into that kind of graph
in the real world.
In the years I've been doing this,
I've never actually run into that kind of graph.
In fact, another part of that is they'll
talk about the distribution.
In fact, you'll often hear something
called an Erdos-Renyi graph, which
is just a random connection.
It is any two vertices are randomly connected
with maybe some probability.
And again, I've never run into that either in the real world.
So the focus of graph theory that we have now
that we have all this data and can compare
the theory and the data, can bury
the theory of the experiment, we see that the theory is wanting.
The theory is not a good stepping stone into the data.
And so what we see is that the real data is not random.
It's often what we call a power law distribution,
that a certain vertices are very popular
and most vertices don't have a lot of connections to them.
Usually the edges are directed, that is, there's meaning.
If I friend you, it's not the same
as you friending me on Facebook or Twitter, or you twit face,
or whatever the technology is of the future.
Usually the edge of self has meaning, has weight.
That weight could be of value.
It could be words, it could be other types of things.
Typically you have multiple edges.
I could send you many friend requests.
You know, that's not the same thing as me sending you
one friend request.
Maybe I can't send you friend requests.
I have multiple friend requests.
I don't know.
But you can have multiple edges between vertices.
And probably most importantly is edges
are often what we call hyper, and this
is a very important concept that is almost never touched
on in graph theory, which is that an edge can connect
multiple vertices at the same time.
A classic example would be an email
that you send to multiple people.
In fact, most emails these days have more than one recipient.
And so, me sending an email to everyone
in the class, one email to everyone in the class,
is a hyper edge.
That edge, the email, connects and has direction,
you know, me with all the recipients.
That's very different than me individually
sending an email to every single one of you, which
would be a standard edge.
So this is an important concept not typically discussed a lot
in standard graph theory that is very important.
So we're going to move beyond the sort of standard definition
of graphs in this class, and that's going to be hard.
We're going to deal with a bigger
definition of linear algebra.
Traditionally, linear algebra is dealing with matrices.
They have indices for the rows and the columns
and the values are real numbers.
Maybe integers, may be complex numbers,
we deal with complex numbers here a lot,
but we're going to be dealing with things like strings.
We're going to be taking linear algebra
and applying it to things like words.
And so that's going to be different.
In fact actually, one of my favorite--
it might be the part of the course that
puts you all asleep, but it's actually
one of my favorite parts of the course.
And then, bigger definition of processing.
One of the dominant things you'll
hear about in this space-- there's
a lot of technologies out there.
A popular technology is called Hadoop,
and it's map/reduce parallel programming paradigm, which
is a great technology to kind of get people started,
but for people with the mathematical sophistication
that we have, we can use programming models
that are simply better.
If you understand mathematics to the level that we do,
we can give you tools for programming parallel computers
that are just more efficient in terms of you write less code
and you get better performance than using techniques that
really are designed for people that don't have
the mathematical, the linear algebraic background
that you have.
So this is really going to be the foundation of the course.
So let me continue here with the course outline.
So this is going to be, essentially,
the nine lecture of the course.
I think this is still pretty accurate.
So we're in the introductory course,
we're going to review the course and goals.
The next course is really going to be dealing
with this concept of associative arrays, which
is the core technology that sort of brings all this together.
When I talk about extending linear algebra to words,
using funny, using, sorry, by using fuzzy algebra that really
gets into mathematical abstract algebraic concepts
called group theory.
It's not nearly as scary as it may sound.
We'll get into that.
It's actually, I think, quite elegant
and really good to know.
Good to know that when you work with the technology
and you work with associative arrays,
that we've actually thought about the mathematics of how
these things work together.
And when we add a feature, it's usually
because it fits mathematically, and when
we don't add a feature, it's usually
because someone has made a request
and like, yeah, that's really going to take people
into a bad place.
Then we're going to get into really sort of the center
part of the class where we're going
to show you how we do analysis of entities
on unstructured data, and then doing analysis
on unstructured data.
We're going to--I've talked about power laws.
We' going to whole class on power law data, modeling
of that deal, but then we have another class
in cross-correlation, and then we're
really going to get into parallel processing
and databases.
And the last two classes aren't ever
going to really be lecturers.
They're going to be really just the demos, and really--
because that's where we're going to really bring a lot of things
together, and we're going to walk you through code examples,
but we're going to need to take some time to really walk you
through them, because you're going to see,
wow, there's all these ideas that we've
talked about coming together in these code examples.
So there is, we have lectures.
If you see in the software that's available in your LLGrid
accounts, all the lectures are there already,
and I think we have lectures one through seven.
So seven is the eighth lecture.
Or, I'm sorry, zero through seven, so seven
is the eighth lecture, and that will
be a kind of a short lecture.
And then the ninth lecture will be just going through examples.
So we believe in this whole philosophy
of taking a linear algebraic point of view
with graphs so much that we even wrote a book about it.
So you have all been given copies of this book.
This really gives you the philosophy and the power
of if you think about graphs using the techniques,
so I'd highly encourage you to flip through it.
There are certain parts of this, certain sections here
that are really almost taken directly out of this book,
but a lot of it is supplementary material.
And what you'll discover is that this course is really
the bridge to these techniques.
This course allows you to go from kind
of this mess of unstructured words
and other types of things, and then
put them into this nice linear algebraic format,
so that is now you can do these techniques that
are described in the book.
So this is almost a bridge that and I'd certainly
encourage you to look at that and to ask me
questions about it at any time.
So now I'm going to come back to our original example, which
was the cyber thing and kind of walk us
through that in a little bit more detail
and maybe highlight a few of the things
that I've talked about already.
So here's a very standard data processing pipeline.
So we want to do is we have raw data,
in this case our raw cyber data, which is a series of records.
And we need to convert that into a set of vertices and edge
lists from which we can then do graphs and graphs analysis.
And, as you'll see, typically what you do
is you write parsers to convert this data from some raw format
to some format that is sort of the next step
in your processing chain.
And then we convert these edge lists
into adjacency matrices, which is how we view our graphs.
One word on adjacency matrix, let me go back here.
This is a graph, [LAUGHS] for those of you who don't know.
The set of vertices and edges, OK?
And this is an adjacency matrix of that graph.
Basically every single row is a vertex,
every single column is a vertex, and if an edge
exists we put a dot here.
There's a formal duality between graph theory,
the fundamental operation of graph theory
is what's called breadth-first search, which is shown here,
giving a starting vertex.
You know, traverse its edges to its neighbors,
and we have the same thing going on here.
Given a starting vertex in a vector,
we do a matrix multiply to identify the neighbors.
And so at the deepest, most fundamental level, graph theory
and linear algebra are linked.
Because the fundamental operation of graph theory
is breadth-first search and the fundamental operation
of linear algebra is vector matrix multiply.
So we wish to form these adjacency matrices so we
can take advantage of that.
Here's a little bit more detail on how
we're going to use this technology D4M to do that.
So we have our raw data, we convert the raw data
into a CSV file-- stands for a comma separated value files.
It's kind of the default format of spreadsheets and tables.
It's become very popular over the years.
I remember when it first came out 20 or 30 years ago,
it wasn't that popular.
You know, it's like people did-- but now it's
become extraordinarily popular file format.
It really is ideally suited for this kind of data.
So we convert that into CSV files.
We then take those CSV files, read them
in using our D4M technology to insert them
into a distributed database.
We can then query that distributed database to get
these associative arrays.
And then we can, from the associative arrays,
now we have the full power to do our graph algorithms.
So D4M really helps you in going, you know,
these few steps and setting the table so that you can then you
graph algorithms.
And you can even then do the graph algorithms
without even using D4M at all.
You can just use regular linear algebraic operations in MATLAB
just fine.
But this is that bridge, that connector.
In fact, I highly recommend people do that.
I highly recommend you use D4M for only the parts
of your problem that it's good for,
and use other technologies, you know,
like MATLAB, ordinary matrices, and other types of things.
That's good for you, you'll get better performance.
It's good for me because, if you have problems,
you'll bother somebody else, and not us.
So that's good, it's good for everybody.
So an example of kind of what this looks like.
Here's a proxy log that we got from the data.
It shows a web, essentially a web log, the first thing we do
is essentially convert this into a CSV file.
So basically, each entry here gets a column,
we have an ID for the actual entry.
In this case it's a source IP, a sever IP, a time stamp.
And then, the actual line of text that's associated with it.
And this can go on and on and on.
We can have many different types here.
And if you are a standard database person,
put your SQL hat, which is sort of the standard database, on.
You would just take that data, and you
would make a table that would have these columns,
and you would insert them and away you would go up.
OK?
But what we discover is that-- the challenge becomes when you
want to look up data quickly.
Or you want to insert data.
Let's say you have an enormous volume of data
that you want to insert and you want
to go to look up any, say, IP address, or any time stamp,
or something like that.
What you discover is that SQL might give you
good performance, but there are databases out there
and technology that will give you better performance.
And these databases are called triple stores,
because they store all the data as-- anyone?
Triples, yes.
They store all the data as triples,
which means we will have a row, and a column,
and a value associated with everything.
So what we do is we take our dense data,
and we do what we call, create an exploded table.
So essentially we have the log ID, let me append that,
and we take the column name and we append its value to it,
and server IP.
And this is what you would call an exploded table.
Now, if you're an old SQL person like me,
this schema will immediately give you hives,
and other types of allergic reactions,
and you will want to go to the hospital,
because you're creating an enormous number of columns.
And in SQL you where you can dynamically create columns,
but at your peril.
Basically, you can create columns dynamically,
but the database reserves the right
to recopy the entire table, and reformat the entire table,
if it so requires.
Well triple stores don't have this requirement.
Triple stores only store the non-blank entries.
And so they're perfectly happy with as many columns you want.
If you have a billion rows, 10 billion columns, completely
fine with that.
So that's a powerful thing.
So I think we talked about that.
So we use the indices as rows, and we
create unique type-value pairs for every single column.
Now, by itself, this schema does you absolutely no good.
You basically just made your life harder,
because, by itself, most database tables,
database systems, including triple stores,
are going to have an orientation.
They're going to be row oriented, or column oriented.
And in this case, the databases we will be talking about
are almost always row oriented.
Which means they can get a row in constant time,
give it a row key like this and you can look it up
very quickly.
And it will give it back to you very quickly.
That's what it's design to do.
And you can insert enormous volumes of data,
and they will preserve that contract.
You can be inserting millions of entries per second,
and you can still look up stuff in milliseconds.
So very powerful technology, but all we've done so far
has made it so that the format of my data
is in this funny format, coming out the other side.
I've just made myself harder.
Well, there is a payoff, which is we also store
the transpose of the table.
So, to anybody who is used to sparse linear algebra,
this is an old ancient technique.
We almost always store A and A transpose,
because in sparse linear algebra we'll
either have a row oriented or column based store.
And then we just want to keep one around
because certain operations are going to be faster
with the transpose, and certain operations
are going to be faster with the matrix.
We do the exact same thing here, and combined
with our exploded schema, now every single one
of those columns is a row.
And now we have indexed every single string
in the entire database.
Once, with one simple schema of two tables.
And not to say that all your databases will only
have two tables, but they might be three tables or four
tables-- is typically what we see,
which is a huge reduction compared
to the standard database schemas that one
uses where it's easy to have dozens and dozens of tables,
even 100 of tables in a well thought out schema.
Here, essentially it's one schema to rule them all.
One table and its pair, and you've
indexed every single thing.
And the cost is, you've doubled your storage.
And every single database person that you
talk to-- if I said in exchange for doubling your storage I
would give you a fast index access to every single string
in your database, they'll usually
take that deal 100 times out of 100.
So that's the power here of this technology,
is you can reference enormous data sets.
And we'll come back to this schema over and over again.
But that's sort of the madness.
We'll be-- as we move forward with associate arrays we always
view things in this way.
And I want you to know, at the end,
there's a really good reason why we do that.
So now we've ingested our data into the database
and we can do queries.
So if I have a binding to a table-- and one
of the things in the deformed technology
that we do that's very nice for you,
is if you're using a table and its transpose,
we hide that for you completely.
If you do an insert into the table,
it will automatically insert its transpose.
And if you do a look-up on rows, it will know to use one table,
and it does a look-up on columns,
it will know to use the other table.
So here we're doing essentially a range query.
This just says give me all the data from this range.
Time stamp May whatever to, it looks
like about a 3 hour time window.
So that's a query into the database,
and it returns an associative array of keys.
And here is what that looks like.
It looks like a bunch of triples.
I'm going to then take the rows of those keys.
So basically I found every single row,
within a particular time stamp, and I
want to get the whole row.
So I can get the row of that and pass that back into the table
to get the actual data.
Now you see I get the whole row.
So I get server IP, and all that stuff, the complete list.
Now I'm going to do a little algebra here.
I want to create the graph.
It's all source IPs and server IPs.
I get to do this with this basically correlation here,
which is essentially a matrix-matrix multiply.
That's it, I've just constructed the entire source IP and server
IP graph from all the data in this time window.
That is the power of D4M.
It allows you to do correlations on this kind of data,
same way we do correlations using traditional techniques
in linear algebra.
And you can be like, "Wow."
This is motivational at this point.
We'll get into exactly how you do this,
and exactly how this works in the example.
But this is really what people use
D4M for, is to get to this point where can do
these types of manipulations.
And the key to deduction theory is doing correlations.
Correlations allow us to determine
what the backgrounds of our data are,
what the clutter of our data are,
and then we are on our way with traditional detection
theory, and signal processing techniques, and all the things
that we know we love.
So this is kind of that connection made manifest.
And so here you can see, we now have an associative array
or an adjacency matrix that eventually, essentially
sorts IPs and their server IPs.
You can actually plot that and that's
how we get the adjacency matrix of this graph,
G. This is what it is.
That's how we constructed-- this is the code we actually
used to construct that graph.
So moving on here.
So this just shows the kind of the whole thing here.
We had a whole week's worth of proxy data.
This just shows you the timing, that it took about two hours
to ingest it, and about three hours to do this processing.
100 million proxy logs, 44.5 billion triples.
This is big data.
You know, really this is big data.
This is the kind of thing that you can do.
Whatever anybody else is doing, you
should be able to do on larger scales than other people
using this technology.
So that brings us to the end of the lecture.
So just to summarize, big data of this type
is found in a wide range of areas, document analysis,
computer networks, DNA sequencing.
There's a kind of a gap between the tools
that people traditionally use to do this problem,
and D4M, this technology, fills this gap.
So, with that, are there any questions
before we get onto the examples slide?
Yes?
AUDIENCE: [INAUDIBLE]?
JEREMY KEPNER: Yes.
AUDIENCE: [INAUDIBLE]?
JEREMY KEPNER: So the data--
AUDIENCE: Can you repeat the question?
JEREMY KEPNER: Oh yes, the question
was does it generalize to multi-dimensions, I guess?
And so, there may be kind of two questions there.
If you had-- if I just correlated all the data,
took that table out, and just fully correlated it,
I would end up with a giant matrix
of blocks, each one representing different correlations.
So the source IP, server IP correlation, the source IP,
source IP correlation, which would be empty
because it's a bipartite graph.
And then all the other correlations
would form different blocks, and that type of thing.
So in that sense we support multi-dimensions.
Whether we support-- but supporting multi-dimensions
in terms of tensors, OK?
Well, generally in sparse linear algebra,
there's very minimal support for higher dimensions.
I'll put a little shout out to my friend Tammy
Kolda, who has a technology called the Sparse Tensor
Toolbox.
She's a scientist at Sandia and so I encourage you to do that.
But all of our stuff is in two dimensions
from a linear algebra perspective.
Also linear algebraically, the math
there is a little bit more common than when
you get into the tensor.
And then you have these kind of weird products
that you're starting, which you can define, but are just
not commonly taught.
So and you might be like, well you're really
losing something there.
I'd really like to have multiple dimensions.
Well, when we've looked at it, the fact of the matter
is the sparsity always increases as you go to higher dimensions.
And the sparsity is sparse enough.
So typically the kinds of data that we
will have-- if you have n vertices,
you'll have maybe 10 times n edges.
So the sparsity is extremely low.
We added a dimension.
This sort of internal data structure costs
associated with adding that dimension would not
reap the benefits.
Because now your sparsity is even lower.
And so that's generally not something
that is commonly used.
But, if you're interested exploring that
I certainly encourage people to download that toolbox.
Very well written software, and people can explore that.
Other questions?
That was an outstanding question.
And other questions at this time?
All right well why don't we stop the recording
right now, and then we will switch-- why don't we
take like a five minute break.
I think there's some water out there and stuff for people.
And then we'll get back to the examples, which probably
will be a lot more fun.
So hopefully wake you up after these new graphs.