Demo - Semantic meaning of word embeddings

[00:00 - 00:04] OK, great. All right, so let me go back to the demo.

[00:05 - 00:12] And this demo is pretty important. We're going to interact with word vectors pretty much.

[00:13 - 00:16] OK, so here's what we're going to do. I am going to-- here we go.

[00:17 - 00:22] Now I've got everything I need. All right, let's explore some word embeddings.

[00:23 - 00:30] So to do this, I'm going to first import a few different utilities that we need. So let's explore some word embeddings.

[00:31 - 00:36] OK, sorry. Let's just say word vectors to be consistent with what we're saying before.

[00:37 - 00:42] OK, so we're going to import a few different libraries first. You can ignore what Gensum is.

[00:43 - 00:53] NLTK, though, is very, very commonly used for natural language processing, or NLP for short. NLTK stands for Natural Language Toolkit, and it contains a bunch of very, very useful utilities.

[00:54 - 01:02] So if there's any library that you take away from this presentation, there are going to be two of them. First is Transformers, which is hugging faces Transformers.

[01:03 - 01:07] Second is this one, L-T-K. A lot of useful utilities.

[01:08 - 01:16] OK, so we're going to import these. And the first thing we're going to do is actually download a focus here.

[01:17 - 01:26] So once that finishes downloading, we're also going to download a mapping. So I mentioned before, we had this mapping from you are cool to three different vectors.

[01:27 - 01:35] Word to VEC is a mapping that someone else has trained for us. So the mapping from words or tokens to the corresponding vectors.

[01:36 - 01:46] So we're going to actually do that now. So here is-- you have to find-- so we're going to look up this model first.

[01:47 - 01:55] And this model is going to be following. Once we looked up the model, we're then going to download it.

[01:56 - 02:09] So here is we're going to download, download and initialize it, load Word to VEC, and then-- for example. OK, perfect.

[02:10 - 02:18] So now let's go ahead and download that model. Once we've downloaded that model, we're then going to use this to actually transform words into the corresponding vectors.

[02:19 - 02:25] So here, for example, we'll start by translating king into this corresponding vector. And this is just in tactic sugar.

[02:26 - 02:38] This allows us to go from a word directly to a vector. Now, earlier though, I had said, yes, token IDs are basically indices and no big dictionary.

[02:39 - 02:45] So let me actually do-- and I'm friends, sorry-- let me actually access this vector in a slightly different way. That makes that more explicit.

[02:46 - 02:54] So every single word will have a corresponding index. And this is the ID, basically.

[02:55 - 03:03] So now what we can do is we can actually access the vector using the index. So this is the ID.

[03:04 - 03:12] This is the token ID that we've been talking about up above. And we actually access a list of vectors right here, or an array of vectors right here.

[03:13 - 03:20] So regardless of how we've got that, whether through the second method or the first, we're going to look at-- we'll get a vector out. So let's see what that vector looks like.

[03:21 - 03:27] Here, we're going to have a vector of 300 dimensions. That's a cool thing to say.

[03:28 - 03:34] I have an array of like 300. Let's see what's actually in this vector.

[03:35 - 03:39] So I'm looking at the first 10 elements here. And it's just a bunch of mumbo jumbo.

[03:40 - 03:47] As far as we can tell, these are basically random numbers. In a second, though, we're going to do something that shows you these numbers actually have some meaning.

[03:48 - 03:59] So I'm going to actually convert a few more words into vectors. So here, we're going to have man, and we're going to woman.

[04:00 - 04:02] OK, great. So now I've got all of these.

[04:03 - 04:07] We're going to do something strange. We're actually going to do arithmetic on these words.

[04:08 - 04:16] So we're going to write answer equals to king minus man plus one. And I'm going to pause here.

[04:17 - 04:21] So you can imagine your head, what do you think? King minus minus man plus woman.

[04:22 - 04:25] In theory, it should give you. What would you expect the word to be output?

[04:26 - 04:30] Or what word would you expect to be output? Let's ponder that for a second.

[04:31 - 04:33] Yes, exactly. Thank you all.

[04:34 - 04:36] Exactly. We expect queen to be outputted, right?

[04:37 - 04:40] So let me actually show you some code that does that. I'm just going to write the code.

[04:41 - 04:51] And then I'll come back to explain what this code does line by line. For now, you can just imagine at a high level, but this code is going to look for the most similar word.

[04:52 - 05:04] It's going to do the nearest neighbors that we talked about before to figure out what this answer vector corresponds to. So the actual details here don't matter right now.

[05:05 - 05:10] What matters is that once I run this code, I will get a word out. And we'll see what that word looks like.

[05:11 - 05:17] So that's this, it's not King Woman Man. And then we're going to do the following.

[05:18 - 05:23] All right. So we actually found the nearest neighbor for the vector answer.

[05:24 - 05:26] And we got queen out. So that's weird.

[05:27 - 05:35] This is really bizarre because addition and subtraction actually has a meaning. All right, we're going to say that it has semantic meaning.

[05:36 - 05:37] So let's do this a few more times, right? This seems neat.

[05:38 - 05:55] So let me show you a utility function that Word2Vec already provides for us. So this does the same thing that we did earlier, just with a different, this negative man, top man equals to 1.

[05:56 - 06:02] OK, so this gives us queen. And positive means that we add it, and negative means that we subtract it.

[06:03 - 06:09] So this is really equal to King minus man plus woman. OK, so now let's do a few more of these, right?

[06:10 - 06:16] If you've-- you might have heard of this example before, King minus woman plus man. But there are plenty other examples too.

[06:17 - 06:27] For example, what if we did Paris minus France plus Germany? All right, you can imagine what you might expect here.

[06:28 - 06:30] Get Berlin, right? You can do this one more time.

[06:31 - 06:45] So let's say that we do cat minus feline plus canine. All right, again, pause for a second here, where you think this would give you-- well, let me run the code.

[06:46 - 06:48] You get doc, right? Yeah, yes, exactly.

[06:49 - 06:49] Thank you. Thank you.

[06:50 - 06:52] So I love it when you guys participate. Sorry, when you all participate.

[06:53 - 06:55] Oh, definitely. Type as much as you can.

[06:56 - 06:59] OK, so we got this, right? So in your notebook, you can try something similar.

[07:00 - 07:05] You can call this function. And replace the positive and the negative with any set of words that you like.

[07:06 - 07:09] The question is, does it account for typos and commonness bellings? Not exactly.

[07:10 - 07:15] So word to effect is just a big, big dictionary of words. So if you type out a word that's not in that dictionary, it'll throw an error.

[07:16 - 07:20] So unfortunately, this is not quite that advanced. This is just a look up table for all of these factors.

[07:21 - 07:32] That's a good question, though. So I think actually, there are some weird ones, like color or theater might be spelled not American, something like that.

[07:33 - 07:36] I forgot what exactly. But some words-- well, I think actually, let me just try it now.

[07:37 - 07:45] I think gray-- yeah, gray not present. Yeah, OK, well, so this is American.

[07:46 - 07:47] Let's try theater. No, that's fine, too.

[07:48 - 07:54] OK, I don't remember those some words that were strange. But basically, as you saw earlier, some words may not show up.

[07:55 - 07:59] OK, so now let's do something else. We showed that adding and subtracting words makes sense.

[08:00 - 08:06] Now what I'm going to do is I'm going to take averages of words. So I'm not going to have any negative words.

[08:07 - 08:10] I'm just going to have positive. So there's going to be lunch, breakfast, dinner.

[08:11 - 08:19] And so in effect, what's happening here is I'm basically taking the average of these three words. So what do folks think is the average of lunch, breakfast, and dinner?

[08:20 - 08:23] What word would you expect? OK, so let's go ahead and do that.

[08:24 - 08:28] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[08:29 - 08:32] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[08:33 - 08:36] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[08:37 - 08:40] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[08:41 - 08:44] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[08:45 - 08:48] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[08:49 - 08:52] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[08:53 - 08:56] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[08:57 - 09:00] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:01 - 09:04] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:05 - 09:08] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:09 - 09:12] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:13 - 09:15] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:16 - 09:19] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:20 - 09:23] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:24 - 09:27] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:28 - 09:31] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:32 - 09:35] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:36 - 09:38] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:39 - 09:41] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:42 - 09:43] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:44 - 09:46] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:47 - 09:50] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:51 - 09:54] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:55 - 09:56] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:57 - 09:58] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[09:59 - 10:01] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[10:02 - 10:03] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[10:04 - 10:05] OK, so let's go ahead and do that. OK, so let's go ahead and do that.

[10:06 - 10:11] OK, so let's go ahead and do that. Now let's dive into this transformer.

[10:12 - 10:13] time.