Face motion capture - Unreal Engine Tutorial

- Motion capturing my face, and bringing that over to the Meta Human, how cool is that? Well, it's actually super easy to do. So, let's get started (claps) with this really fun and short lesson. Now, when we imported the Meta Human, we were already prompted with a message saying, hey, we need some plugins in order to make the Meta Human work. If you have dismissed that, instead of clicking on enable those plugins, we have to go to the plugin window to enable them back. Let me just show you where that is. On the top here, click on Settings. We've been here before to change the project settings, for if you want to change your key bindings and all. And, right here, you can find an option called Plugins. Clicking on that will open up the plugin library. From here, we can just enable any component that we need. Now, by default, a whole bunch of plugins have already been enabled, because we picked out a blank video production project, instead of a blank game production, or game development project, which, on their end, is going to enable and disable other plugins. Now, by default, you don't have that motion capture plugin enabled, so let's look for that. Just search for Live Link on top, that is the plugin that we need. And here it is Live Link, let's enable that. And you also want to enable Live Link Control Rig, which already seems to be enabled, probably while I was preparing these lessons. All right, and then you want to click here on Restart Now. Every time that you're going to enable or disable plugins, you need to restart the engine, but that's okay, everything is still running fine. And, with Live Link enabled, we can now close the plugin library. We can now go over to the Window on top, go to Virtual Production, and from there, find Live Link. Click on that, which will open up a new panel. And from there, we have to select the device that is going to capture the motion. And in this case, that device is going to be the Apple iPhone. And unfortunately it only works with iPhone, and that is because the app that we're going to work with is called Live Link Face, which is created by Unreal Engine to make this whole process super easy. Of course, you can start creating your own blueprints and all, and make sure motion capture suit from any brand work with Unreal. The Meta Human is already prepared to do that, but again, you're going to need knowledge on how blueprints work, or you just use this app, and you don't need to create any blueprints. So, let's open up the Live Link Face, and straight away, you should see yourself, with this weird mesh, which actually is the motion capture. Now we are going to have to go to Settings here on top, tap on that, and you'll see here, Live Link on top. It currently says Jordy_iPhone, tap on it, and right here, you'll find Add Target. Tap on that. And you want to put in the IP address of your computer that is running the Unreal Engine. Now, there are multiple ways to find the IP address on your computer. If you have absolutely no idea where to find that, well, here's one way of locating it. If you go back to the Settings, then go into Project Settings, we can scroll all the way down here and locate UDP Messaging, right here. Click on it, and you'll find here, Unicast Endpoint. And by the way, you want to make sure that this here is enabled, otherwise things might not work, but by default, it's already enabled, so no need to worry about that. But anyways, here, Unicast Endpoint, there are two always default IP addresses in there, one with all the zeros, and one with the 127. And then the last one here is the IP address of your computer, in my case, that's 10.0.0.60. Okay, don't change anything in here, let's just close that window, because we need to enter that IP address into the app. So, let me just do that. 10.0.0.60. There we go. Click Add. And now you should see your iPhone appear in the Live Link window. If it doesn't, then it's probably like a firewall issue or something, so try to look at your network settings, but it should, it should appear in there, and there's nothing else that we need to do, just make sure that it's there. What we can now do is simply select the Meta Human itself, and here, under Settings, you'll see Link Face Subjects, and if you open up that dropdown menu, you should see your iPhone in there, so select that. And we're also going to select Link Face Head. And these are two different motion trackings, one is the face itself, the expressions, and the other one is the head movement. Now, also in the app, you have to make sure to go to the settings, and make sure here that Stream Head Rotation is enabled, otherwise, it won't stream the head movement capture. Right, everything is set, we can now go into the simulation mode. We are currently here just in the editor mode, the mode in which we build our environments. But on top of here, we can start playing the game, or go into simulation mode. Now, if you click here on the menu, we actually have some different options on how we would like to simulate this. We can start simulating it in a new viewport, as a standalone game. Obviously, these are for when you're developing a game, but we are using Unreal Engine for something different, so we're usually going to just pick Simulate. Click on that. And there we go, we are now in simulation mode. Look at this! (laughs) (Jordy babbling) (Jordy laughing) This is so cool. It just works straight out of the box, and it's so fun to play around with. Look at me, that's me, that's my Meta Human. And I can just turn my head around, it will just follow that movement. I can talk, and it will also follow my expressions and all. Really fun to play around with. Now, let me talk a little bit more about this whole simulation mode. Let me just put my phone over there. To go out of this simulation mode, and I can just talk to you guys like this now inside of the screencast, but anyways, to go out of it, simply hit the Escape key on your keyboard, or here, click the Stop simulation button, which will also take you out. So, these are the two different modes, editor mode and simulation mode. But why is there a simulation mode? Well, we just can't do everything in editor modes. And to showcase that better to you, let's create an object with some gravity. For instance, this chair here in the back, I'm just going to zoom in a little bit more on that. Let's say that we want to let that chair fall. Maybe that's part of our scene. Now, if this chair would fall in the editor mode, then it's just impossible to create your environment. We don't want everything to keep falling as we are building something. Now, by default, if we go back into simulation mode here, that chair won't fall. And that is because we haven't given any physics to it yet. So, let's go out of the simulation mode. Now, with that chair selected here, we can then go into the Details panel, and you should find an option here called Physics. And here, we have the option to simulate physics. Now, unfortunately, we cannot enable that option, because we haven't set any collisions yet. You see, an object, or a 3D model, just flies through other 3D models, it doesn't have a collision. We need to give an area to that chair to tell Unreal, hey, this area has to bump into other 3D models and not go through it. Now, that can be done from within the mesh itself. So, if we scroll up here, we should find Static Mesh right here. If we double-click on it, we should see the chair itself. And on top here, you'll see an option called Collision. Click on it, and we have a couple of options. I'm just going to pick the very simple Box Collision. As you'll see, that will just kind of like draw a cube around my chair, and that is going to act as the collision box. Now, obviously, you want that to be a little bit more precise, so we could add multiple boxes in here to go more around the legs of that chair and everything. But for this example, that box is fine. And also you need to keep in mind that, the more collision boxes you create, or the more complex that becomes, the heavier it'll also be for your computer, and we've seen with the engine what that can bring. So, let's save this mesh right here so that the collision box is saved with the mesh. Close that window, and now we should be able to enable that Simulate Physics option. There we go. And we could even set a certain mass, and that's going to define how heavy that object is, and how fast it should fall down. A chair, what is that? I don't know, like 20 kilograms or something, I'm not sure. Anyways, if we go now into the simulation mode, you'll see that the chair will drop and fall on the floor. So, that's the reason why we have a simulation mode and an editor mode. Now, already good to know is that once we're going to make an export to actually render this out as a movie that we can use in something like Premier to edit with, it's always going to run through that simulation mode. So if you want to film something like a chair that is falling, don't worry, once you've set everything correctly, and it's working in the simulation mode, you can also be sure that it will be captured once you're going to render out your video. All right, and that was the basics of Live Link, how we can use that, and, of course, the difference between editor and simulation mode. In the next lesson, we're going to place some lighting, because, let's be honest, this indoor scene looks like crap.