Introduction to ARCore Augmented Faces, Unity

BO PANG: Hi, I’m Bo. I’m a software
engineer on ARCore, Google’s Platform for building
augmented reality experience. I’ll be showing you how to
implement face effect using the Augmented Faces API in
ARCore with the Unity engine. Links to Android and iOS
versions of this video can be found in the description. Augmented Faces doesn’t
require a special helper, such as a depth sensor. Instead, it use phone’s
camera and machine learning to provide three
pieces of information in real time synchronized
with the camera feed. First, a center pose
for a face, which helps you do things
like render a head. Second, region poses, which
are useful for placing assets on or near the forehead or nose. And third, a 468 point
dense 3D face mesh, which allow you to paint
detail texture that accurately follow facial movement. So, pretty cool, right? Now, let’s see how you can
utilize the Augmented Faces feature in ARCore to build your
own app using Unity engine. You will need to
download the ARCore Unity SDK from our GitHub Res page and
import it into a Unity Project as a custom package. To start your project
for ARCore development, make sure to set the Unity
target platform to Android and enable ARCore
supported checkbox in Unity Player Settings. Next, let’s see how to enable
ARCore in your own Unity scene. First, delete the main camera
and add ARCore device prefab into the scene. You will notice that the prefab
contains two game object. The ARCore device game object
uses ARCore session component to manage the ARCore
session lifecycle. An ARCore session can be passed
by disabling this component. The first person
camera game object serves as the main
camera of the scene. ARCore automatically update
transforms each frame based on the device movement. It is also configured to render
the camera image on the screen. Now, let’s set up
the ARCore session. First, create a new ARCore
session config asset inside the project window. Let’s call this augmented
face session config. Next, select the
session config asset. Disable plane finding and light
estimation as those features are not compatible with
augmented face yet. Then set the Augmented
Face mode to mesh. Now, this session config is
set up for Augmented Faces. Select the ARCore device
game object and drag the session config asset into
the session config field. Once you have done
that, you will notice there is an error at the
bottom saying augmented face mode on mesh require
front facing camera. This is true. So let’s change the
device camera direction to front facing on
ARCore session component. With the current
scene setup, the app will start the ARCore session
and render the front camera image on the screen. Even though it is set up to
detect face in the camera view port, we can’t see any effects
because we haven’t put anything there yet. Now, let’s add the example fox
effect from the ARCore Unity SDK to your app. First, let’s create a root game
object called face attachment to hold our fox face
effect game objects. Next, let’s add the 3D
meshes from the fox effect and configure it to update
with the face regions. Drag the fox sample
FBX to create an instance inside the
face attachment object. This FBX contains
fox ears and nose models, which are connected
to a predefined face rig. This makes sure those
models are always attached to the face regions. You can create
similar face assets using the fox sample FBX and
the canonical face mash FBX in ARCore unity
SDK as a reference. Now, add the ARCore
augmented face rig component to the fox sample game object
and check the Auto Bind checkbox. With Auto Bind
enabled, this component will automatically connect
to the augmented face object detected by ARCore
and update its face rig using face region poses. Next, let’s add the fox
face or texture overlay. Inside the face
attachment object, create a new game object
called face texture. Add the ARCore Augmented
Faces mesh filter component and check the Auto
Bind checkbox. With Auto Bind
enabled, this component will automatically
connect to the detect face and generate a 3D face mesh. Here, we want to use
the face mesh to render a texture on the face. So let’s add the
mesh render component and set it to use the
fox face material. Now, build and run the app on
an ARCore compatible Android phone. You’ll see that
the fox face effect is overlaid on your face. Now you have everything
you need to get start with ARCore augmented
face in Unity, you’ll be able to use your own
assets to create a custom face effect. For more information about this
tutorial and ARCore in general, you can check out our
sample app source code and documentation in
the description below. We can’t wait to
see what you build and hope you share it with
using the ARCore hashtag. Thank you for watching.

15 Replies to “Introduction to ARCore Augmented Faces, Unity”

  1. Is there a way to distinguish this from reality? Because getting the ability to create fake videos indistinguishable from real, on a handheld device is moving into some pretty scary territory.

  2. Is there any reason why it can only be used with the front-facing camera? I think that restricts our creativity with this feature.

Leave a Reply

Your email address will not be published. Required fields are marked *