# Facial Landmarks for Cubism
-A library that extracts facial landmarks from a webcam feed and converts them
+A library that extracts facial landmarks from a webcam feed (computed by [OpenSeeFace](https://github.com/emilianavt/OpenSeeFace)) and converts them
into Live2D® Cubism SDK parameters.
*Disclaimer: This library is designed for use with the Live2D® Cubism SDK.
Video showing me using the example program:
-<https://www.youtube.com/watch?v=Ve9GaeyMytc>
+<https://youtu.be/SZPEKwEqbdI>
+## Old version using dlib
+The old version using dlib is no longer actively maintained, but if
+you want to use it, you can get it on the [dlib-stable branch](https://github.com/adrianiainlam/facial-landmarks-for-cubism/tree/dlib-stable).
+
+## Spin-off: Mouse Tracking for Cubism
+
+An alternative version using mouse cursor tracking and audio based lip
+syncing instead of face tracking is available at
+<https://github.com/adrianiainlam/mouse-tracker-for-cubism>.
+
+The main advantage is a much lower CPU load.
## Supporting environments
-This library was developed and tested only on Ubuntu 18.04 using GCC 7.5.0.
-However I don't think I've used anything that prevents it from being
-cross-platform compatible -- it should still work as long as you have a
-recent C/C++ compiler. (The library should only require C++11. The Cubism
+This library was tested on Ubuntu 18.04 using GCC 7.5.0, and on
+Windows 10 using Visual Studio 2019.
+
+The library should only require C++11. The Cubism
SDK requires C++14. I have made use of one C++17 library (`<filesystem>`)
in the example program, but it should be straightforward to change this
if you don't have C++17 support.
-I have provided some shell scripts for convenience when building. In an
-environment without a `/bin/sh` shell you may have to run the commands
-manually. Hereafter, all build instructions will assume a Linux environment
-where a shell is available.
+## Build instructions
-If your CPU does not support AVX instructions you may want to edit the
-relevant build scripts to remove the `-D USE_AVX_INSTRUCTIONS=1` variable
-(or change it the SSE3 etc). However there could be a penalty in performance.
+1. Clone OpenSeeFace from <https://github.com/emilianavt/OpenSeeFace>,
+ refer to its documentation and install its dependencies.
-## Build instructions
+2. Install dependencies.
-1. Install dependencies.
+ On Linux:
- You will require a recent C/C++ compiler, `make`, `patch`, CMake >= 3.16,
- and the OpenCV library (I'm using version 4.3.0). To compile the example
+ You will require a recent C/C++ compiler, `make`, `patch`, and CMake >= 3.16. To compile the example
program you will also require the OpenGL library (and its dev headers)
among other libraries required for the example program. The libraries I
- had to install (this list may not be exhaustive) are:
+ had to install on Ubuntu 18.04 (this list may not be exhaustive) are:
libgl1-mesa-dev libxrandr-dev libxinerama-dev libxcursor-dev libxi-dev libglu1-mesa-dev
-2. Clone this repository including its submodule (dlib)
+ On Windows:
- git clone --recurse-submodules https://github.com/adrianiainlam/facial-landmarks-for-cubism.git
+ You will need a recent C/C++ compiler (I've tested Visual Studio 2019),
+ `patch`, CMake >= 3.16, and a `sh` shell (Git Bash works fine). I did not
+ need to install any other library, but I have not tested on a clean machine.
-3. To build the library only:
+3. Clone this repository
- cd <path of the git repo>
- ./build.sh
+ git clone https://github.com/adrianiainlam/facial-landmarks-for-cubism.git
-4. You will require a facial landmark dataset to use with dlib. I have
- downloaded mine from
- <http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2>.
- Extract the file and edit the "config.txt" file to point to the
- path to this file.
+4. To build the library only: (Skip this step if you want to build the example
+ program. It will be done automatically.)
- Note: The license for this dataset excludes commercial use. If you want
- to use this library in a commercial product you will need to obtain a
- dataset in some other way.
+ cd <path of the git repo>
+ ./build.sh
To build the example program:
-5. Copy the extracted dlib dataset from step 4 to the "example" folder
- of this repo.
-
-6. Download "Cubism 4 SDK for Native R1" from the Live2D website:
+5. Download "Cubism 5 SDK for Native R1" from the Live2D website:
<https://www.live2d.com/en/download/cubism-sdk/download-native/>.
- Extract the archive -- put the "CubismSdkForNative-4-r.1" folder under
+ Extract the archive -- put the "CubismSdkForNative-5-r.1" folder under
the "example" folder of this repo.
Note: The Cubism SDK is the property of Live2D and is not part of this
project. You must agree to Live2D's license agreements to use it.
-7. Go into the
- "example/CubismSdkForNative-4-r.1/Samples/OpenGL/thirdParty/scripts"
+6. Go into the
+ "example/CubismSdkForNative-5-r.1/Samples/OpenGL/thirdParty/scripts"
directory and run
./setup_glew_glfw
-8. Go back to the "example" directory and run
+7. Go back to the "example" directory and run
./build.sh
-9. Now try running the example program. From the "example" directory:
+ On Linux, the build should be complete after running this script.
+ On Windows, you will also need to go into "./demo_build/scripts"
+ and run one of the scripts corresponding to your MSVC version.
+ The script will ask for your configurations. I've tried "1 2 1"
+ which works fine for me.
+
+8. Now try running the example program.
+
+ First, (in a separate terminal) go to where you have downloaded
+ OpenSeeFace, and run
+
+ python3 ./facetracker.py -v 4 --model 3 -M
+
+ I tried models 3 and 4, and I think both work reasonably well.
+ Please feel free to explore other options provided by OSF.
+
+ Back to the original terminal, from the "example" directory:
+ (for Windows, replace "make_gcc" with your MSVC version)
cd ./demo_build/build/make_gcc/bin/Demo/
./Demo
* `--translate-y`, `-y`: Vertical translation of the model within the window
* `--model`, `-m`: Name of the model to be used. This must be located inside
the "Resources" folder.
+ * `--old-param-id`, `-o`: If set to 1, translate new (Cubism 3+) parameter
+ IDs to old (Cubism 2.1) IDs. This is necessary, for example, for
+ [the Chitose model available from Live2D](https://www.live2d.com/en/download/sample-data/).
* `--config`, `-c`: Path to the configuration file for the Facial Landmarks
for Cubism library. See below for more details.
I have decided to make pretty much every parameter tweakable. The file
"config.txt" lists and documents all parameters and their default values.
You can change the values there and pass it to the example program using
-to `-c` argument. If using the library directly, the path to this file
+the `-c` argument. If using the library directly, the path to this file
should be passed to the constructor (or pass an empty string to use
default values).
The license text can be found in LICENSE-MIT.txt, and also at the top of
the .cpp and .h files.
-The library makes use of the dlib library, provided here as a Git
-submodule, which is used under the Boost Software License, version 1.0.
-The full license text can be found under lib/dlib/dlib/LICENSE.txt.
-
The example program is a patched version of the sample program provided
by Live2D (because there's really no point in reinventing the wheel),
and as such, as per the licensing restrictions by Live2D, is still the
Their licenses can be found here:
<https://www.live2d.com/en/download/cubism-sdk/download-native/>.
-The library requires a facial landmark dataset, and the one provided by
-dlib (which is derived from a dataset owned by Imperial College London)
-has been used in development. The license for this dataset excludes
-commercial use. You must obtain an alternative dataset if you wish to
-use this library commercially.
-
This is not a license requirement, but if you find my library useful,
I'd love to hear from you! Send me an email at spam(at)adrianiainlam.tk --
replacing "spam" with the name of this repo :).
issues or pull requests on GitHub, or send questions or patches to me
(see my email address above) if you prefer email. Thanks :)
+Special thanks to GitHub user @Arkueid for adding Windows support!
projects / facial-landmarks-for-cubism.git / blobdiff