Adding options to enable auto blink, auto breath and random motion
[facial-landmarks-for-cubism.git] / README.md
CommitLineData
830d0ba4
AIL
1# Facial Landmarks for Cubism
2
3A library that extracts facial landmarks from a webcam feed and converts them
4into Live2D® Cubism SDK parameters.
5
6*Disclaimer: This library is designed for use with the Live2D® Cubism SDK.
7It is not part of the SDK itself, and is not affiliated in any way with Live2D
8Inc. The Live2D® Cubism SDK belongs solely to Live2D Inc. You will need to
9agree to Live2D Inc.'s license agreements to use the Live2D® Cubism SDK.*
10
11This block diagram shows the intended usage of this library:
12
13![Block diagram showing interaction of this library with other components](block_diagram.png)
14
15Video showing me using the example program:
c175b891 16<https://youtu.be/SZPEKwEqbdI>
830d0ba4
AIL
17
18
19## Supporting environments
20
21This library was developed and tested only on Ubuntu 18.04 using GCC 7.5.0.
22However I don't think I've used anything that prevents it from being
23cross-platform compatible -- it should still work as long as you have a
2b1f0c7c 24recent C/C++ compiler. The library should only require C++11. The Cubism
830d0ba4
AIL
25SDK requires C++14. I have made use of one C++17 library (`<filesystem>`)
26in the example program, but it should be straightforward to change this
27if you don't have C++17 support.
28
29I have provided some shell scripts for convenience when building. In an
30environment without a `/bin/sh` shell you may have to run the commands
31manually. Hereafter, all build instructions will assume a Linux environment
32where a shell is available.
33
2b1f0c7c
AIL
34If your CPU does not support AVX instructions you may want to edit "build.sh"
35and "example/demo.patch" to remove the `-D USE_AVX_INSTRUCTIONS=1` variable
36(or change AVX to SSE4 or SSE2). However there could be a penalty in
37performance.
830d0ba4
AIL
38
39## Build instructions
40
411. Install dependencies.
42
43 You will require a recent C/C++ compiler, `make`, `patch`, CMake >= 3.16,
44 and the OpenCV library (I'm using version 4.3.0). To compile the example
45 program you will also require the OpenGL library (and its dev headers)
46 among other libraries required for the example program. The libraries I
47 had to install (this list may not be exhaustive) are:
48
49 libgl1-mesa-dev libxrandr-dev libxinerama-dev libxcursor-dev libxi-dev libglu1-mesa-dev
50
512. Clone this repository including its submodule (dlib)
52
53 git clone --recurse-submodules https://github.com/adrianiainlam/facial-landmarks-for-cubism.git
54
2b1f0c7c
AIL
553. To build the library only: (Skip this step if you want to build the example
56 program. It will be done automatically.)
830d0ba4
AIL
57
58 cd <path of the git repo>
59 ./build.sh
60
614. You will require a facial landmark dataset to use with dlib. I have
62 downloaded mine from
63 <http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2>.
64 Extract the file and edit the "config.txt" file to point to the
65 path to this file.
66
67 Note: The license for this dataset excludes commercial use. If you want
68 to use this library in a commercial product you will need to obtain a
69 dataset in some other way.
70
71To build the example program:
72
735. Copy the extracted dlib dataset from step 4 to the "example" folder
74 of this repo.
75
766. Download "Cubism 4 SDK for Native R1" from the Live2D website:
77 <https://www.live2d.com/en/download/cubism-sdk/download-native/>.
78
79 Extract the archive -- put the "CubismSdkForNative-4-r.1" folder under
80 the "example" folder of this repo.
81
82 Note: The Cubism SDK is the property of Live2D and is not part of this
83 project. You must agree to Live2D's license agreements to use it.
84
857. Go into the
86 "example/CubismSdkForNative-4-r.1/Samples/OpenGL/thirdParty/scripts"
87 directory and run
88
89 ./setup_glew_glfw
90
918. Go back to the "example" directory and run
92
93 ./build.sh
94
959. Now try running the example program. From the "example" directory:
96
97 cd ./demo_build/build/make_gcc/bin/Demo/
98 ./Demo
99
100
101## Command-line arguments for the example program
102
103Most command-line arguments are to control the Cubism side of the program.
104Only one argument (`--config`) is used to specify the configuration file
105for the Facial Landmarks for Cubism library.
106
107 * `--window-width`, `-W`: Specify the window width
108 * `--window-height`, `-H`: Specify the window height
109 * `--window-title`, `-t`: Specify the window title
110 * `--root-dir`, `-d`: The directory at which the "Resources" folder will
111 be found. This is where the model data will be located.
112 * `--scale-factor`, `-f`: How the model should be scaled
113 * `--translate-x`, `-x`: Horizontal translation of the model within the
114 window
115 * `--translate-y`, `-y`: Vertical translation of the model within the window
116 * `--model`, `-m`: Name of the model to be used. This must be located inside
117 the "Resources" folder.
118 * `--config`, `-c`: Path to the configuration file for the Facial Landmarks
119 for Cubism library. See below for more details.
120
121
122## Configuration file
123
124Due to the differences in hardware and differences in each person's face,
125I have decided to make pretty much every parameter tweakable. The file
126"config.txt" lists and documents all parameters and their default values.
127You can change the values there and pass it to the example program using
2b1f0c7c 128the `-c` argument. If using the library directly, the path to this file
830d0ba4
AIL
129should be passed to the constructor (or pass an empty string to use
130default values).
131
2b1f0c7c
AIL
132## Troubleshooting
133
1341. Example program crashes with SIGILL (Illegal instruction).
135
136 Your CPU probably doesn't support AVX instructions which is used by dlib.
137 You can confirm this by running
138
139 grep avx /proc/cpuinfo
140
141 If this is the case, try to find out if your CPU supports SSE4 or SSE2,
142 then edit "build.sh" and "example/demo.patch" to change
143 `USE_AVX_INSTRUCTIONS=1` to `USE_SSE4_INSTRUCTIONS=1` or
144 `USE_SSE2_INSTRUCTIONS=1`.
145
830d0ba4
AIL
146## License
147
148The library itself is provided under the MIT license. By "the library itself"
149I refer to the following files that I have provided under this repo:
150
151 * src/facial_landmark_detector.cpp
152 * src/math_utils.h
153 * include/facial_landmark_detector.h
154 * and if you decide to build the binary for the library, the resulting
155 binary file (typically build/libFacialLandmarksForCubism.a)
156
157The license text can be found in LICENSE-MIT.txt, and also at the top of
158the .cpp and .h files.
159
160The library makes use of the dlib library, provided here as a Git
161submodule, which is used under the Boost Software License, version 1.0.
162The full license text can be found under lib/dlib/dlib/LICENSE.txt.
163
164The example program is a patched version of the sample program provided
165by Live2D (because there's really no point in reinventing the wheel),
166and as such, as per the licensing restrictions by Live2D, is still the
167property of Live2D.
168
169The patch file (example/demo.patch) contains lines showing additions by
170me, as well as deleted lines and unchanged lines for context. The deleted
171and unchanged lines are obviously still owned by Live2D. For my additions,
172where substantial enough for me to claim ownership, I release them under
173the Do What the Fuck You Want to Public License, version 2. The full license
174text can be found in LICENSE-WTFPL.txt.
175
176All other files not mentioned above that I have provided in this repo
177(i.e. not downloaded and placed here by you), *excluding* the two license
178documents and files generated by Git, are also released under the Do What
179the Fuck You Want to Public License, version 2, whose full license text
180can be found in LICENSE-WTFPL.txt.
181
182In order to use example program, or in any other way use this library
183with the Live2D® Cubism SDK, you must agree to the license by Live2D Inc.
184Their licenses can be found here:
185<https://www.live2d.com/en/download/cubism-sdk/download-native/>.
186
187The library requires a facial landmark dataset, and the one provided by
188dlib (which is derived from a dataset owned by Imperial College London)
189has been used in development. The license for this dataset excludes
190commercial use. You must obtain an alternative dataset if you wish to
191use this library commercially.
192
193This is not a license requirement, but if you find my library useful,
194I'd love to hear from you! Send me an email at spam(at)adrianiainlam.tk --
195replacing "spam" with the name of this repo :).
196
197## Contributions
198
199Contributions welcome! This is only a hobby weekend project so I don't
200really have many environments / faces to test it on. Feel free to submit
201issues or pull requests on GitHub, or send questions or patches to me
202(see my email address above) if you prefer email. Thanks :)
203