Commit | Line | Data |
---|---|---|
830d0ba4 AIL |
1 | # Facial Landmarks for Cubism |
2 | ||
3 | A library that extracts facial landmarks from a webcam feed and converts them | |
4 | into Live2D® Cubism SDK parameters. | |
5 | ||
6 | *Disclaimer: This library is designed for use with the Live2D® Cubism SDK. | |
7 | It is not part of the SDK itself, and is not affiliated in any way with Live2D | |
8 | Inc. The Live2D® Cubism SDK belongs solely to Live2D Inc. You will need to | |
9 | agree to Live2D Inc.'s license agreements to use the Live2D® Cubism SDK.* | |
10 | ||
11 | This block diagram shows the intended usage of this library: | |
12 | ||
13 | ![Block diagram showing interaction of this library with other components](block_diagram.png) | |
14 | ||
15 | Video showing me using the example program: | |
16 | <https://www.youtube.com/watch?v=Ve9GaeyMytc> | |
17 | ||
18 | ||
19 | ## Supporting environments | |
20 | ||
21 | This library was developed and tested only on Ubuntu 18.04 using GCC 7.5.0. | |
22 | However I don't think I've used anything that prevents it from being | |
23 | cross-platform compatible -- it should still work as long as you have a | |
24 | recent C/C++ compiler. (The library should only require C++11. The Cubism | |
25 | SDK requires C++14. I have made use of one C++17 library (`<filesystem>`) | |
26 | in the example program, but it should be straightforward to change this | |
27 | if you don't have C++17 support. | |
28 | ||
29 | I have provided some shell scripts for convenience when building. In an | |
30 | environment without a `/bin/sh` shell you may have to run the commands | |
31 | manually. Hereafter, all build instructions will assume a Linux environment | |
32 | where a shell is available. | |
33 | ||
34 | If your CPU does not support AVX instructions you may want to edit the | |
35 | relevant build scripts to remove the `-D USE_AVX_INSTRUCTIONS=1` variable | |
36 | (or change it the SSE3 etc). However there could be a penalty in performance. | |
37 | ||
38 | ## Build instructions | |
39 | ||
40 | 1. Install dependencies. | |
41 | ||
42 | You will require a recent C/C++ compiler, `make`, `patch`, CMake >= 3.16, | |
43 | and the OpenCV library (I'm using version 4.3.0). To compile the example | |
44 | program you will also require the OpenGL library (and its dev headers) | |
45 | among other libraries required for the example program. The libraries I | |
46 | had to install (this list may not be exhaustive) are: | |
47 | ||
48 | libgl1-mesa-dev libxrandr-dev libxinerama-dev libxcursor-dev libxi-dev libglu1-mesa-dev | |
49 | ||
50 | 2. Clone this repository including its submodule (dlib) | |
51 | ||
52 | git clone --recurse-submodules https://github.com/adrianiainlam/facial-landmarks-for-cubism.git | |
53 | ||
54 | 3. To build the library only: | |
55 | ||
56 | cd <path of the git repo> | |
57 | ./build.sh | |
58 | ||
59 | 4. You will require a facial landmark dataset to use with dlib. I have | |
60 | downloaded mine from | |
61 | <http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2>. | |
62 | Extract the file and edit the "config.txt" file to point to the | |
63 | path to this file. | |
64 | ||
65 | Note: The license for this dataset excludes commercial use. If you want | |
66 | to use this library in a commercial product you will need to obtain a | |
67 | dataset in some other way. | |
68 | ||
69 | To build the example program: | |
70 | ||
71 | 5. Copy the extracted dlib dataset from step 4 to the "example" folder | |
72 | of this repo. | |
73 | ||
74 | 6. Download "Cubism 4 SDK for Native R1" from the Live2D website: | |
75 | <https://www.live2d.com/en/download/cubism-sdk/download-native/>. | |
76 | ||
77 | Extract the archive -- put the "CubismSdkForNative-4-r.1" folder under | |
78 | the "example" folder of this repo. | |
79 | ||
80 | Note: The Cubism SDK is the property of Live2D and is not part of this | |
81 | project. You must agree to Live2D's license agreements to use it. | |
82 | ||
83 | 7. Go into the | |
84 | "example/CubismSdkForNative-4-r.1/Samples/OpenGL/thirdParty/scripts" | |
85 | directory and run | |
86 | ||
87 | ./setup_glew_glfw | |
88 | ||
89 | 8. Go back to the "example" directory and run | |
90 | ||
91 | ./build.sh | |
92 | ||
93 | 9. Now try running the example program. From the "example" directory: | |
94 | ||
95 | cd ./demo_build/build/make_gcc/bin/Demo/ | |
96 | ./Demo | |
97 | ||
98 | ||
99 | ## Command-line arguments for the example program | |
100 | ||
101 | Most command-line arguments are to control the Cubism side of the program. | |
102 | Only one argument (`--config`) is used to specify the configuration file | |
103 | for the Facial Landmarks for Cubism library. | |
104 | ||
105 | * `--window-width`, `-W`: Specify the window width | |
106 | * `--window-height`, `-H`: Specify the window height | |
107 | * `--window-title`, `-t`: Specify the window title | |
108 | * `--root-dir`, `-d`: The directory at which the "Resources" folder will | |
109 | be found. This is where the model data will be located. | |
110 | * `--scale-factor`, `-f`: How the model should be scaled | |
111 | * `--translate-x`, `-x`: Horizontal translation of the model within the | |
112 | window | |
113 | * `--translate-y`, `-y`: Vertical translation of the model within the window | |
114 | * `--model`, `-m`: Name of the model to be used. This must be located inside | |
115 | the "Resources" folder. | |
116 | * `--config`, `-c`: Path to the configuration file for the Facial Landmarks | |
117 | for Cubism library. See below for more details. | |
118 | ||
119 | ||
120 | ## Configuration file | |
121 | ||
122 | Due to the differences in hardware and differences in each person's face, | |
123 | I have decided to make pretty much every parameter tweakable. The file | |
124 | "config.txt" lists and documents all parameters and their default values. | |
125 | You can change the values there and pass it to the example program using | |
126 | to `-c` argument. If using the library directly, the path to this file | |
127 | should be passed to the constructor (or pass an empty string to use | |
128 | default values). | |
129 | ||
130 | ## License | |
131 | ||
132 | The library itself is provided under the MIT license. By "the library itself" | |
133 | I refer to the following files that I have provided under this repo: | |
134 | ||
135 | * src/facial_landmark_detector.cpp | |
136 | * src/math_utils.h | |
137 | * include/facial_landmark_detector.h | |
138 | * and if you decide to build the binary for the library, the resulting | |
139 | binary file (typically build/libFacialLandmarksForCubism.a) | |
140 | ||
141 | The license text can be found in LICENSE-MIT.txt, and also at the top of | |
142 | the .cpp and .h files. | |
143 | ||
144 | The library makes use of the dlib library, provided here as a Git | |
145 | submodule, which is used under the Boost Software License, version 1.0. | |
146 | The full license text can be found under lib/dlib/dlib/LICENSE.txt. | |
147 | ||
148 | The example program is a patched version of the sample program provided | |
149 | by Live2D (because there's really no point in reinventing the wheel), | |
150 | and as such, as per the licensing restrictions by Live2D, is still the | |
151 | property of Live2D. | |
152 | ||
153 | The patch file (example/demo.patch) contains lines showing additions by | |
154 | me, as well as deleted lines and unchanged lines for context. The deleted | |
155 | and unchanged lines are obviously still owned by Live2D. For my additions, | |
156 | where substantial enough for me to claim ownership, I release them under | |
157 | the Do What the Fuck You Want to Public License, version 2. The full license | |
158 | text can be found in LICENSE-WTFPL.txt. | |
159 | ||
160 | All other files not mentioned above that I have provided in this repo | |
161 | (i.e. not downloaded and placed here by you), *excluding* the two license | |
162 | documents and files generated by Git, are also released under the Do What | |
163 | the Fuck You Want to Public License, version 2, whose full license text | |
164 | can be found in LICENSE-WTFPL.txt. | |
165 | ||
166 | In order to use example program, or in any other way use this library | |
167 | with the Live2D® Cubism SDK, you must agree to the license by Live2D Inc. | |
168 | Their licenses can be found here: | |
169 | <https://www.live2d.com/en/download/cubism-sdk/download-native/>. | |
170 | ||
171 | The library requires a facial landmark dataset, and the one provided by | |
172 | dlib (which is derived from a dataset owned by Imperial College London) | |
173 | has been used in development. The license for this dataset excludes | |
174 | commercial use. You must obtain an alternative dataset if you wish to | |
175 | use this library commercially. | |
176 | ||
177 | This is not a license requirement, but if you find my library useful, | |
178 | I'd love to hear from you! Send me an email at spam(at)adrianiainlam.tk -- | |
179 | replacing "spam" with the name of this repo :). | |
180 | ||
181 | ## Contributions | |
182 | ||
183 | Contributions welcome! This is only a hobby weekend project so I don't | |
184 | really have many environments / faces to test it on. Feel free to submit | |
185 | issues or pull requests on GitHub, or send questions or patches to me | |
186 | (see my email address above) if you prefer email. Thanks :) | |
187 |