Commit | Line | Data |
---|---|---|
830d0ba4 AIL |
1 | # Facial Landmarks for Cubism |
2 | ||
30b18873 | 3 | A library that extracts facial landmarks from a webcam feed (computed by [OpenSeeFace](https://github.com/emilianavt/OpenSeeFace)) and converts them |
830d0ba4 AIL |
4 | into Live2D® Cubism SDK parameters. |
5 | ||
6 | *Disclaimer: This library is designed for use with the Live2D® Cubism SDK. | |
7 | It is not part of the SDK itself, and is not affiliated in any way with Live2D | |
8 | Inc. The Live2D® Cubism SDK belongs solely to Live2D Inc. You will need to | |
9 | agree to Live2D Inc.'s license agreements to use the Live2D® Cubism SDK.* | |
10 | ||
11 | This block diagram shows the intended usage of this library: | |
12 | ||
13 | ![Block diagram showing interaction of this library with other components](block_diagram.png) | |
14 | ||
15 | Video showing me using the example program: | |
c175b891 | 16 | <https://youtu.be/SZPEKwEqbdI> |
830d0ba4 | 17 | |
30b18873 AIL |
18 | ## Old version using dlib |
19 | The old version using dlib is no longer actively maintained, but if | |
20 | you want to use it, you can get it on the [dlib-stable branch](https://github.com/adrianiainlam/facial-landmarks-for-cubism/tree/dlib-stable). | |
21 | ||
b3b6e2a1 AIL |
22 | ## Spin-off: Mouse Tracking for Cubism |
23 | ||
24 | An alternative version using mouse cursor tracking and audio based lip | |
25 | syncing instead of face tracking is available at | |
26 | <https://github.com/adrianiainlam/mouse-tracker-for-cubism>. | |
27 | ||
28 | The main advantage is a much lower CPU load. | |
830d0ba4 AIL |
29 | |
30 | ## Supporting environments | |
31 | ||
32 | This library was developed and tested only on Ubuntu 18.04 using GCC 7.5.0. | |
30b18873 AIL |
33 | |
34 | Currently it only supports Unix-like environments, as I am using | |
35 | `<sys/socket.h>` etc to communicate with OpenSeeFace. If there is demand | |
36 | for it, I can try to make it work on Windows as well (contributions | |
37 | welcome). | |
38 | ||
39 | The library should only require C++11. The Cubism | |
830d0ba4 AIL |
40 | SDK requires C++14. I have made use of one C++17 library (`<filesystem>`) |
41 | in the example program, but it should be straightforward to change this | |
42 | if you don't have C++17 support. | |
43 | ||
30b18873 | 44 | ## Build instructions |
830d0ba4 | 45 | |
30b18873 AIL |
46 | 1. Download OpenSeeFace from <https://github.com/emilianavt/OpenSeeFace>, |
47 | refer to its documentation and install its dependencies. | |
830d0ba4 | 48 | |
30b18873 AIL |
49 | Note: I have forked OSF and added a small feature (flipping the webcam |
50 | feed horizontally like a mirror). If you want, you can clone my fork | |
51 | instead: <https://github.com/adrianiainlam/OpenSeeFace>. | |
830d0ba4 | 52 | |
30b18873 | 53 | 2. Install dependencies. |
830d0ba4 | 54 | |
30b18873 | 55 | You will require a recent C/C++ compiler, `make`, `patch`, and CMake >= 3.16. To compile the example |
830d0ba4 AIL |
56 | program you will also require the OpenGL library (and its dev headers) |
57 | among other libraries required for the example program. The libraries I | |
e01443f7 | 58 | had to install on Ubuntu 18.04 (this list may not be exhaustive) are: |
830d0ba4 AIL |
59 | |
60 | libgl1-mesa-dev libxrandr-dev libxinerama-dev libxcursor-dev libxi-dev libglu1-mesa-dev | |
61 | ||
30b18873 | 62 | 3. Clone this repository |
e01443f7 | 63 | |
30b18873 | 64 | git clone https://github.com/adrianiainlam/facial-landmarks-for-cubism.git |
830d0ba4 | 65 | |
30b18873 | 66 | 4. To build the library only: (Skip this step if you want to build the example |
2b1f0c7c | 67 | program. It will be done automatically.) |
830d0ba4 AIL |
68 | |
69 | cd <path of the git repo> | |
70 | ./build.sh | |
71 | ||
830d0ba4 AIL |
72 | To build the example program: |
73 | ||
30b18873 | 74 | 5. Download "Cubism 4 SDK for Native R2" from the Live2D website: |
830d0ba4 AIL |
75 | <https://www.live2d.com/en/download/cubism-sdk/download-native/>. |
76 | ||
261d3d0e | 77 | Extract the archive -- put the "CubismSdkForNative-4-r.2" folder under |
830d0ba4 AIL |
78 | the "example" folder of this repo. |
79 | ||
80 | Note: The Cubism SDK is the property of Live2D and is not part of this | |
81 | project. You must agree to Live2D's license agreements to use it. | |
82 | ||
30b18873 | 83 | 6. Go into the |
261d3d0e | 84 | "example/CubismSdkForNative-4-r.2/Samples/OpenGL/thirdParty/scripts" |
830d0ba4 AIL |
85 | directory and run |
86 | ||
87 | ./setup_glew_glfw | |
88 | ||
30b18873 | 89 | 7. Go back to the "example" directory and run |
830d0ba4 AIL |
90 | |
91 | ./build.sh | |
92 | ||
30b18873 AIL |
93 | 8. Now try running the example program. |
94 | ||
95 | First, (in a separate terminal) go to where you have downloaded | |
96 | OpenSeeFace, and run | |
97 | ||
98 | ./python3 ./facetracker.py -v 4 --model 3 -M | |
99 | ||
100 | The `-M` option is currently available only on my OSF fork. | |
101 | ||
102 | I tried models 3 and 4, and I think both work reasonably well. | |
103 | Please feel free to explore other options provided by OSF. | |
104 | ||
105 | Back to the original terminal, from the "example" directory: | |
830d0ba4 AIL |
106 | |
107 | cd ./demo_build/build/make_gcc/bin/Demo/ | |
108 | ./Demo | |
109 | ||
110 | ||
111 | ## Command-line arguments for the example program | |
112 | ||
113 | Most command-line arguments are to control the Cubism side of the program. | |
114 | Only one argument (`--config`) is used to specify the configuration file | |
115 | for the Facial Landmarks for Cubism library. | |
116 | ||
117 | * `--window-width`, `-W`: Specify the window width | |
118 | * `--window-height`, `-H`: Specify the window height | |
119 | * `--window-title`, `-t`: Specify the window title | |
120 | * `--root-dir`, `-d`: The directory at which the "Resources" folder will | |
121 | be found. This is where the model data will be located. | |
122 | * `--scale-factor`, `-f`: How the model should be scaled | |
123 | * `--translate-x`, `-x`: Horizontal translation of the model within the | |
124 | window | |
125 | * `--translate-y`, `-y`: Vertical translation of the model within the window | |
126 | * `--model`, `-m`: Name of the model to be used. This must be located inside | |
127 | the "Resources" folder. | |
8ff44985 AIL |
128 | * `--old-param-id`, `-o`: If set to 1, translate new (Cubism 3+) parameter |
129 | IDs to old (Cubism 2.1) IDs. This is necessary, for example, for | |
130 | [the Chitose model available from Live2D](https://www.live2d.com/en/download/sample-data/). | |
830d0ba4 AIL |
131 | * `--config`, `-c`: Path to the configuration file for the Facial Landmarks |
132 | for Cubism library. See below for more details. | |
133 | ||
134 | ||
135 | ## Configuration file | |
136 | ||
137 | Due to the differences in hardware and differences in each person's face, | |
138 | I have decided to make pretty much every parameter tweakable. The file | |
139 | "config.txt" lists and documents all parameters and their default values. | |
140 | You can change the values there and pass it to the example program using | |
2b1f0c7c | 141 | the `-c` argument. If using the library directly, the path to this file |
830d0ba4 AIL |
142 | should be passed to the constructor (or pass an empty string to use |
143 | default values). | |
144 | ||
145 | ## License | |
146 | ||
147 | The library itself is provided under the MIT license. By "the library itself" | |
148 | I refer to the following files that I have provided under this repo: | |
149 | ||
150 | * src/facial_landmark_detector.cpp | |
151 | * src/math_utils.h | |
152 | * include/facial_landmark_detector.h | |
153 | * and if you decide to build the binary for the library, the resulting | |
154 | binary file (typically build/libFacialLandmarksForCubism.a) | |
155 | ||
156 | The license text can be found in LICENSE-MIT.txt, and also at the top of | |
157 | the .cpp and .h files. | |
158 | ||
830d0ba4 AIL |
159 | The example program is a patched version of the sample program provided |
160 | by Live2D (because there's really no point in reinventing the wheel), | |
161 | and as such, as per the licensing restrictions by Live2D, is still the | |
162 | property of Live2D. | |
163 | ||
164 | The patch file (example/demo.patch) contains lines showing additions by | |
165 | me, as well as deleted lines and unchanged lines for context. The deleted | |
166 | and unchanged lines are obviously still owned by Live2D. For my additions, | |
167 | where substantial enough for me to claim ownership, I release them under | |
168 | the Do What the Fuck You Want to Public License, version 2. The full license | |
169 | text can be found in LICENSE-WTFPL.txt. | |
170 | ||
171 | All other files not mentioned above that I have provided in this repo | |
172 | (i.e. not downloaded and placed here by you), *excluding* the two license | |
173 | documents and files generated by Git, are also released under the Do What | |
174 | the Fuck You Want to Public License, version 2, whose full license text | |
175 | can be found in LICENSE-WTFPL.txt. | |
176 | ||
177 | In order to use example program, or in any other way use this library | |
178 | with the Live2D® Cubism SDK, you must agree to the license by Live2D Inc. | |
179 | Their licenses can be found here: | |
180 | <https://www.live2d.com/en/download/cubism-sdk/download-native/>. | |
181 | ||
830d0ba4 AIL |
182 | This is not a license requirement, but if you find my library useful, |
183 | I'd love to hear from you! Send me an email at spam(at)adrianiainlam.tk -- | |
184 | replacing "spam" with the name of this repo :). | |
185 | ||
186 | ## Contributions | |
187 | ||
188 | Contributions welcome! This is only a hobby weekend project so I don't | |
189 | really have many environments / faces to test it on. Feel free to submit | |
190 | issues or pull requests on GitHub, or send questions or patches to me | |
191 | (see my email address above) if you prefer email. Thanks :) | |
192 |