Motion capture with Tenserflow.js/PoseNet + MaxMSP + Blender
The key steps of the video above:
- Use the PoseNet of TensorFlow based on web-based machine learning for motion capture;
- Link the PoseNet page to MaxMSP with the Node for Max module provided by MaxMSP;
- Human motion data captured by PoseNet is sent back to MaxMSP through SocketIO;
- MaxMSP sends the received data to Blender via OSC;
- Blender uses the received data to control the deformation animation in real time.
Most about three topic:
- Demonstrate PoseNet motion capture.
- Data transfer between MaxMSP’s Node for Max and web pages (in fact, several previous examples are about this with video tutorials).
- Blender’s Eevee real-time rendering, and the reception of OSC data.
Google has a machine learning framework TensorFlow,
There is a PoseNet model in TensorFlow.js,
PoseNet is a real-time motion capture model that can run on web pages.
I fine-tuned it and downloaded the model locally to load it, saving you some debugging troubles caused by unsuccessful loading.
MaxMSP And Node For Max
It seems to link the web page part with MaxMSP.
In fact, it greatly expands the function of MaxMSP with the help of the powerful Web front end and NodeJS.
I have used it many times in the previous article.
This link below has the complete implementation process of MaxMSP + Node for Max + web page + Socket.IO + OSC:
Tutorial to voice control HTML GIF animation with mobile phone using MaxMSP and Nodejs
To implement the effect of this example, you may don’t need the Max part.
Just use NodeJS as a local server, host PoseNet pages, and use an OSC library like OSC.js to send data to Blender.
My purpose of using Max is because Max’s UDP Object is so easy to use.
The Blender part has been demonstrated before. The model, NodeOSC plug-in, and Eevee rendering are all old recipes, and there is no change.
My other related videos:
PS: I didn’t create this car model, I gave the reference link before.
Blender Car Model
Please support its author:
There are many articles, patches, source code and some advanced Patron-only content there.
Or you can get a source code/patch directly in gumroad.
If you like something, could buy me a coffee :D
More articles in my site: