Motion capture with Tenserflow.js/PoseNet + MaxMSP + Blender

3 min readApr 29, 2020

The key steps of the video above:

Use the PoseNet of TensorFlow based on web-based machine learning for motion capture;
Link the PoseNet page to MaxMSP with the Node for Max module provided by MaxMSP;
Human motion data captured by PoseNet is sent back to MaxMSP through SocketIO;
MaxMSP sends the received data to Blender via OSC;
Blender uses the received data to control the deformation animation in real time.

Most about three topic:

Demonstrate PoseNet motion capture.
Data transfer between MaxMSP’s Node for Max and web pages (in fact, several previous examples are about this with video tutorials).
Blender’s Eevee real-time rendering, and the reception of OSC data.

PoseNet

Google has a machine learning framework TensorFlow,

TensorFlow has a JavaScript version of TensorFlow.js,

There is a PoseNet model in TensorFlow.js,

PoseNet is a real-time motion capture model that can run on web pages.

I fine-tuned it and downloaded the model locally to load it, saving you some debugging troubles caused by unsuccessful loading.

It seems to link the web page part with MaxMSP.

In fact, it greatly expands the function of MaxMSP with the help of the powerful Web front end and NodeJS.

I have used it many times in the previous article.

This link below has the complete implementation process of MaxMSP + Node for Max + web page + Socket.IO + OSC:
Tutorial to voice control HTML GIF animation with mobile phone using MaxMSP and Nodejs

To implement the effect of this example, you may don’t need the Max part.

Just use NodeJS as a local server, host PoseNet pages, and use an OSC library like OSC.js to send data to Blender.

My purpose of using Max is because Max’s UDP Object is so easy to use.

The Blender part has been demonstrated before. The model, NodeOSC plug-in, and Eevee rendering are all old recipes, and there is no change.

My other related videos:

PS: I didn’t create this car model, I gave the reference link before.

PoseNet
https://github.com/tensorflow/tfjs-models/tree/master/posenet

Blender Car Model
Please support its author:
https://gumroad.com/bluefoxcreation#RclL

Source code and patch:
https://gum.co/yZnuM

(Source codes are js and MaxMSP patch. NO Blender file there).

Donate:
https://www.patreon.com/avantcontra
There are many articles, patches, source code and some advanced Patron-only content there.

Or you can get a source code/patch directly in gumroad.
https://gumroad.com/avantcontra
If you like something, could buy me a coffee :D

More articles in my site:
https://floatbug.com