Motion capture with Tenserflow.js/PoseNet + MaxMSP + Blender

avantcontra
3 min readApr 29, 2020

--

The key steps of the video above:

  • Use the PoseNet of TensorFlow based on web-based machine learning for motion capture;
  • Link the PoseNet page to MaxMSP with the Node for Max module provided by MaxMSP;
  • Human motion data captured by PoseNet is sent back to MaxMSP through SocketIO;
  • MaxMSP sends the received data to Blender via OSC;
  • Blender uses the received data to control the deformation animation in real time.

Most about three topic:

  • Demonstrate PoseNet motion capture.
  • Data transfer between MaxMSP’s Node for Max and web pages (in fact, several previous examples are about this with video tutorials).
  • Blender’s Eevee real-time rendering, and the reception of OSC data.

PoseNet

Google has a machine learning framework TensorFlow,

TensorFlow has a JavaScript version of TensorFlow.js,

There is a PoseNet model in TensorFlow.js,

PoseNet is a real-time motion capture model that can run on web pages.

I fine-tuned it and downloaded the model locally to load it, saving you some debugging troubles caused by unsuccessful loading.

MaxMSP And Node For Max

It seems to link the web page part with MaxMSP.

In fact, it greatly expands the function of MaxMSP with the help of the powerful Web front end and NodeJS.

Note: Max8 needed

I have used it many times in the previous article.

This link below has the complete implementation process of MaxMSP + Node for Max + web page + Socket.IO + OSC:
Tutorial to voice control HTML GIF animation with mobile phone using MaxMSP and Nodejs

To implement the effect of this example, you may don’t need the Max part.

Just use NodeJS as a local server, host PoseNet pages, and use an OSC library like OSC.js to send data to Blender.

My purpose of using Max is because Max’s UDP Object is so easy to use.

Blender

The Blender part has been demonstrated before. The model, NodeOSC plug-in, and Eevee rendering are all old recipes, and there is no change.

My other related videos:

Interactive between mobile phone and Blender animation through OSC

How to make interactive audiovisual effect in 5 minutes using Blender and MaxMSP

PS: I didn’t create this car model, I gave the reference link before.

Talk is cheep, show me the code!

Source code and patch:
https://gum.co/yZnuM

https://www.patreon.com/posts/36499862/

(Source codes are js and MaxMSP patch. NO Blender file there).

Donate:
https://www.patreon.com/avantcontra
There are many articles, patches, source code and some advanced Patron-only content there.

Or you can get a source code/patch directly in gumroad.
https://gumroad.com/avantcontra
If you like something, could buy me a coffee :D

More articles in my site:
https://floatbug.com

--

--

avantcontra
avantcontra

Written by avantcontra

Digital artist, creative technologist. Interactive media/AI/generative art. avantcontra.com