Research:Bloom, a garden fed by motion
Contents
Bloom - video
Introduction
I was hired for 10 days by numediart to develop an artistic visualisation of a new framework/application called Motion Machine during the eNTERFACE15 workshop.
Motion Machine's purpose is to provide a support for motion capture features extraction, both realtime via a kinect sensor for instance, and offline, when using motion capture recordings. A team of 15 people worked during one month to propose a set of features. Some of them are low level such as velocity or energy of skeleton's nodes, some are much more advanced such as Müller algorithm or HMM-based gesture récognitionHMM-based gesture récognition.
My work as a creative programmer was to build a graphical system that uses these features in a aesthetic manner.
Seeds
Motion machine already had a UI that displays the skeleton and the features' curve. A new paradigm was to be found to propose something interesting and aside of the scientific approach.
Based on former studies (harmonic plants and followers), the idea of seeds floating in a volume around the skeleton quickly arouse.
Bascis
Each seed is placed randomly in the space and moves slowly, generally towards the center of the space, where the skeleton tends to be. When a seed enters the attraction radius of a skeleton's node, it starts to accelerate towards it (pull phase). When it is close enough, the node pushes the seed away (push phase). By a careful tuning of the pull & push radius, the seed is never stuck around a node, and keeps moving around the whole skeleton.
The model has many advantages:
- autonomous: seeds are not depending on a skeleton, they can move in the void
- rich paradigm: seeds can grow into many forms, perfect for object-oriented programming
- fast to setup: points moving in a 3d space is easy to code and monitor
The first working version was ready after 2 days of work (see visuals here above).
Headaches
Until now, the seeds were completly independent of the skeleton. The main problems appeared quite fast once I wanted to make them grow:
- the seeds feed on feature, how to make them accessible through the nodes?
- features are heterogeneous, how to normalise them?
- to start growing, seeds have to stop near a compliant node, thus how to decide when and where to stop?
- their evolution has to happen gradually, how to manage grow and decay?
- and, to make the structures move nicely, a simple physical engine was required.
Each of these issues have been solved individually, as explained below.
BNode structure
The approach I took from the start was to work outside of the main MotionMachine project, that already contains a set of methods to draw. A ofxMoMa addon has been built by alexis moinet to make all the important classes accessible in a standard OF project.
To make nodes information accessible to the seeds, I've implemented a BNode structure containing:
- position (ofVec3f)
- orientation (ofQuaternion)
- relation to parent Bnode (if some)
- relations to child BNodes
- features - for each of them:
- a boolean if the feature is enabled
- 1 or several float(s) for the feature value(s)
BNodes are updated at each update by a specialised class called BNodeManager. The manager extracts the relevant info from the track and other objects (TimedMat mainly) and updates the BNode.
All the BNodes are stored in a pivot structure shared upon all the objects, called SeedExchange.
Therefore, when a Seed is created, it receives a pointer to the SeedExchange structure and is able to read the BNodes parameters. The first issue is solved.
Normalisation
Normalisation is a key point when you want to use data to control other data. Typically, the possible range is rescaled between 0 & 1 - see Data mapping for more details.
The feature Solicitation was one of the first to be ready when I started the project. It was computed on the node's level, making it a perfect candidate to feed the seeds.
The main issue was that it was not normalised. A generic FeatureNormaliser, separated from the other features availble, has been developped to do the job.
The results of the normaliser are available in the BNodes.
Attaching the seed
The first problem: Where to stop the seed?
A bit of 3d geometry has been required to solve this issue. At each frame, the seed knows which nodes influence it. Using the same information, an attach point can be processed. The idea was to attach it as close as possible from its actual position, meaning that if it is above a bone (a segment between 2 BNode), the attachment must be located between them. The procedure deals with 2 different cases:
- seed is outside the bone - solution is straightforward: attach point is equal to the closest BNode
- seed is above the bone - the attach point is between the start and the end of the bone
The method to get the position on the bone.
static ofVec3f projectOnBloomNode( ofVec3f * pt, BNode * origin, BNode * target ) {
ofVec3f bv = target->position - origin->position;
bv.normalize();
ofVec3f opt = (*pt) - origin->position;
float proj = opt.dot( bv );
if ( proj >= 0 ) return origin->position + bv * proj;
else return ofVec3f( origin->position );
};
The second problem: How and when to stop the seed?
Finding where to attach the seed was easy. Deciding when to attach it is (much) more tricky. A parameter satisfaction in each seed is a container that rises or shrinks (can be positive or negative). This parameter is bounded by an arbitrary minimum and maximum. As it rises or shrinks continuously and gradually, it can be used as a dampener and delay the actions by waiting for it to reach one of the bounds.
Because I wanted the satisfaction to behave differently depending on the kind of seed, I needed a way to describe how the seed reacts to the feature and how much of satisfaction is added or removed at each update.
Affinity
The way satisfaction evolves is managed by this class. An affinity instance represents the reaction to a value. It can be attached to any kind of normalised value. It has few parameters and a main method called process.
Parameters:
- minimum and maximum input value, each of them between 0 & 1, allowing to narrow the range of response
- low and high response, representing the normalised output when value is high (at maximum or above) and when it is low (minimum or below), between -1 & 1
- increase and decrease, two multipliers applied on low and high responses, that represent the amount of satisfaction to add, both being above 0 but not bounded to 1
- power allowing to curve the response range and increase the response at the bounds
- and other parameters, of minor importance
This concept is crucial to the approach. It allows to define at a high level and in a centralised way the reactions of the seeds in constant interaction with nodes.
Life cycle
Rooting
When satisfaction reaches the maximum for the first time, the attachment process happens. At that moment, its behaviour changes radically. It stops moving freely and stays close to the skeleton and follows the evolution feature of closest BNode.
Grow & decay
The satisfaction is used to decide when to add a new part to the plant (growing process) or when to remove a part (decay process). This part of the life cycle does not happen in the Seed class and is managed in the sub-classes.
Death
If the feature is perceived as too negative by the seed, it will eventually die and detach from the skeleton. The death process has not been finished in the current version of the code, due to time constraints.
Weed & salpetre
Two sub-classes of Seed have been implemented:
- Weed - green leaves
- Salpetre - a mycelium and white corollas
Each of these sub-classes have a different Affinity configuration related to Solicitation feature.
Weed likes the solicitation to be high. Meaning that, when the solicitation of the closest BNode is high, the satisfaction of the weed increases.
Salpetre has the opposite reaction. When the solicitation is low, its satisfaction increases.
As they both feed on the same feature, the importance of the affinity configuration is clearly enlighted. Due to the feature process, salpetre tends to appear at the extremities of the skeleton, where it is generally low, and weed concentrates around the hips where it is generally high.
Graphics
Last but not least, the graphical elements. These objects also include a damping effect, motion being as important as pure shape part in the aesthetics of an animated visualisation.
To make it easy to build, I've implemented a base class that regroups all required parameters such as positions, color and a simple physical model. This base (nearly abstract) object is called Position.
The salpetre's mycelium is a growing tree of positions, rendered as transparent lines.
The weed's leaves require a more complex object. They both uses Branch objects, that inherits from Position. Branches render 4 points that describe the corners of a quad. UV's coordinates can be set on each of these points, enabling texturing.
Rendering optimisation
Openframeworks provides easy to use methods to draw shapes on screen. Because each shape makes a serie of calls to the standard opengl methods to draw, they are not efficient when you need to draw a huge amount of graphical elements.
To avoid calling thousands of times the standard OF methods, I've created 2 vectors in the Seed class, one for lines, and one for quads. During draw(), only 4 calls are required to start and stop the shape rendering. A snippet of code will make this clearer.
// drawing all lines at once
glBegin( GL_LINES );
for_each( seeds->begin(), seeds->end(), [](Seed * s){ s->linevertices(); });
glEnd();
// drawing all leaves at once
garden.bindLeafTexture();
glBegin( GL_QUADS );
for_each( seeds->begin(), seeds->end(), [](Seed * s){ if ( s->getType() == SSTYPE_WEED ) { s->quadvertices(); } });
glEnd();
garden.unbindLeafTexture();
The color of the leaves and the lines are managed in the Position object, or the Branch object.
virtual void linevertices() {
if ( color != NULL ) glColor4f( color[ 0 ], color[ 1 ], color[ 2 ], color[ 3 ] );
glVertex3f( origin->x, origin->y, origin->z );
glVertex3f( x, y, z );
}
This makes the display very efficient without using VBO's or shaders, although they would be required to draw a bigger number of graphical elements.
Results
by FrankieZafe (talk) 18:17, 8 September 2015 (CEST)