msm: Manchester Shape Model libraries and tools

The msm library contains classes, functions and tools to build and use linear statistical shape models.
See also this tutorial which includes example data and pre-prepared parameter files.

Introduction

A shape is represented as a set of points, stored in a msm_points object, which encodes the co-ordinates in a vector (x0,y0,x1,y1,...).
Shape is defined as that variation in geometry which is invariant to some class of transformations. Usually one considers similarity transformations (scale, rotation and translation), though others are also used. To build a model from sets of shapes, one first aligns the shapes to remove the effects of the global transform.
The alignment and representation of transformations is encoded using the msm_aligner objects.
The shape model itself is a linear model of the form
  shape = mean + P*b
where mean is the mean shape, P is a matrix of modes of variation and b is a vector of shape parameters.

Data File Formats

Key data files encode sets of points (*.pts) and curves (*.crvs).

msm_points: Points file

All the tools assume that points are provided in text files in one of the two formats shown below.
The msm_points class encodes a set of points which can be read from/written to a text file in the format shown below.

This is the default format which is written by the tools and library functions:
version: 1
n_points: 3
{
316.069 201.255
410.593 206.549
333.881 320.539
}
Note that points in a simpler two column format can also be read in:
3
316.069 201.255
410.593 206.549
333.881 320.539
The first line indicates how many points follow.

msm_curves: Curves file

The msm_curves class encodes one or more "curves" as a set of point indices.
Each curve is assumed to join the given points in the order supplied.
If the curve is not open, then it is assumed to be closed and the last point is connected to the first point.
The crvs file is a text file defining the curves in a fairly easy to read/edit way.
For example, face_68pts.crvs defines the curves on a 68 point face markup:
curves: {
  curve:  { name: Chin open: true indices: { 0:14 } }
  curve:  { name: LEyebrow open: false indices: { 15:20 } }
  curve:  { name: REyebrow open: false indices: { 21:26 } }
  curve:  { name: LEye open: false indices: { 27:30 } }
  curve:  { name: REye open: false indices: { 32:35 } }
  curve:  { name: Nose open: true indices: { 37:45 } }
  curve:  { name: Nostrils open: true indices: { 46 47 } }
  curve:  { name: OuterLips open: false indices: { 48:59 } }
  curve:  { name: TopLip open: true indices: { 48 60 61 62 54 } }
  curve:  { name: BottomLip open: true indices: { 54 63 64 65 48 } }
}
The elements of each curve are:

Organising Files

Generally a shape is defined by a set of points annotated on an image.
For convenience we usually use similar names for points and image files, so that image7.pts will be the points annotated on image7.png. This is not a strict requirement though.
Usually we also arrange that all the images are in one directory (say "images") and all the points are in another (say "points"), leading to a file structure like this:
The models directory contains parameter files for the various tools.
The files to be used by any tool are stored in a standard format in an image_list text file.

Image Lists

The image list file (e.g. image_list.txt) contains the paths to points and associated images in the following format:
// Directories containing data:
points_dir: ../points/
image_dir: ../images/

images:
{
  image001.pts : image001.png
  image002.pts : image002.png
  image003.pts : image003.png
  // ...
}
Comments can be added using "//" - anything after this on the line will be ignored (C++ style).

Manipulating Points

Extracting a subset of points

The msm_subset_points tool creates a new points file containing a subset of points from another file.
Useage:
msm_subset_points pt_index.txt in.pts out.pts
This loads in a list of point indices from pt_index.txt and a set of points from in.pts. It then saves only the subset into out.pts.
The index file just lists integers, in range [0,n_pts-1], to be used.
Note, it can have comments using //
For instance:
 0 1 // Eyes
 5 6 7 8 9 // Mouth
It can be used to process all the files in a directory with a (bash) script such as:
mkdir new_points
for i in *.pts
do
  msm_subset_points pt_index.txt $i new_points/$i
done

Concatenating sets of points (and curves)

msm_concat_points points1.pts points2.pts ...  output.pts
Load in two or more of points, write out a file containing all points (in the order listed).
If shape1.crvs and shape2.crvs are the
curves files for points1 and points2, then a new curves file suitable for the concatenated points can be generated using:
msm_concat_curves shape1.crvs shape2.crvs all_shapes.crvs
This assumes that each curves file includes the point with the largest index (which it uses to work out how many points are in the set).

Reflecting sets of points

msm_reflect_points -i input.pts -x 511 -o output.pts
Load in a set of points, reflect about line x=value, save to output.pts

Equally spacing points along curves

msm_equally_space_points -p param_file [-c curves][-od output_dir]
For each curve, fit a cubic bezier, then move internal points so that they are equally spaced along the curve.
The parameter file has this form:
// Curves file (or curves themselves) defining which points to be moved.
// All internal points on a curve are slid along the cubic bezier through
// the points so that they are equally spaced.
curves: equal_spacing.crvs

//: Directory to new points
out_points_dir: /home/equal_spaced_points/

//: Smallest separation of points on bezier curve
min_bez_sep: 0.5

image_dir: /home/images/
points_dir: /home/points/
images: {
  image1.pts : image1.jpg
  image2.pts : image2.jpg
}

Warping points using a Thin Plate Spline

msm_apply_tps_warp -sc src_control.pts -dc dest_control.pts -sp src.pts -dp new_dest.pts
Uses a Thin Plate Spline, constructed from src_control points and dest_control points to warp src points.
Results saved to new_dest.pts
Useful to add new intermediate points to a set, controlled by existing points.

Building Shape Models

To build a shape model, from within the models directory run a command such as:
msm_build_shape_model -p build_shape_model.params
The parameter file should look something like this:
/: Aligner for shape model
aligner: msm_similarity_aligner

//: Define how to align mean shape in reference frame
// Options: { first_shape, mean_pose }
ref_pose_source: first_shape

//: Object to apply limits to parameters
param_limiter: msm_ellipsoid_limiter { accept_prop: 0.98 }

// Minimum number of shape modes
min_modes: 0

// Maximum number of shape modes
max_modes: 99

// Proportion of shape variation to explain
var_prop: 0.95

//: File to save model to
shape_model_path: shape_model.msm

image_dir: ../images/
points_dir: ../points/
images: {
  image1.pts : image1.jpg
  image2.pts : image2.jpg
  // ...
}
This will build a linear statistical shape model from the supplied examples (image1.pts etc) and save it to the named file (shape_model.msm) The main choices to be made are

msm_aligner : How to align shapes

Options include: The number of modes of the model is chosen to be in the range [min_modes,max_modes] such that the variance explained by the chosen modes is at least a proportion var_prop of the total variance of the training data.

Examining Shape Modes

Animations of the shape model modes can be viewed with the qmsm_shape_mode_viewer tool (in the uomqvxl package).
Alternatives EPS files can be generated using the msm_draw_shape_modes tool. This creates vector graphics which are more suitable for including in documents (e.g. LaTeX papers).

Getting shape parameters

To get the shape parameters associated with each set of points, use:
msm_draw_shape_modes -p draw_shape_modes.params
where draw_shape_modes.params looks something like this:
//: File to load model from
shape_model_path: face_68pts.msm

curves_path: face_68pts.crvs

// Maximum number of shape modes
max_modes: 5

//: Number of SDs to vary mode by
n_sds: 3.0

//: Current number of shapes per mode
n_per_mode: 3

//: When true, generate 2*n_per_mode frames of a movie for each mode
make_movie: false

//: When true, overlap all the shapes to highlight changes
//  Shapes on one side of mean are drawn with dashes, on the other with dots.
overlap_shapes: false

//: Radius of points to display (if <0, then don't draw points)
point_radius: 2

//: Line width
line-width: 1

line_colour: black
point_colour: red

//: Colour of background (or "none" for transparent)
background_colour: white

// Approximate width of region to display shape
width: 200

base_name: face
output_dir: ./
This will generate one file per mode, showing the mean shape and the effect of varying each parameter by +/-3SDs.
If you change the parameter make_movie to true, it will generate a sequence of movie frames, which can be collected and displayed with your favourite animation tool.
If you change the parameter overlap_shapes to true, the shapes for a single mode will be centered on the same point, making it easier to see small changes (though this can look cluttered).
msm_get_shape_params -p param_file.txt -o params.txt
param_file.txt should contain:
shape_model_path:  shape_model.msm

image_dir: ./images
points_dir: ./points
images:
{
  image1.pts : image1.png
// ...
}
Parameters are saved to params.txt, one line per example.
The first columns contain the pose parameters (as defined by the msm_aligner), the next the shape parameters. So with a similarity transform (msm_similarity_aligner) the first four numbers correspond to the four pose parameters.
If you add the -no_pose flag, then the pose parameters are not included.
If you add the -use_pts_name flag, then the points name string is put in the first column.
If you have built a shape model with parameter file build_shape_model.params, you can use the same file.
To get histograms for each parameter, use the -h flag to define the path:
msm_get_shape_params -p param_file.txt -o params.txt -h histo
This will then save histogram information to histo_b1.txt, bisto_b2.txt etc
You can display this with gnuplot:
  plot "histo_b1.txt" with boxes
By default the parameters are in units defined in the model frame.
You can change to use units of standard deviation (ie normalise them) by adding the -rel_p flag:
msm_get_shape_params -p param_file.txt -o params.txt -h histo -rel_p
In that case the histograms can easily be compared to that of a unit gaussian in gnuplot:
  plot "histo_b1.txt" with boxes, exp(-0.5*x*x)/sqrt(2*pi)
The range of the histo can be set with" -hr 4.0", and the number of bins with "-hn 101".

Other tools

Manipulating sets of images and points: Drawing tools: Using Shape Models:

Advanced Features

Exploiting symmetry

When you have annotated an approximately symmetric object, such as the face. You get two examples per image by reflecting both the image and the points. In this case the points must be renumbered (using the list supplied by the ref_point_index() function).
When points are reflected, their order must be changed. For instance, consider 10 points around the eyes of a face:
         3                7
    2    0    4      8    1    6
         5                9
when reflected, this becomes:
         7                3
    6    1    8      4    0    2
         9                5
To construct the permutation list, identify the new index for each point:
Original : 0 1 2 3 4 5 6 7 8 9
Reflected: 1 0 6 7 8 9 2 3 4 5
The lower line gives the list describing the permutation required.
When supplied in a parameter file, this can be summarised as { 1 0 6:9 2:5 }
The tool msm_guess_ref_sym can estimate this mapping given one or more shapes and a pair of points which map to each other:
msm_guess_ref_sym -i image_list.txt -r0 1 -r1 7 -o ref_sym.txt
This information can be used in the parameter files by adding the line:
reflection_symmetry: { 1 0 6:9 2:5 }
When this is supplied the tool will generate a reflected version of the shape and include it in the set, doubling the number of examples available.