Segmentation & Registration for Microscopy Imaging

Introduction

In the past year, in addition to the algorithms, we built a software platform that address each level of image processing problems, from low-level vision tasks to fully-automatic or interactive pattern recognition.

This software has been made accessible to our collaborators through a CVS repository, enabling them to use any version of our algorithms, including the latest ones.

For the low-level vision tasks, it includes a set of filters ranging from basic deblurring to high-end anisotropic filtering. Therefore the user has the capability to first bring the maximum detail into relief.

Enhanced features in those images can be extracted with the extensive set of 2D/3D segmentation tools based on the Level-Sets methodology with fast implementations.

The proper layer of interactivity is given by the VTK based user interface, and includes semi-automatic contour extraction solutions like the Live-Wire algorithm.

The resulting package is robust, fast and widely adaptable to any kind of specific imaging problems.

Using one method and/or the other, the user is able to build its own successive operations which will lead to the best solution.

The filtering methods

Chart of the Filtering methods implemented

• calImageFilter: the highest class, of general image filtering
  • calScalarFilter: category of filters which apply to single component images
    • calEnhanceFilter: category of filters which enhance the grey-level images
      • calAOSFilter: class that implements the Reaction-Diffusion and the Beltrami Flow;
      • calHomotopicBackgroundCorrection: class that removes the background framed ghost in the tilled images;
      • calImageStrech, calHistogramEqualization, calNormalizeData are basic enhancement filters
    • calFeatureFilter: filters that are used as cost for the Level-Sets methods
      • calBoundaryBasedForce (force for the Level-Set method)
      • calFourierFilter: Multiscale filtering (Hessian, Laplacian)
      • calGreyLevelFilter: as simple as it seems;
      • calLaplacianZeroCrossing: feature useful for the live-wire
  • calVectorFilter: category of vector transforms
    • calGradientVectorFlow: implementation of the algorithm of Xu & Prince;
    • calWarpImageFilter: implements the Level-Sets Registration method.

Example 1: Background Removal

Original Image
Background removed

Example2: 2: Filtering with the Beltrami Flow

Original Image
Beltrami Flow

Level-Sets Methods

Chart of the Level-Sets Methods implemented

• calInterface: general class of moving interfaces
  • calFastMarching: algorithm for the Eikonal equation
  • calHamiltonJacobi: general Hamilton-Jacobi formulation
    • calAOSPde: Additive-Operator-Splitting techniques
      • calAosSubjectiveSurface: AOS version of the Subjective Surfaces
      • calAosActiveContour:
        • calAOSGeodesicActiveContour: Weickert version of the geodesic active contour
        • calAOSLaplacian: AOS version of the Laplacian equation
    • calExplicitPde: explicit scheme methods
      • calLevelSet: classic level-set implementation and calLevelSetNB: narrow-band implementations

The Live-Wire method

The problem of diversity in the microscopy images

The structure of the duct is very complex. The following images display the structure of a mouse duct at several different time in its life.

DIfferent states of the mouse mammary gland

Virgin	Pregnancy	Lactation	Involution

The complexity of the object is changing at each step of its life.

Extracting the complete structure of the mammary gland is very challenging. The mammary gland wether it is cut at a certain depth of its structure will have a complete different connectivity, and can appear

1. as a circle/ellipse, where you can see the duct;
2. as a branching structure, when you intersect at the level of bifurcations;
3. as group of unconnected objects, when you are observing a terminal duct lobular unit (TDLU);
4. or as all the former categories in the same image, due to the tree strucure of the gland

The following image shows each one of those categories. The right image is the most common case where all different connected structures are in the image at the same time.

Different problems of segmentation to address

A simple duct	A branching duct	A Terminal Duct Lobular Unit (TDLU)	Everything at the same time

It is important to notice that the Terminal Duct Lobular Units (TDLU) are groups of unconnected components. Therefore there is no evidence in the image of a closed limit of this structure in the slices. What the user wants in this case is to draw a contour that contains all of the units. There are very few methods based on the Active Contour Models that are able to achieve this task. The only method that can accurately extract patterns that are not in images, but are rather apparent - like the triangle of Kanisza - is the Subjective Contours method [6].

However, the diversity encountered in the images makes it impossible to devise a method that will extract all the structures of interest. Each feature has a particular signature that might be recognized by a specific technique, thus that can be enhanced by a specific filtering technique.

A specific technique can be devised for each different task. But each algorithm will need its set of parameters, and multiplying this number of parameters by the number of different problems will make the task of a non-expert user very tough.

This is the reason why we need the help of a semi-interactive technique.

A semi-interactive method

Approaches in image segmentation are ranging from fully automatic methods to fully manual methods. The first ones totally avoid user's interaction but even if they are well adapted to specific cases their success can not be guaranteed in more general cases. The second ones are time-consuming, unrepeatable and inaccurate. To overcome these problems, interactive (or semi-automatic) methods combine knowledge of the user and computer capabilities to provide supervised segmentation.

The ideal semi-automatic tool should be able to offer the following to a non-expert user:

1. the possibility to draw quickly the boundary of an anatomical object, restricting his intervention to the initialization of a start point in the image;
2. drawing in real-time a polygonal line between this start point and the current cursor position, capturing the contour to extract;
3. this line should correspond to a certain amount of constraints, such as internal forces (regularity) and external forces (the image itself) and action of the user, making it reproducible;
4. the method should be able to generate a kind of learning to better estimate the different parameters of the model.

The line extraction methods

The general approach is to define a boundary as the minimum of an energy function that comprises many components such as internal and external forces.

In the literature, there exist many techniques to perform this minimization

Classical active contours

The classical active contours (Snakes), introduced by Kass et al are widely recognized for this task, but they present four main problems:

1. the user must initialize a polygonal line which is close to the final contour;
2. the method is very sensitive to the initialization step and often gets trapped in a local minimum of the energy;
3. user's control cannot be applied during the extraction
4. the parametrization of the model is complex for a non-expert user

The minimal path theory

In this approach the image is defined as an oriented graph characterized by its cost function and the boundary segmentation problem becomes an optimal path search problem between two nodes in the graph.

This approach overcomes the problem of local minima by using either dynamic programming (Dijkstra), or a front propagation equation (Cohen and Kimmel), mapping the non-convex cost function into a convex function.

Falcao and Udupa with their Live-Wire and Mortensen and Barrett with their Intelligent-Scissors were the first to introduce interactivity into this optimal path approach.

Their method is based on Dijkstra's graph search algorithm and gives to the user a large control over the segmentation process.

The idea is the following:

• a start point is selected by the user on the boundary to be extracted, and an optimal path is computed and drawn in real time between this start point and the current cursor position;
• parts of the boundary are frozen when considered "stable", enabling the user to extract closed contours. This is called Bordery Cooling
• the algorithm analyzes the information given by the path extracted in order to adapt the parameters of the model to extract similar paths. This is called Training.

Example: Segmentation with the Live-Wire

What you see on the following images needs some explanation:

• The portion of the path in blue is frozen
• The portion of the path in green is the free path

Tests on a cancerous cell

This test is performed on the image filtered with the Beltrami Flow.

Segmenting the outer boundary of a cancerous cell with the Live-Wire.

Animation

This test is performed on the image filtered with the Beltrami Flow.

Segmenting the inner boundary of a cancerous cell with the Live-Wire.

Animation

Tests on a mammary duct image

For the inner boundary, a potential based only on the gradient of the image is sufficient to extract the closed contour.

Segmenting the inner boundary of a duct with the Live-Wire.

Animation

For the outter boundary, a potential based only on the gradient of the image is not sufficient to extract the closed contour. Since there is no evidence of such a contour, a potential which includes information about the contrast and the grey-level value at the pixel on the edge of the contour extracted gives better results.

Segmenting the outter boundary of a duct with the Live-Wire.

Animation

The Registration problem

Example of images

The following images represent a couple of successive slices to register altogether. It is important to notice that the image dimensions is superior to 5000x5000 pixels.

Couple of images to register

More than just a problem of dimensions, registration of the images is a very difficult task for the images of the mammary gland. Several reasons are

1. the ratio of the in-plane resolution, and the distance between two slices is high: the in-plane resolution is 2.72 microns per pixel, and the distance between section about 40 microns.
2. the slices are done by cutting the material; therefore the tissues can be damaged during the cutting process and we can loose an important correlation between slices;
3. the structure is highly branching, and the connectivity between branches might be lost between two distant slices.

Registration methods

We could classify the image registration methods into 2 classes: feature-based & direct methods. The first one first detects features in the image, thus rely on the accuracy of this detection. The second ones estimate the transformation between the two given images from the raw-data.

We implemented the method of Vemuri et al which falls in this second category. The principle of the method is to reformulate the registration problem into a curve evolution approach which can be implemented into a Level-Set framework.

The method tend to exploit the topological properties of the method developed by J.A. Sethian. They achieve image intensity morphing, and derive a simple non-linear PDE for the corresponding coordinate registration. Advantages of this new formulation is that it is fast and that it has existence and uniqueness of the solution to the evolution model. The basic principle is to evolve an image into the other one by using a speed given by the difference between our image and the target.

However, the method in practice needs the algorithm to be modified. An artificial speed must be added in order to stop iterating only when the target is reached, and not when the speed of the evolution is zero which can occur in some cases. The model used in practice is biased, and while it really morphs an image into another one, it does compute the coordinate registration we are looking for. The model just adds a percentage of the differnce of the images until the result is our final image. Moreover, there is no proof of existence & uniqueness of a solution to the coordinate registration.

In conclusion, this is not applicable in our case.

Desired method

Regarding the different problems that we mentionned earlier, the desired method should have the desirable features:

1. Of course this method should be fast. The knowledge we have an the Additive-Operator-Splitting (AOS) techniques is something that could be capitalized by trying to implement the method in the curve evolution framework.
2. The dimensions of the image emphasizes the fact that only a multi-scale method is acceptable in our case.
3. Since the framework of segmentation is quite well developed, this method should be able to integrate the features in order to accelerate its convergence. The segmentation provided by the subjective surfaces and the geodesic flow implementations should be an input of this method.
4. Another way of doing it is to simultaneously segment and register the slices, integrating both processes in the same framework.

Plan for next year

1. The diversity of the problems encountered in the images leaded to the construction of an interactive method:
   The Live-Wire method is a semi-automatic method which uses the mouse interaction to achieve difficult segmentation tasks. We re-implemented it with the Eikonal equation which is more accurate than the Dijkstra minimal path algorithm. We need now to find the features in the images that can be extracted to ease the segmentation of each different problem. We need also to devise a method to automatically balance the weights of each different feature, to reduce the user interaction to the minimum required.
2. The registration method is currently based in an implementation which gives results that can be greatly improved. We are thinking about building a better implementation using the last developments of the Level-Sets methods, in particular the AOS implementation adapted from our work on the Subjective Surfaces.
   We would like to implement a coupled method where registration & segmentation are achieved at the same time by a new formulation which integrates both.
   This method would be used in a multi-scale framework, and would at the same time register two images and computing the transformation from one image to the other in order to warp the boundaries of the object extracted in one image to the other giving automatically the segmentation result.
   This registration could be also used to create intermediary images between two distant slices.
3. The last step is to reconstruct the surface of the ducts in 3D.
   Once the 2D structures are extracted, we will adjoin them in a 3D surface. It could be done roughly, but we are confident that the topology adaptability of our methods should give rise to an interesting new method.
   Once this surface is extracted, we would be able to extract its underlying tree-structure with the same minimal path techniques developed for the Live-Wire. We could also implement a new skeletonization algorithm to improve the accuracy of such a result. We are currently invastigating some PDE schemes that could do the job.
   This tree-structure will give a description of the ducts shape, and will enable to guide a virtual camera inside it. This navigation will give rise to observation and measurements on the extent of the pathology observed.

References

math.lbl.gov/~deschamp