Ultrasound Video Analysis

The main focus of my doctoral research was the automated analysis of ultrasound videos, with a particular focus on fetal heart scanning. Previous work has been successful in automatically detecting objects of interest in still ultrasound images, which can be useful for post-acquisition measurement on typically carefully-selected images. My aim was to extend this to full video analysis, working towards creating software that can, to some degree, understand what is happening in the scan’s video stream as the scan is under way. This has the potential to provide software tools to assist sonographers acquiring scans, and thereby improve the effectiveness of ultrasound as a screening process.

I advocate the use of Bayesian filtering techniques as a natural framework to use in this context. The key advantages of this group of probabilistic techniques are a) they naturally deal with the uncertainty that is inherent in the interpretation of the video, and b) allow an engineer to build in prior knowledge of how ultrasound videos typically evolve in order to combine information from multiple frames into a consistent hypothesis about the entire video. This is crucial in the setting of ultrasound video analysis, where the single video frames can be quite uninformative due to the indistinct nature of the imagery and the presence of imaging artefacts, but the spatial and temporal relationships between the variables of interest are very strong. Similar techniques have proved popular in situations such as 3D human pose tracking from video footage, and localisation of mobile robots in uncertain environments (the so-called ‘SLAM’ problem).

In my journal paper “Automated Annotation and Quantitative Description of Ultrasound Videos of the Fetal Heart”, I present and evaluate a Bayesian filtering framework for analysing fetal heart videos. The algorithm uses stochastic methods (particle filters) to represent and solve the filtering problem. The video above shows some examples of the output of this algorithm (top) compared to the manual annotations (bottom). Comparing the “no filtering” and “with filtering” examples shows how important the filtering framework is for achieving good results.

In more recent work (“Localizing Cardiac Structures in Fetal Heart Ultrasound Video”), I extended this framework to additionally track diagnostically important anatomical structures such as valves and vessels, as shown in the video below.

Along with colleagues, I have begun to look at deep learning as an alternative method for ultrasound video analysis.

This work was supervised by Alison Noble at the Institute of Biomedical Engineering, University of Oxford, and was performed in collaboration with Christos Ioannou (John Radcliffe Hospital, Oxford).

Analysis of Fetal Heart Ultrasound Videos

For my work on ultrasound video analysis, I have focused on videos of the fetal heart. Ultrasound screening is the standard clinical method for antenatal detection of congenital heart defects, such as septal defects, vessel coarctation, tetralogy of Fallot, arrhythmias, and others. Unfortunately, screening for these defects is a highly skilled task as it involves checking a number of different anatomical structures in multiple viewing planes, and, as such, detection rates vary.

Three Fetal Heart Ultrasound Viewing Planes

Three fetal heart viewing planes: the four chamber view (4C), left ventricular outflow tract view (LVOT), and the three vessels view (3V). Key: LV/RV left/right ventricle, LA/RA left/right atrium, (d)Ao (descending) aorta, PA pulmonary artery, SVC superior vena cava, Tr trachea, Rb rib, Vb vertebra.

Using Bayesian filtering, I have developed software that tracks the fetal heart, including its position, orientation, viewing plane, and the position in the cardiac cycle. I hope that this could aid sonographers during the acquisition process and provide a basis for more specific automated diagnostic screening processes.

Rotation Invariant Features in Ultrasound Image Analysis

Many traditional computer vision algorithms are designed to detect objects that occur in images in a small range of orientation. However, in a number of medical imaging applications, this approach is not appropriate. This is particularly true in fetal ultrasound imaging, where the orientation of the fetus relative to the probe varies within and between scanning sessions.

Instead of simply applying traditional techniques to multiple rotated versions of the image, I have investigated more principled approaches to rotation invariant detection that build rotation invariance into the features used by machine learning algorithms.

A flowchart for calculating rotation invariant features

A framework for rotation-invariant detection using image gradient. First a continuous gradient orientation histogram is formed and represented by a small number of Fourier series coefficients. These coefficient images are then convolved with a set of complex-valued basis functions. The phase of the resulting complex number is invariant to the rotation of the underlying patch.

I use complex-valued filters to create descriptive feature sets for image patches that are analytically invariant to the rotation of the patch (this follows the work of Liu et al.). This forms a description of the patch that may be used for classification/detection and other machine learning tasks. This is described in detail in “Object Localisation In Ultrasound Images Using Invariant Features”. There is also a diagrammatic explanation in these [slides].

The Monogenic Signal for Medical Image Analysis

The monogenic signal is an image representation with a number of attractive properties for tasks such as edge and feature detection. It has been found to be particularly useful for analysing ultrasound images, because it copes well with the variable contrast, speckle artefacts and indistinct features in the images. It also forms the basis for calculating a number of other quantities such as local phase, local amplitude/energy, local orientation, feature symmetry, feature asymmetry and phase congruency.

Example of feature symmetry and asymmetry calculated from an ultrasound image

Feature symmetry (top) and asymmetry (bottom) may be calculated from the monogenic signal and are useful for detecting ‘blob’ structures and boundaries respectively in images. © IEEE 2015

In trying to understand many of the concepts surrounding the monogenic signal I found that the current literature does not offer any introductory text, and that this makes the area somewhat impenetrable. I therefore wrote a tutorial-style “Introduction to the Monogenic Signal” document to help explain the concepts to others in my lab.

In my paper “Object Localisation In Ultrasound Images Using Invariant Features”, I demonstrated an effective detection framework for object in ultrasound images that makes use of the monogenic signal to increase invariance to speckle artefacts and contrast changes.

Registration of the Human Femur

The thickness and distribution of femoral cortical bone is thought to be an important factor determining a person’s susceptibility to hip fracture. Cortical thickness can be estimated from a CT scan, giving a surface representation of a subject’s femur with associated cortical thickness values.

Example of a registration failure

Example of a registration failure, where the lesser trochanters on the two surfaces are not aligned.

However, in order to conduct large-scale quantitative medical studies, it is necessary to first find the correspondence between points lying on the different surfaces. This requires a reliable, automated registration procedure. In my fourth-year undergraduate research project, I investigated and implemented a number of improvements to an existing registration procedure.

One contribution was testing a ‘locally affine’ transformation, which I found to reduce unnecessary warping during the registration procedure due to its local nature.

Gaussian and mean curvatures calculated from a femur surface

Gaussian (left) and mean (right) curvature quantify local surface shape.

A second important contribution was incorporating surface differential geometry into the cost function of the point matching. This ensures that the aligned points on the two surfaces have similar local geometry.

The outcome of this work was improvements to the ‘wxRegSurf’ software, freely available at the project website along with more information, pretty pictures and lists of related publications. You can also read my [Project Report] or [Presentation].

This work was conducted under the supervision of Andrew Gee at Cambridge University Engineering Department.