Pharm Drone

My dad, Dr. Roy Golsteyn, is a scientist at the University of Lethbridge in the department of Biology. His latest research project focuses on finding cancer drugs from the flora of southern Alberta. Important to his research is a significant amount of fieldwork to locate plants and bring them to the lab for further analysis.

The best time to collect plants is while they are blooming. However, it is difficult to locate flowers in the semi-desertic landscape of southern Alberta.

Over the 2019/2020 winter break, my dad presented me with an interesting challenge: is it possible to locate flowers given a top-down image of a prairie field taken from a DJI Phantom Pro drone?

This project can be divided into three tasks:

1. Using a consumer-grade drone to survey a site with the potential for prairie flowers.
2. Analyzing drone imagery to detect the presence of flowers.
3. Using the result of that analysis to aid the harvesting process.

In this article, I will focus on the image analysis portion of this project. I will outline our initial approach to solve this computer vision task and outline areas for improvements.

Implementation

The implementation was split into two parts: first, we needed to train a model that could differentiate flower pixels from non-flower pixels (grass, earth, rocks, etc.). Then, we could use this model and apply it onto new images captured by a drone.

Training the model

We had access to a large database of southern Alberta flowers of 110 species, 4000 images taken over 7 years. This database contained multiple pictures of different plant species in different lighting conditions.

The approach we took was to identify several colour clusters from the dataset. We manually cropped small image patches from the flower portion of the images. We then reshaped all the patches as a list of pixel values and filtered the value component.

Left with only the hue and saturation component of each pixel, we applied k-means, with $k=6$, to retrieve a list of 6 colour values and the standard deviation of each cluster associated with flower pixels. These colours and deviations were used when conducting color thresholding on new images.

Processing new images

To determine if flowers were present in a new image, we made the following assumptions:

1. Flowers stand in contrast with their surrounding environment: Keeping only edges would remove possible false-positives. We applied a Canny edge detector on the image.
2. Flowers can be segregated from their surrounding environment based on their colour (see above): Applying colour thresholding using the colours from the colour model we generated would leave us with a mask of where flowers could be present.
3. If a flower is present in an image, then more than one pixel will have a value that makes it a flower candidate: A single-pixel could indicate noise. We only keep clusters of pixels in the mask. This is done with an erosion morphological operation.

The result was a small algorithm in Python, using OpenCV, which contours potential flowers. These contours, alongside the picture's geolocation information, could be used to determine if a particular site is ideal for flower harvesting.

Future work

This algorithm was tested on a set of 7 images. It confirmed our assumptions that colour could be a good attribute for detecting the presence of flowers. The next step is to verify if this approach generalizes to a large image dataset of drone prairie images.

We plan to move away using k-means for the colour model in favour of a supervised classification algorithm.

We are also planning to develop a mobile application to allow researchers access to the processed images while on site. This application should help researchers select the best site to harvest.

In the meantime, feel free to check the code on GitHub!