AR glasses tell if gerbera can be harvested

The Augmented Horticulture project started last year. For four years, WUR has been developing four use cases with a consortium of several horticultural suppliers. Each use case consists of a series of underlying software modules. For example, at this time researchers work on recognizing QR codes with AR glasses, so that the glasses can retrieve relevant information about a specific plant from a database. This module will be demonstrated in the summer of 2020.

AR glasses train harvesting personnel, recognize stress...

This module is part of the first use case: having AR glasses count the number of stamens in the flower of a gerbera. The purpose of this application is to train new harvest personnel in recognizing harvest-ready flowers. In the second use case, AR glasses recognize the degree of stress of plants in a greenhouse. This is measured by infrared sensors, and converted by the glasses into relevant information that can be projected in color over the image of the crop in the glasses.

determine position and convert speech into notes

The third use case concerns position determination in the greenhouse: not only of the person with the glasses, but also of his or her viewing direction. The AR glasses then know where the wearer is looking, and therefore automatically for which plant information must be displayed. The recognition of a QR code is then no longer necessary. In the fourth use case, the glasses can convert speech into text. This is ideal for, for example, scouts or for breeders: they do not have to write down their findings with pen and paper, but can talk to the glasses.

Microsoft's HoloLens 2

The project uses Microsoft's HoloLens 2: currently the most powerful AR glasses. During the project, researchers also look at the application of crop growth models that predict production and the link with phenotype databases.

The Augmented Horticulture project is funded by the Top Sector Horticulture & Starting Materials and the consortium consisting of Syngenta Seeds, Wageningen University & Research, Florensis, LetsGrow.com, Itelligence, Mprise Agriware and KPN.

Augmented reality for Horticulture, image processing

The hololens 2 comes equipped with a camera. The image is sent to a computer. Vision algorithms on this computer can then detect and count the flowers in the image. The resulting value is send back to the hololens and displayed as an overlay for the user.

This method works for many different computer vision algorithms from ripeness to disease detection and could become a valuable tool to train new technicians.

- Unfortunately, your cookie settings do not allow videos to be displayed. - check your settings

Hololens, local climate

Greenhouses are becoming more and more advanced with sensors constantly measuring the climate of the interior of the greenhouse. Knowing these values can help a grower diagnose problems in yield and disease.

The Hololens provides a handsfree way to quickly access this data. In our test compartment we placed 16 wireless temperature and humidity sensors to monitor the micro climate within the greenhouse. As the user walks through the compartment the hololens creates a 3D map of the environment, it can also store the location of sensors within this map.

- Unfortunately, your cookie settings do not allow videos to be displayed. - check your settings

Augmented reality for Horticulture, annotation and NDVI (normalized different vegetation index)

We selected the Hololens 2 as our AR platform for our explorations in augmented reality for Horticulture , due to its state of the art array of sensors and cameras. The tasks we develop from this device will serve as prototypes for future programs in the greenhouse.

Annotation

Data plays an important role in crop breeding and growing. Traditional methods of annotation involve using a tablet or notebook to score plant traits. The Hololens can help by making this task hands free and automatic though speech to text algorithms. QR code is placed in the greenhouse, this code tells the hololens what plant is being investigated. After scanning the code the latest annotated values are retrieved from the cloud database and shown to the user. The user can now work and enter the new values by just speaking. The values then changes in Realtime as they are stored in the cloud database.

NDVI

Plant stress is also an important indicator of crop health.

Many deviancies from nutritional imbalances to fungus and herbivores to climate imbalances all effect the plants stress. The Normalized Difference Vegetation Index, or NDVI for short, can be used to detect this. NDVI combines the red channel of a color image with an infrared image to detect plant stress before it is visible to the naked eye.

These images can be saved and reconstituted into a NDVI movie from the greenhouse. A next step for this approach would be to combine it with deep learning to display on the Hololens areas of lower NDVI in real time to the user.

- Unfortunately, your cookie settings do not allow videos to be displayed. - check your settings