SWEEPER, the sweet pepper harvesting robot

Can I get and use photos of the robot?

A ZIP file with a collection of high-resolution photos of the robot can be downloaded here: These photos may be used for your own publications if the photo reference: “Source: www.sweeper-robot.eu” is given.

Where can I find detailed public information about Sweeper?

• Cordis SWEEPER project EU website

Can I make an appointment for a video/photo session with the robot in a greenhouse?

No. The SWEEPER research project was finalised on
October 31st 2018. The robot will not be demonstrated in the greenhouse anymore. Please contact jochen.hemming@wur.nl for further questions.

What kind of camera are you using?

The camera is a combined colour (RGB) and time-of-flight (TOF) model. Both methods are using the same optics and sensor chip. The result is a combined colour and distance image (RGB-D). The camera is a custom made prototype model developed especially for SWEEPER by the Swedish company Fotonic. During the Sweeper project Fotonic was acquired by another company that has discontinued the development of this specific camera. In consequence, the camera will not be marketed.

Why does the camera use a flashlight?

The robot is equipped with custom designed, very high intensity LED flashlight units to suppress the effects of changing environmental light conditions as much as possible. Strong flashlight ensures constant illumination of the images taken by the camera during the day and night as well as on overcast or sunny days. Flashing the LEDs instead of using constant light generates a much stronger light pulse. The LED modules also do not heat up and use much less energy.

Can the robot work in varying light conditions? Can it work day and night?

Yes, SWEEPER can operate in all light conditions, from broad daylight to darkness. It is equipped with a flashlight that emits a really short pulse and our camera shutter is opened during the pulse only. This blocks sunlight very effectively.

How many hours per day can the robot work? How often does it need to recharge?

General maintenance, consisting of cleaning, and daily recharging of the batteries is needed. We expect that such a robot can operate 20h/day.

Does the robot perform maturity/quality classification?

Yes. The maturity of a detected pepper is evaluated by calculating two
features for each detected pepper:

• Measuring the colour of the fruit (average hue level in the detected region).
• Measuring shape features of the fruit (perimeter/area ratio).

Pepper fruits are observed from slightly below. If the bottom part of the pepper is mature, it is most likely that the whole sweet pepper is mature. More information about this can be found in the public deliverable D5.6 (see Cordis SWEEPER project EU website).

Is it possible to harvest differently coloured fruit with this robot?

During the project we harvested yellow peppers for practical reasons: the grower that took part in the project (De Tuindershoek) grows yellow peppers only. In principle, the robot can harvest peppers of different colours (red, orange) by simply adjusting the target colour in the detection algorithms. However, the robot was not tested to do this. To be able to let the robot harvest green peppers, the detection routines must be adapted to detect green peppers in the green crop by using deep-learning image analysis techniques or special hardware like multispectral cameras, for example. Even though this was not tested, we expect this to be a viable solution with the developed knowledge in the project.

What happens if leaves disturb the image or the grasping?

Image disturbance: The prototype will not see or harvest the pepper and move on to the next detected pepper. For this reason, performing some leaf picking before harvest increases the harvest success rate. We call this modification the “modified crop”. In future we expect that a crop variety will be bred that will have a better pepper visibility.

Grasping disturbance: Currently, almost all peppers that are cut, will drop into the finger-type catching device. However, while approaching the pepper, the fingers or other parts of the end-effector may push forwards the target stem, and in such cases the pepper cannot be reached by the cutting knife. This affects the harvest performance of the prototype. We expect to mend this fully with a re-design of the catching device and by reducing the overall dimensions of the end-effector.

The peppers fall quite a long distance when dropped into the container, does this damage the fruit?

During testing we did not observe fruit damage caused by this operation. Dropping the fruits into the blue container was also a temporary solution. In the final robot concept we use a different method. The peppers are placed onto a conveyor belt, so they do no longer drop a long distance.

Can the robot perform additional tasks?

The robot prototype was not programmed to do so. However, we do expect that this will be the added value of a robot working autonomously in the greenhouse: using its already available cameras and 3D detection equipment it could gather an enormous amount of data about the crop. All this is of value to the grower for his crop management and his yield prediction. In other projects we already work on these concepts and we can not only measure data like fruit size, colour, ripeness, and anomalies. We also plan to detect diseases and even internal fruit quality.

How can performance be improved?

The experiments identified current shortcomings, bottlenecks, and improvements that can still be made to the developed robot prototype. Methods to increase the function of each submodule should be studied further. This includes investigating a different type of robot arm, a smaller design of the end-effector, a redesign of the catching device and a faster concept for fruit storage.

To what extent is human involvement necessary? Does the robot operate fully autonomously?

The existing prototype is operated with human supervision. Once it is brought into the cropping row, it can autonomously advance in the row and can harvest peppers on the left and right side of the row.In future, in a fully automated high-tech greenhouse, we expect that the robot may work autonomously 24/7 with a little downtime for maintenance, and recharging batteries. Effectively, we estimate that it will be able to work 20h per day.

What are the safety requirements? Can humans be in the greenhouse while the robot is working?

The existing research prototype has a number of safety features. In its current developmental state, it may only be operated by skilled and trained personnel, and humans should keep a 2m distance to the working robot. Under existing conditions this means that no other humans are allowed to enter the next cropping row to the left and to the right of the one the robot is working in.

The current machine guidelines do not allow humans to work closely to the robot, because it can move too fast and its mass is too high. However, working in a closed environment makes it slightly easier to introduce robots than in outdoor conditions (e.g. autonomous cars and precision agriculture).

In future we expect that human-robot collaboration will be a very common approach and a good solution to profit from skills from both humans and robots. Existing guidelines are required to be adapted for these set-ups. Authorities already showed interest in discussing the options.

Can we integrate the robot with existing packaging or sorting equipment?

Yes, that is the intention. In fact, the company involved in this project (Bogaerts Greenhouse Logistics) already has the basics on the market. The step that remains to be solved, is the automatic traffic management for the harvest trolleys and robots, localisation, and guidance. And last but not least, the complete assurance that the robot will not harm any humans working in its vicinity in the greenhouse.

Do I have to hire technical staff to operate and maintain the robot? What expertise do I need to operate it?

We expect that the number of autonomous vehicles and robots in greenhouses will increase. The final plan for maintaining those must be made still. We expect that a technical qualified person is needed at each greenhouse site. A helpdesk is needed to give long-distance support worldwide.

What is the speed? Success %?

The robot was designed for a single-row plant-stem cropping system using an optimised pepper cultivar for robotic harvesting. As this optimised cultivar is not yet available, it was tested in a commercial greenhouse first. For the double-row plant-stem growing system 18% of ripe fruit were harvested in the commercial crop and 49% of ripe fruit were harvested in the modified crop. For the modification most occluding leaves and fruit clusters were pruned away beforehand. Under a single-row plant-stem cropping system assumption 29% of ripe fruit were harvested in a commercial crop and 61% in a modified crop. The average cycle time to pick one fruit was 24 seconds. During the greenhouse experiments the robot was not operated at maximum speed for security reasons. Under laboratory conditions we have proofed that the time to pick one fruit is not more than 15 seconds.

How well does the robot cut peduncles (fruit stems)?

The cutting device makes very clean cuts, very closely to the plant stem, often at, or close, to the so-called abscission zone. In experiments we proofed that neither the fruit nor the remaining crop have a higher risk on infections. Further, the quality and shelf life of the picked fruits are not affected.

What progress did you achieve in Sweeper compared to Crops?

In Sweeper we made a great improvement compared to our previous research robot Crops. That robot was able to harvest 6% in the commercial, unmodified crop and 33% of the fruit in the modified crop. Furthermore it took the crops robot to harvest a pepper about 94 seconds. With the Sweeper robot we can harvest 29% of ripe fruit in a commercial crop and 61% in a modified crop (under a single row plant stem cropping system assumption – where the robot was designed for). The average cycle time to pick one fruit in the greenhouse with Sweeper was 24 seconds. Under laboratory conditions we have proofed that the time to pick one fruit is not more than 15 seconds.

How much will the robot cost?

Based on a robot that can harvest one pepper per 12 seconds and has a harvest success rate of 95%, our economic analysis showed that the room for investment for one robot is between 75 and 100k Euros assuming a payback time of 7 years. However, we cannot forecast what the actual cost of a single robot. For growers, a total solution with several robots, autonomous ground vehicles (AGVs), and a fully automated logistic system is important, and the total costs should be competitive with current systems.

Can I buy the robot?

The SWEEPER robot is a unique prototype that is the result of a research project. It is not possible to buy (copies of) it. We do expect that after some further development, commercial pepper harvesting robots should be on the market in about 3-5 years.

What are the main bottlenecks for market introduction?

The crop: Peppers should have long peduncles, little clustering, long internode distances and good visibility. The results in this research project were obtained in a pruned crop by clearing peppers from clusters and some leaves. We expect that in 4-5 years breeders will be able to develop new commercial cultivars that are more suitable for robotic harvesting to achieve a success rate of 90-100%.

Economics: To develop a future single-stem row-cropping system that is robotised and has a similar or even higher overall economic revenue compared to the current double-stem-row cropping system. We expect that in a very densely planted single-stem-row system the net crop yield can be slightly higher, which will work positively.

How many robots do I need for 1 ha of greenhouse?

This will depend on the final speed. We estimate that 1 robot per ha will be needed.

Is there a patent filed for (parts of) the robot?

Yes, we have filed a patent for the end-effector.

Will there be a follow-up project?

The SWEEPER project was accomplished on 31 October 2018. We are currently looking into possibilities to acquire a follow-up project.

Why did you choose to harvest peppers and not tomatoes, for example?

For high-tech greenhouses cucumbers, tomatoes, and peppers are the most widely cultivated crops. We already worked on cucumbers some 20 years ago. Many activities in robotic harvesting of tomatoes have been ongoing in several countries for many years already. At the time we started the SWEEPER project, we believed that peppers would pose the biggest challenge, and we decided to go for it and push the boundaries of science as much as possible.

Over 15 years ago you showed a cucumber harvesting robot. What happened with it, why is it not on the market?

When we delivered that robot we realised that the market for high-wire cultivation technology was not yet matured and the need for automation was not yet so high. As such, at that time there was no real business case for a cucumber harvester. We do believe that this has changed these days. With the building-blocks from Sweeper, we think it would be feasible to get a cucumber harvester onto the market in a relatively short time.

How much public money did this project cost?

The EU budget used within Sweeper was about 4 million Euros. Additionally, another approximately 10% came from National public funds.

What happened to your partner IRMATO?

Our partner IRMATO filed for bankruptcy halfway through the project. IRMATO’s activities in the Sweeper project stopped on 31 January 2018. With IRMATO we were able to develop and test a first prototype we called the ‘basic system’. The final prototype, the ‘advanced system’, was built around the existing greenhouse technologies of new partner Bogaerts Greenhouse Logistics, like the harvesting trolley and the mobile platform. We also completely redesigned the end-effector concept.