The computer-controlled broccoli harvester. Photo Pieter Blok A field full of broccoli does not grow uniformly. The variation in shapes and sizes means the farmer has to harvest selectively, picking only the mature plants every time. Robots can take over that work – as long as they can spot the ripe broccoli. Pieter Blok helped them with that in his research for his PhD (cum laude).
A huge red vehicle drives slowly across the field. The broccoli harvester from the US company that Pieter Blok (Agrosystems Research & Farm Technology Group) has been working with moves at just under one kilometre per hour. ‘That’s slow, but people harvest at about half a kilometre per hour,’ Blok says. ‘So this is faster and you save on labour.’ Broccoli is the ideal vegetable for robotic harvesting, says the agrotechnologist. ‘It is a relatively exposed and easily identifiable crop, so it lends itself pretty well to deep learning.’
All around the world, broccoli is usually harvested selectively. ‘Young broccoli comes from the nursery in so-called rooted plugs and is planted out with a machine,’ says Blok. ‘One plant may start growing immediately, while another needs more time to take root. If you harvest them all at once, you waste the small ones.’ At 40 to 50 eurocents per head of broccoli, this means significant losses for the farmer. Most selective harvesting is currently done by hand. ‘Until my PhD research, there weren’t any robots that could do it.’
Uncertain computer
Before you can get a robot to harvest broccoli, you have to take a few hurdles. ‘There are two main challenges in image processing,’ begins Gert Kootstra of the Farm Technology Group, who was involved in the project as co-supervisor. ‘The first is variation: the robot has to be able to recognize broccoli despite differences in field, soil type and light.’ Blok adds: ‘There are also variations in size, colour and texture between individual plants and cultivars.’ So Blok trained the computer system using pictures of broccoli in all shapes and variations. He sometimes made these images redder or bluer or zoomed in or out. This taught the system to recognize the vegetable in different lights and at different sizes.
On the training images, the researchers pinpoint the location of the broccoli themselves – which takes time and money. So Blok had the computer ask the researcher questions to focus and speed up its learning. ‘The system not only gives the outcome, but also indicates the level of certainty about it,’ says co-supervisor Kootstra. ‘And Pieter reduced the degree of uncertainty. Then we could test the system he had trained on millions of images.’
Five commercial broccoli harvesters are now deployed in the US
After training it with 50 to 100 images, the researchers got the system to locate mature broccoli on 14,000 unlabelled images. ‘Then we selected the images the system was least sure about. It probably got it wrong there.’ The researchers annotated 50 of these images and added them to the existing training set. ‘The system is now more confident about some of the images it was previously unsure about, because it has learned from similar examples.’
Predicting what you can’t see
The second challenge for the broccoli robot is occlusion: when something else, like leaves, covers the head of broccoli. How can the robot know whether the broccoli is ready for harvesting if it can’t see it, or barely? To address this, Blok took two photos: one with occlusion and one without. In the photo without occlusion, he precisely labelled the location and size of the broccoli. He then trained the system with the image of covered broccoli, but with the labels. ‘Then the system predicts what it cannot see,’ says Kootstra. And it works well: ‘The margin of error is low, even when the broccoli is more than half covered.’
It was clear from its first trial in America that Blok’s system was robust enough, he recalls: ‘We had developed and trained the system with Dutch images only. It hadn’t seen any American broccoli yet, but it worked right away.’ The American customer is very satisfied. Five commercial broccoli harvesters are now being deployed in the US.
The robot has to recognize broccoli despite differences in field, soil type and light
The next step is to make a European version, says Blok. ‘That one is smaller, harvests fewer rows at a time and can be pulled by a conventional tractor. The European robot will soon be launched on the market.’ The successful application of his scientific findings in the form of a field robot earned Blok a distinction for his PhD.
The broccoli robot replaces about 15 workers who would otherwise harvest the vegetables manually. It will only need one operator to sit at the controls and oversee the whole process. Is it worrying that robots will take over jobs like this? ‘I don’t think robots are putting us out of work,’ Kootstra replies. ‘They will change the type of jobs we do though, probably for the better. This kind of harvesting is not nice work, and people only do it for the money. The harvesting robot creates new kinds of jobs: for technically skilled people who develop, run and repair the system.’
From field to fishing boats
Pieter Blok and Gert Kootstra and colleagues are applying the same machine-learning system used for broccoli harvesting in Dutch trawler fisheries, to analyse the bycatch. This enables fishers to see at a glance how much bycatch is in their nets, and what it is. ‘If it’s a lot, the fishers have good reason to fish elsewhere,’ says Kootstra. That’s because the government currently requires them to bring all bycatch ashore for registration. ‘Then death is certain for those fish, and the fishermen need a lot more storage space.’ With automatic camera recording of bycatch, the fish can still be put back in the sea. ‘Some will survive and you’ve done the required documentation.’ After tests in Wageningen, the system will soon be installed on the vessels themselves.
