[{"_path":"/research/modeling","_draft":false,"_partial":false,"_locale":"en","_empty":false,"title":"Virtual plants and AI","description":"We use virtual plants to train neural networks. This allows us to detect plant organs without the need of collecting and annotating field data.","excerpt":{"type":"root","children":[{"type":"element","tag":"section-cover","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:title":""},"children":[{"type":"element","tag":"h1","props":{"id":"plant-modeling-and-ai"},"children":[{"type":"text","value":"Plant modeling and AI"}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"We focus on the generation of synthetic ground truth data (images or point clouds) using virtual plant models, and on novel analysis techniques for 3D and 3D+time data."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/modeling/1-arabidopsis-model.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"The work on the plant models has resulted in approved models for Arabidopsis thaliana and tomato plants. The model of A. thaliana was successfully used to train neural networks for the semantic segmentation of images of real plants and to produce high-quality point clouds for subsequent machine learning and analysis tasks."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/modeling/2-chenopodium-model.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"This motivates the work to go further into the realistic rendering of plants and improve the physical coherence of the generated 3D plants. Existing state-of-the-art models, including ours, do not correctly detect and handle intersecting organs, for example. This problem is currently investigated together with other key issues related to photo-realistic rendering of plants."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/modeling/3-skeletons-zoom.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"The challenge remains to robustly segment 3D plant data into its constituent organs can be tackled using several methods, from geometric methods to machine learning methods that use 2D image segmentation or 3D point cloud segmentation. An additional challenge is the precise extraction of the plant's skeleton from a 3D representation."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/modeling/4-arabidopsis.png"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"Tracking the plant growth over time raises the issue on the space-time registration of the collected 3D data. The combination of plant models and machine learning may help us predict the plant's shape ahead of time."}]}]}]},"img":"/assets/research/modeling/0-cover.png","body":{"type":"root","children":[{"type":"element","tag":"section-cover","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:title":""},"children":[{"type":"element","tag":"h1","props":{"id":"plant-modeling-and-ai"},"children":[{"type":"text","value":"Plant modeling and AI"}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"We focus on the generation of synthetic ground truth data (images or point clouds) using virtual plant models, and on novel analysis techniques for 3D and 3D+time data."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/modeling/1-arabidopsis-model.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"The work on the plant models has resulted in approved models for Arabidopsis thaliana and tomato plants. The model of A. thaliana was successfully used to train neural networks for the semantic segmentation of images of real plants and to produce high-quality point clouds for subsequent machine learning and analysis tasks."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/modeling/2-chenopodium-model.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"This motivates the work to go further into the realistic rendering of plants and improve the physical coherence of the generated 3D plants. Existing state-of-the-art models, including ours, do not correctly detect and handle intersecting organs, for example. This problem is currently investigated together with other key issues related to photo-realistic rendering of plants."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/modeling/3-skeletons-zoom.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"The challenge remains to robustly segment 3D plant data into its constituent organs can be tackled using several methods, from geometric methods to machine learning methods that use 2D image segmentation or 3D point cloud segmentation. An additional challenge is the precise extraction of the plant's skeleton from a 3D representation."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/modeling/4-arabidopsis.png"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"Tracking the plant growth over time raises the issue on the space-time registration of the collected 3D data. The combination of plant models and machine learning may help us predict the plant's shape ahead of time."}]}]}],"toc":{"title":"","searchDepth":2,"depth":2,"links":[]}},"_type":"markdown","_id":"content:3.research:1.modeling.md","_source":"content","_file":"3.research/1.modeling.md","_extension":"md"},{"_path":"/research/adaptive-systems","_draft":false,"_partial":false,"_locale":"en","_empty":false,"title":"Adaptive systems","description":"Farming robots must be able to work in complex and variable environments. For example, plants are complex, time-varying objects. Outdoor fields are very uncontrolled environments, too.","excerpt":{"type":"root","children":[{"type":"element","tag":"section-cover","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:title":""},"children":[{"type":"element","tag":"h1","props":{"id":""},"children":[{"type":"element","tag":"binding","props":{"value":"$doc.title"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"We investigate advanced, open-ended learning techniques to gain insight in how farming robots can adapt their image processing capacities when facing plants on which they have not been trained, and insights in how they can learn to optimise the collection of visual information when facing complex plant scenes."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/adaptive-systems/1-curiosity.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"We work on curiosity-driven goal-directed exploration behaviours to move an image sensor around a plant. The artificial curiosity system assigns interest values to pre-defined goals, and drives the exploration towards those that are expected to maximise the learning progress."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/adaptive-systems/2-rl.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"We also train agents to move the camera using Reinforcement Learning. In this technique the agent has to learn how to map situations to actions so as to maximize a numerical reward. The learner is not told which actions to take, but instead has to discover which actions yield the most reward by trying them. In our case, the reward is derived from building an accurate 3D representation of a plant using a small number of images."}]}]}]},"img":"/assets/research/adaptive-systems/0-cover.png","body":{"type":"root","children":[{"type":"element","tag":"section-cover","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:title":""},"children":[{"type":"element","tag":"h1","props":{"id":""},"children":[{"type":"element","tag":"binding","props":{"value":"$doc.title"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"We investigate advanced, open-ended learning techniques to gain insight in how farming robots can adapt their image processing capacities when facing plants on which they have not been trained, and insights in how they can learn to optimise the collection of visual information when facing complex plant scenes."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/adaptive-systems/1-curiosity.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"We work on curiosity-driven goal-directed exploration behaviours to move an image sensor around a plant. The artificial curiosity system assigns interest values to pre-defined goals, and drives the exploration towards those that are expected to maximise the learning progress."}]}]},{"type":"element","tag":"section-split","props":{},"children":[{"type":"element","tag":"template","props":{"v-slot:media":""},"children":[{"type":"element","tag":"p","props":{},"children":[{"type":"element","tag":"img","props":{"alt":"dashboard screenshot","src":"/assets/research/adaptive-systems/2-rl.jpg"},"children":[]}]}]},{"type":"element","tag":"p","props":{},"children":[{"type":"text","value":"We also train agents to move the camera using Reinforcement Learning. In this technique the agent has to learn how to map situations to actions so as to maximize a numerical reward. The learner is not told which actions to take, but instead has to discover which actions yield the most reward by trying them. In our case, the reward is derived from building an accurate 3D representation of a plant using a small number of images."}]}]}],"toc":{"title":"","searchDepth":2,"depth":2,"links":[]}},"_type":"markdown","_id":"content:3.research:2.adaptive-systems.md","_source":"content","_file":"3.research/2.adaptive-systems.md","_extension":"md"}]