Spatio-temporal registration of plants non-rigid 3-D structures

Tian Zhang¹, Bashar Elnashef¹, Sagi Filin¹,

¹Technion - Israel Institute of Technology

Plant4D Our 4-D registration framework enables tracking and modeling plant and organ growing dynamics.

Abstract

Monitoring plants growth dynamics requires continuously tracing their evolution over time. When using point cloud data, such a process requires associating the individual organs among scans, spatially aligning them, and accounting for their evolution, decay, or split. It is common to address this challenge by abstracting the point cloud into its skeletal form and defining point correspondence by Euclidean measures. As the paper demonstrates, standard skeletonization approaches do not capture the actual plant topology, and Euclidean measures do not document its evolving form. To address this alignment challenge, we propose in this paper a registration model that traces high-degree deformations and accommodates the complex plant topology. We develop an embedded deformation graph-based solution and introduce manifold measures to trace the plant non-isometric development. We demonstrate how a path-seeking strategy and invariant features capture the plant topological form, and then use a probabilistic linear assignment solution to associate organs across scans. By minimizing deviations from rigidity, our registration form maintains elasticity, and by solving locally rigid transformations, regularized by structure-related constraints, we secure smoothness and optimality. We also demonstrate how data arrangement and linear path-finding models make our solution computationally efficient. Our model is applied on high quality laser triangulation data, commonly tested in 4-D plant registration studies, but is also verified on low resolution and noisy pointsets reconstructed from a limited number of images by multiview stereo (MVS). Results show high quality and accuracy when applied to plant species exhibiting complex geometric structures with improved performance compared to state-of-the-art methods.

Overview

we propose in this paper a form attentive representation which resolves the skeletonization matters, provides a coherent plant representation, and improves the quality of the registration. For that, we propose an alignment pipeline that enforces local rigidity with global consistency of the evolving plant structure.

1. Embeded deformation Graph formulation

To generate a compact representation for the complex plant structure, we develop an instance-attentive EDG, which follows the plant semantic information. We begin by semantically segmenting the plant to its stem- and leaf-related instances.

Illustration of our instance attentive design, which first differentiates leaf and stem instances, then extract leaf instances using BPC algorithm.

Our embedded deformation graph (EDG) is formed by the stem medial axis, geodesic curves, and orthogonal cut profiles along the leaf surface.

2. Prior-driven hierarchical correspondence

we consider a prior-driven approach to match leaf instances across scans. The matching is solved hierarchically, at the instance-level and then point-wise on the instances to address shape expansion. We formulate the instance-level matching as a linear assignment problem , a combinatorial optimization problem well-established in the fields of scheduling.

The point correspondence is driven by equally sampled points along the internodes and petioles, and along the geodesic curves connecting the insertion point and apex. In establishing correspondence between points along the leaf cut profiles a definition of left and right directions is warranted.

3. As rigid as possible Formulation with surface aware constraints

Our model of the elastic growth is also local, but for securing smoothness while maintaining local rigidity we adapt the as-rigid-as-possible (ARAP) formulation (Sorkine and Alexa, 2007) to facilitate the deformation. For greater accuracy, we extend it to better reflect the deformations the leaf undergoes. Generally, the ARAP formulation allows for generating a new deformed shape while attempting to preserve the rigidity of the reference one both locally and globally. For optiml alignment, we constraint the outcome to have a minimal distance to the target surface.

Results

Dataset Overview

Model analysis

Segmentation evaluation

Early stage

Later stage

Semantic segmentation results using SVM+DBSCAN and our method while applied on maize and tomato samples and demonstrated at dfferent growth stages.

Extreme point detection

Our extracted salient points demonstrated on a, b) tobacco and maize at day #8 of scanning, and c) tomato at day #10 of scanning. Insertion and apex-points are marked in green and red, respectively. Evaluation of apex detection for the test samples, a) distribution of leaf dimension, b) geodesic error, c) relative error (geodesic error over the leaf dimension).

Dense organ correspondence

Early stage

Later stage

Our dense correspondence between two epochs (#1 and #2) for the tobacco, maize and tomato (a,b,c) at early and advanced growth stages. The early stage and advanced stages are relative descriptions given in scan days.

Non-rigid registration evaluation

Registration on pairs of scans

Registration residual for, a) tobacco between days #6 and #8 , b, c) maize between days #2 and #4 and #8 and #10, and d, e) tomato between days #2 and #4 and #10 and #12. Points are colored by the registration residual (Eq. 13). Residuals that fall outside of the colorbar range, > 1 mm, are marked by red.

Registration on pairs of scans

Comparative evaluation of point correspondence between the source and target maize scans visualized for inspection of mapping details and quality of the transformations. a, b) correspondence for maize samples from scan days 0 & 2 by the Dif-FM and our method; c, d, e) correspondence for maize samples from scan days 4 & 6 by the MS4D, MS4D-FM and our method. An RGB color cube represents coordinate variations in the x; y, and z directions. Color from the latter scan is projected to the earlier by the transformation parameters, acting as a measure for the correct correspondence (Pan et al., 2021). Continuous mapping suggests correct correspondence, in contrast to discontinuous one. Both examples demonstrate failure using baseline methods, while ours produces accurate and consistent point correspondence.

Evaluation on sequences of scans

Registration accuracy related statistics for the sequence registration for tobacco, maize and tomato sequences. Vertical bars represent the std. associated with the point residuals per plant about the mean errors.

Application of shape correspondence on temporal Scans

Temporal correspondence tracing the growing for, a) tobacco, b) maize, and c) tomato plants. The normalized point position in last scan denotes color in the RGB color space, while corresponding points in time are associated with the same color. Points connection across scans is added to visualize consistency of correspondence in time.

Quantitative analysis on the dataset

The vanilla version of the proposed model, without point-to-surface (P2S) constraint, obtained mean residuals of 0.35, 0.17, and 0.13 mm on the selected species, while the standard deviation (std.) of residuals was 0.36, 0.22, and 0.11 mm, respectively. The final model equipped with P2S constraint, recorded up to 200% improvement in terms of registration residual, when compared to the vanilla version.

Phenological applications

Tracking organ development in time

Tobacco sequence organ-length growth in response to shaded and high-heat environments. left) organ length development in a shaded environment, right) organ length development in a high-heat environment. Dashed lines trace stem growth, and solid lines trace leaf blades growth, with color referring to individual leaves. The leaf colors in the top and side views correspond to their development graph, visually depicting the plant and organ development.

Visualizing plant growth dynamics

Temporal interpolation of a maize plant growth. Interpolation was performed between scan days, 0; 2; 4; and 6. Points position in the last scan are normalized to correspond to the RGB color space (Pan et al., 2021), while corresponding points in time are associated with the same color.

Application to lower quality 3-D pointsets

Reconstruction from MVS data

Reconstructed cotton and maize plants by 20 images and using MVS, a1) reconstructed cotton at 23 DAP, a2) leaf surface detail demonstrating high levels of noise and uneven point distribution; b1) reconstructed cotton at 30 DAP, b2) detail of the stem and petioles demonstrating high noise levels, b3, 4) top view of the reconstructed leaves with details demonstrating voids and uneven point density along the reconstructed leaf surface; c1) reconstructed maize at 43 DAP, c2-4) details demonstrating significant noise on structure, voids, and varying resolution; d1) reconstructed maize at 50 DAP, d2, 3) leaf blade details showing the effect of noise on the reconstructed surface and voids. Processing such pointsets for 4-D registration has not been seen in the literature.

Segmentation and organ correspondence

Instance segmentation and organ correspondence for, a1,2) cotton plants at 23 and 30 DAP, and c1,2) maize plants at 43 and 50 DAP. a3,4) instance segmentation results for the cotton plants, demonstrating correct segmentation of the individual plant organs and correspondence (re ected by color consistency) despite the high noise level, uneven sampling and voids. b1-4) details of the stem and petiole segmentation and correspondence results in the presence of excessive noise uneven point sampling. c2,3) maize instance segmentation and correspondence showing correct segmentation into organs and correspondence (re ected in color consistency between segmentations).

Void handling

Handling voids in the reconstruction demonstrated on a, b) cotton and maize plant leaf blades, in both showing (1) the original data with large voids, (2) data following denoising, LOP, meshing and inner loops detection, and (3) completed voids. c) demonstrative example of the stem and petioles path seeking solution with voids in the reconstructed shape, (1) the classied stem and petioles raw set of points multiple voids, (2) computed geodesic paths along the pointset to all insertion points, and (2) the extracted paths prior to the consolidation phase.

Registration on MVS

Registration residual on a) wide-leaf cotton plant (23 - 30 DAP), b) maize (43 - 50 DAP), and c) their residual distributions in the form of a quartile box plot. Registered scan points are colored by the residual magnitude, where errors that fall outside of the colorbar, > 2 mm, are marked in red. A close-up of cotton registration errors shows that where the leaf blade surface variations become more complex the P2P registration errors become more pronounced, whereas the P2S secures points being transformed to the actual surface.

Correspondence

Consistent plant point and instance correspondence on tobacco and maize samples. The color scheme is by an RGB color cube here representing coordinates in the x; y; and z directions. This coloration is applied to evaluate the correspondence quality, where continuous mapping suggests correct correspondence, in contrast to discontinuous one.

BibTeX


      @article{Zhang2023Pheno4d
        author    = {Tian Zhang and Bashar Elnashef and Sagi Filin},
        title     = {Spatio-temporal registration of plants non-rigid 3-D structures},
        journal   = {ISPRS Journal of Photogrammetry and remote sensing},
        year      = {2023},
      }