Learning global spatial information for multi-view object-centric models

Abstract

Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene and infer object-centric representations from several observation views. In general, multi-object scenes can be uniquely determined if both the properties of individual objects and the spatial arrangement of objects are specified; however, existing multi-view object-centric models only infer object-level representations and lack spatial information. This insufficient modeling can degrade novel-view synthesis quality and make it difficult to generate novel scenes. We can model both spatial information and object representations by introducing a hierarchical probabilistic model, which contains a global latent variable on top of object-level latent variables. However, how to execute inference and training with that hierarchical multi-view object-centric model is unclear. Therefore, we introduce several crucial components which help inference and training with the proposed model. We show that the proposed method achieves good inference quality and can also generate novel scenes properly.

GRAPHICAL ABSTRACT

Keywords:

• Deep generative models
• variational autoencoder(VAE)
• object-centric representation
• disentanglement

1. Introduction

Extracting representations of individual objects from direct perception (e.g. visual input) is called object-centric representation learning. It is considered to be effective in many aspects such as robotics [1–3], reasoning [4], and sample efficiency [5]. Among them, the deep generative model (DGM)-based method is one of the promising directions, as it can infer rich information about objects in a fully unsupervised manner, and some of its models can also generate novel scenes [6–13].

Since this kind DGM-based models capture and reconstruct the environment from limited observation, this can be used as a simulator of the environment. Thus, they are regarded as so-called world model [14] or internal model [15]. World model is a basis of model-based reinforcement learning(RL), and it has been used for RL agents and robots [16,17].

Recently, several studies have been working on multi-view object-centric models, which predict unobserved views of a scene: novel view synthesis and infer object-centric representations from several observation views [18,19]. In addition, this setting can also be regarded as an object-centric expansion of the Generative Query Network (GQN) [20].

To represent multi-object scenes properly, the properties of individual objects and the spatial arrangement of objects should be specified. However, existing multi-view object-centric methods only explicitly model representations of individual objects, and each of them includes its own spatial information separately. Namely, the relationship between objects is not represented in this case. This modeling can degrade novel view synthesis and segmentation because the model needs to solve occlusions and spatial ambiguity virtually without prior knowledge about objects' spatial relationships. This can be serious especially when the number of available observation views is limited. In addition, this modeling can also lead to the inability to generate physically plausible novel scenes, because they need to place objects independently, which often brings about collisions and misplacement of objects.

Spatial information of the whole scene can be represented by introducing another latent variable besides object-level latent variables. This variable can be independent of object-level latent variables or can be modeled as a global latent on top of object-level latent variables. In the former formulation, those latent variables need to be disentangled during training, which can often be challenging. From another point of view, it is a common assumption in cognitive science that humans recognize the whole and parts, or objects, separately [21]. In feature integration theory [22], visual feature maps become ‘master map of locations’ combined with positions of objects in visual space. Then, attention binds those features to recognize individual objects. Modeling the spatial arrangement of objects as a global latent variable closely adheres to this point of view due to its hierarchical property. For the above reasons, we believe that introducing a global latent variable is a more natural step for improving object-centric generative models.

In this paper, we build a new multi-view object-centric model which has a latent variable for global spatial information to improve inference: novel view synthesis and segmentation. In addition, our model can also generate physically plausible novel scenes. The problem is that how to execute inference and training with the proposed model is unclear. In regards to this point, we propose several crucial components: Global Encoder, Sequential Encoder, and Structured Prior. Firstly, the model needs to infer global representation from variable number of input views, thus we introduced an encoder using Product of Experts (PoE) [23], which we refer to as Global Encoder. Also, we applied Normalizing Flow (NF) [24] to enhance the representational capacity of the latent variable. Secondly, we introduced an encoder using an auto-regressive model called Sequential Encoder. This module supports the Global Encoder and stabilizes inference and training with the hierarchical probabilistic model of the proposed method. Lastly, the structured, complex posterior of object-level latent variables need to be sampled, including spatial relationships between objects in order to generate physically plausible novel scenes. Therefore, we introduced a learnable prior called Structured Prior, which is implemented by Transformer [25].

The proposed model has representation about the whole, or spatial information of a scene and has representations about a part of a scene (i.e. individual properties of objects); hence, we call this model as Whole-Part Representation Learning Model for Object-Centric 3D Scene Inference and Sampling (WeLIS).

Contributions

• We introduce a multi-view object-centric model with a global latent variable that represents the spatial configuration of objects in a scene.

• We introduce essential components for introducing a global latent variable as well as stable learning and inference for our multi-view object-centric model.

• We show that the proposed model performs well in terms of inference quality, and can generate novel scenes and corresponding segmentation masks properly, unlike existing methods.

2. Related work

Object-Centric Models: The objective of object-centric representation learning is to obtain representations of each object from visual input. It is applicable to many domains such as robotics [1,2,27], reasoning [4] and reinforcement learning [5]. One of the promising directions of object-centric representation learning is variational autoencoder (VAE)-based [28] methods, as they can obtain rich information about objects that can also be disentangled, and some of them can generate novel scenes outside of the empirical distribution [8–13]. VAE-based object-centric models are often referred to as ‘scene interpretation models’ due to their inference ability. Some models are capable of object-centric generation [29–31]; however, these methods focus on generation and not for inference.

Multi-view information: In a robotics field, multi-view information has been used for many area such as 3D reconstruction as stereo imaging [32], SLAM and novel view synthesis [33], and 3D pose estimation of rooms and objects [34]. In terms of neural network-based methods, Generative Query Network (GQN) [20] for novel view synthesis and Neural Radiance Fields (NeRF) [35] for 3D modeling depend on multi-view information.

Multi-View Object-Centric Models: Lately, several studies have been working on multi-view scene interpretation models [18,19]. These methods predict images of a 3D scene from arbitrary viewpoints in an object-centric way, using several observation views as input. This problem setting can be regarded as an object-centric expansion of GQN family [20,26,36]. The above-mentioned single-view setting is referred to as multi-object-single-view (MOSV), and the multi-view setting is defined as multi-object-multi-view (MOMV) in [18].

Table 1 compares methods related to the MOMV setting. Novel view synthesis stands for prediction of unobserved viewpoints, and novel scene generation stands for random generation of novel scenes. Among these models, ROOTS and MulMON [18,19] are the only generative models that work on a MOMV setting. In addition to this, O3V [37] have a similar problem setting, but it takes a fixed number of viewpoints as input. ObSuRF [38] and uORF [39] also work on MOMV setting, but these are not generative models and are based on Slot Attention [7] and NeRF [35]. Therefore, these methods cannot generate novel samples and do not obtain low-dimensional representation space, or latent space.

Table 1. Comparison of the methods related to Multi-Object-Multi-View (MOMV) setting.

Model	Object-centric	Novel view synthesis	Inference	Sampling (NSG${}^{\mathrm{a}}$)
GQN	✗	✓	✓	✓ [26]
ObSuRF, uORF${}^{\mathrm{b}}$	✓	✓	✓	✗
ROOTS	✓	✓	✓	✗
MulMON	✓	✓	✓	✗${}^{\mathrm{c}}$
WeLIS (Ours)	✓	✓	✓	✓

${}^{\mathrm{a}}$NSG stands for Novel Scene Generation

${}^{\mathrm{b}}$These are not generative models. ${}^{\mathrm{c}}$MulMON can generate objects separately, but this results in inconsistent, blurred images.

In this paper, we introduce a new MOMV model with a global latent variable, which can potentially represent the spatial arrangement of objects in a scene. The proposed model is the only MOMV model that can generate novel scenes. In this paper, we set MulMON as our baseline model due to the following reasons. Firstly, MulMON uses segmentation masks to represent objects' regions in contrast to ROOTS which uses bounding boxes. Bounding box-based methods, which originate from [11] tend to be more sensitive to object size and shape [40]. Secondly, MulMON is more robust in terms of the number of observation views required. Although we built our model based on MulMON, a similar approach can be applied to ROOTS and other methods.

Introducing the global latent variable itself has been done in some studies in various areas [41,42]. In particular, Generative Neurosymbolic Machine [13] introduces a global latent variable to bounding box-based single-view object-centric model for random generation. However, how to execute stable inference and training with the proposed MOMV model is unclear and is thus the point of investigation in this paper.

3. Method

Our objective was to introduce a MOMV model that has both object-level representations and the spatial arrangement of the scene to obtain further inference quality and novel scene generation ability.

We achieved this by introducing a global latent variable for global spatial information on top of object latent variables, which is formulated as a hierarchical probabilistic model. In this paper, we introduce several crucial components to realize stable inference and training with this MOMV model. An overview of the proposed model is shown in Figure 1.

Figure 1. Overview of the proposed model. In this example, the model predicted an image and corresponding segmentation masks from unobserved query viewpoint ${\mathbf{v}}^q$, using three observation views ${\mathbf{x}}^1$-${\mathbf{x}}^3$. ${\mathbf{z}}^g$ is a global latent variable that represents spatial relationships between objects, and ${\mathbf{z}}_k$ is object-level latent variables that represent each object. Note that the ${\hat{x}}^q$ and segmentation shown in this figure are the actual results from our model.

In this section, we first explain the problem setting in this paper, and then the probabilistic model of our proposed model. Lastly, the training method and additional details of the introduced components are explained.

3.1. Problem setting

The target of this research is 3D scenes with multiple objects, and thus a dataset with N data points can be defined as $\mathcal{D}=\{{\mathbf{X}}_1,\dots ,{\mathbf{X}}_N\}$, where ${\mathbf{X}}_i=\left\{\right({\mathbf{x}}_i^1,{\mathbf{v}}_i^1),\dots ,({\mathbf{x}}_i^M,{\mathbf{v}}_i^M\left)\right\}$, which corresponds to a scene with M views, and $\mathbf{v}$ is a viewpoint vector that represents the camera pose.

The task is to predict queried unobserved views of the scene and infer K representations $\mathbf{z}=\{{\mathbf{z}}_1,\dots ,{\mathbf{z}}_K\}$ given several observations $\left\{\right({\mathbf{x}}_i^1,{\mathbf{v}}_i^1),\dots ,({\mathbf{x}}_i^{M_{obs}},{\mathbf{v}}_i^{M_{obs}}\left)\right\}$. Here, each ${\mathbf{z}}_k$ corresponds to an object in the scene. In practice, the M views in ${\mathbf{X}}_i$ is split into $M_{obs}$ observations and $M_{qry}$ queries in the training process. $M_{obs}$ can be varied in each iteration if necessary.

3.2. Probabilistic model

3.2.1. Generative model

As mentioned above, we introduced the hierarchical probabilistic model with global latent ${\mathbf{z}}^g$ on top of object-level latent variables $\mathbf{z}=\{{\mathbf{z}}_1,\dots ,{\mathbf{z}}_K\}$. In the following sections, object-level latent variables are abbreviated as $\mathbf{z}$, as required. The marginal likelihood is thus: (1) $\begin{array}{r}{p}_{\gamma }(\mathbf{x}\mid \mathbf{v})=\int \int p_{\theta }(\mathbf{x}|\mathbf{z},\mathbf{v})p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g)p\left({\mathbf{z}}^g\right)\mathrm{d}\mathbf{z}\mathrm{d}{\mathbf{z}}^g,\end{array}$(1) where $\gamma =\{\theta ,\varphi \}$. The generative model $p_{\theta }(\mathbf{x}|\mathbf{z},\mathbf{v})$ is a spatial Gaussian mixture model derived from prior works [8,18] that can be written as: (2) $\begin{array}{r}{p}_{\theta }(\mathbf{x}|\mathbf{z},\mathbf{v})=\prod _{j=1}^D\sum _{k=1}^K{p}_{\theta }(C_j=k\mid {\mathbf{z}}_k,\mathbf{v})\otimes p_{\theta }(x_{jk}|{\mathbf{z}}_k,\mathbf{v}),\end{array}$(2) where $D\in \mathbb{N}$ is the number of image pixels, and K is the number of mixture components which are often referred to as ‘slots’. The factor $p_{\theta }(C_j=k\mid {\mathbf{z}}_k,\mathbf{v})$ is a pixel-wise assignment of slots, or segmentation mask. The decoder has parameters θ and implemented by a broadcast decoder [43]. ${\mathbf{z}}_k$ and $\mathbf{v}$ are first concatenated and projected onto a single vector ${\mathbf{f}}_k$ by an MLP and then fed to the decoder. The decoder generates K components $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_K\}$ and corresponding segmentation masks from $\{{\mathbf{f}}_1,\dots ,{\mathbf{f}}_K\}$.

The second factor in the RHS of Equation (1(1) $\begin{array}{r}{p}_{\gamma }(\mathbf{x}\mid \mathbf{v})=\int \int p_{\theta }(\mathbf{x}|\mathbf{z},\mathbf{v})p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g)p\left({\mathbf{z}}^g\right)\mathrm{d}\mathbf{z}\mathrm{d}{\mathbf{z}}^g,\end{array}$(1) ): $p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g)$ is a learnable prior, which we refer to as the Structured Prior, and the third factor is a standard Gaussian distribution. The Structured Prior is explained in Section 3.3.

3.2.2. Inference model

We used amortized variational inference [28] and computed variational approximate posteriors because the marginal likelihood in Equation (1(1) $\begin{array}{r}{p}_{\gamma }(\mathbf{x}\mid \mathbf{v})=\int \int p_{\theta }(\mathbf{x}|\mathbf{z},\mathbf{v})p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g)p\left({\mathbf{z}}^g\right)\mathrm{d}\mathbf{z}\mathrm{d}{\mathbf{z}}^g,\end{array}$(1) ) is intractable. We factorized the variational posteriors as: (3) $\begin{array}{r}{q}_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})=q_{\psi }({\mathbf{z}}_1,\dots ,{\mathbf{z}}_K\mid {\mathbf{z}}^g)q_{\psi }({\mathbf{z}}^g\mid \mathbf{X}),\end{array}$(3) where $\mathbf{X}=\left\{\right({\mathbf{x}}^1,{\mathbf{v}}^1),\dots ,({\mathbf{x}}^{M_{obs}},{\mathbf{v}}^{M_{obs}}\left)\right\}$. The first RHS term and second RHS term are approximate posteriors of $\mathbf{z}$ and ${\mathbf{z}}^g$ respectively. We refer to these encoders as Sequential Encoder and Global Encoder.

The Global Encoder adopts PoE [23] to deal with a variable number of input observation views $M_{obs}$. In particular, we used PoE with prior expert [44]. In addition to this, we applied normalizing flow (NF) [24] after the PoE. Although this is optional, the posterior should be expressive enough to represent the spatial configuration of complex 3D multi-object scenes. Therefore, the variational approximate posterior is: (4) $\begin{array}{rl}& {\mathbf{z}}^g\sim q_{\psi }({\mathbf{z}}^g\mid \mathbf{X})\\ & =\left(p\right({\mathbf{z}}_1^g\left)\prod _{v=1}^{M_{obs}}{q}_{\psi }\right({\mathbf{z}}_1^g\mid {\mathbf{x}}_v,{\mathbf{v}}_v\left)\right)\prod _{t=1}^T{|det\frac{\mathrm{\partial }{f}_{\psi }}{\mathrm{\partial }{\mathbf{z}}_t^g}|}^{-1},\end{array}$(4) where T is the number of transformations of NF, and $M_{obs}$ is the number of available observation views. We used planar flow for its simplicity, but any methods can be applied and can improve the results potentially. Thus, the transform $f_{\psi }$ in the above equation is $f_{\psi }\left({\mathbf{z}}_t^g\right)={\mathbf{z}}_t^g+{\mathbf{u}}_t\cdot h({\mathbf{w}}^T{\mathbf{z}}_t^g+{\mathbf{b}}_t).$The Sequential Encoder was implemented in auto-regressive manner using LSTM [45]. We tried several architectures, but non-auto-regressive models were unstable as long as we have tested. Firstly, this auto-regressive part infers K initial latent variables ${\mathbf{z}}_1^0,\dots ,{\mathbf{z}}_K^0$ and then, they are updated using iterative amortized inference (IAI) [46] as MulMON and IODINE [9] do. However, we omitted image-sized inputs as in [47]. Thus, the Sequential Encoder with parameter ψ can be defined as: (5) $\begin{array}{rl}& {\mathbf{z}}_1^0,\dots ,{\mathbf{z}}_K^0\sim \prod _{k=1}^K{q}_{\psi }({\mathbf{z}}_k^0\mid {\mathbf{z}}_{1:k}^0,{\mathbf{z}}^g),\end{array}$(5) (6) $\begin{array}{rl}& {\mathbf{z}}_k^{l+1}={\mathbf{z}}_k^l+g_{\psi }({\mathbf{z}}_k^l,{\mathbf{z}}^g,{\mathrm{\nabla }}_{{\mathbf{z}}_k^l}{\mathcal{L}}_l^{IAI}),\end{array}$(6) where, subscript l is the update steps, ${\mathrm{\nabla }}_{{\mathbf{z}}_k^l}{\mathcal{L}}_l^{IAI}$ is a gradient of negative log-likelihood in that iteration, and $g_{\psi }$ is a refinement network that is implemented by an MLP followed by an LSTM. Then, the posterior can be updated arbitrary times. The details of ${\mathcal{L}}_l^{IAI}$ is explained in the next section.

3.2.3. Training

All the parameters in this model can be trained end-to-end by maximizing evidence lower bound (ELBO) on the log-marginal likelihood $\log p({\mathbf{x}}_1,\dots ,{\mathbf{x}}_N)$. The ELBO of the proposed model is (7) $\begin{array}{rl}{\mathcal{L}}_{ELBO}& =E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log \frac{p_{\theta }(\mathbf{x}|\mathbf{z},\mathbf{v})p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g)p\left({\mathbf{z}}^g\right)}{q_{\psi }(\mathbf{z}\mid {\mathbf{z}}^g)q_{\psi }({\mathbf{z}}^g\mid \mathbf{X})}]\\ & =E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{v}\left)\right]\\ & -KL\left[q_{\psi }\right({\mathbf{z}}^g\mid \mathbf{X})\mid \mid p({\mathbf{z}}^g\left)\right]\\ & -E_{q_{\psi }({\mathbf{z}}^g\mid \mathbf{X})}\left[KL\left[q_{\psi }\right(\mathbf{z}\mid {\mathbf{z}}^g)\mid \mid p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g\left)\right]\right].\end{array}$(7) Here, KL stands for Kullback–Leibler divergence. Then, introducing two coefficient ${\text{β}}_1$ and ${\text{β}}_2$ for balancing KL terms, the objective can be written as (8) $\begin{array}{rl}\mathcal{L}& =E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{v}\left)\right]\\ & -{\text{β}}_{1}KL\left[q_{\psi }\right({\mathbf{z}}^g\mid \mathbf{X})\mid \mid p({\mathbf{z}}^g\left)\right]\\ & -{\text{β}}_2{E}_{q_{\psi }({\mathbf{z}}^g\mid \mathbf{X})}\left[KL\left[q_{\psi }\right(\mathbf{z}\mid {\mathbf{z}}^g)\mid \mid p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g\left)\right]\right].\end{array}$(8) Moreover, due to the iterative process of IAI, negative log-likelihood is computed as (9) $\begin{array}{rl}& E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{v}\left)\right]=\frac{2}{L(L+1)}\sum _{l=0}^{L-1}(l+1){\mathcal{L}}_l^{IAI},\end{array}$(9) (10) $\begin{array}{rl}& {\mathcal{L}}_l^{IAI}={\mathbb{E}}_{{\mathbf{z}}^l\sim q_{\psi }({\mathbf{z}}^l,{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}|{\mathbf{z}}^l,\mathbf{v}\left)\right],\end{array}$(10) where L is the number of iterations in IAI. We set L = 5 in this paper. As shown in Equation (9(9) $\begin{array}{rl}& E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{v}\left)\right]=\frac{2}{L(L+1)}\sum _{l=0}^{L-1}(l+1){\mathcal{L}}_l^{IAI},\end{array}$(9) ), log-likelihoods in the later steps of iterative refinement are emphasized as in MulMON and IODINE.

In addition to this, we used GECO [48] as in some existing methods [10,47]. GECO reinterprets the maximization of ELBO as a constrained optimization problem that takes KL terms as the main objective and log-likelihood as the constraint. This optimization is done by the method of Lagrange multipliers, namely, the coefficient of KL terms β is regarded as a Lagrange multiplier and automatically tweaked during training. GECO is not necessary for our model, but a proper balancing of β further improves random generation quality.

The reinterpreted optimization objective thus becomes (11) $\begin{array}{rl}& \arg \underset{\psi ,\varphi }{min}KL\left[q_{\psi }\right({\mathbf{z}}^g\mid {\mathbf{x}}_{obs},{\mathbf{v}}_{obs})\mid \mid p({\mathbf{z}}^g\left)\right]\\ & +E_{q_{\psi }({\mathbf{z}}^g\mid \mathbf{X})}\left[KL\left[q_{\psi }\right(\mathbf{z}\mid {\mathbf{z}}^g)\mid \mid p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g\left)\right]\right]\\ & \mathrm{s}\mathrm{u}\mathrm{c}\mathrm{h}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t}-E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{v}\left)\right]\leqq R,\end{array}$(11) where $R\in \mathbb{R}$ is the maximum allowed negative log-likelihood, which is a hyperparameter in GECO instead of β, and ${\mathbf{x}}_{obs}$, ${\mathbf{v}}_{obs}$ stands for observation views. Note that even when we use GECO, adjusting relative strength between two KL terms in 11(11) $\begin{array}{rl}& \arg \underset{\psi ,\varphi }{min}KL\left[q_{\psi }\right({\mathbf{z}}^g\mid {\mathbf{x}}_{obs},{\mathbf{v}}_{obs})\mid \mid p({\mathbf{z}}^g\left)\right]\\ & +E_{q_{\psi }({\mathbf{z}}^g\mid \mathbf{X})}\left[KL\left[q_{\psi }\right(\mathbf{z}\mid {\mathbf{z}}^g)\mid \mid p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g\left)\right]\right]\\ & \mathrm{s}\mathrm{u}\mathrm{c}\mathrm{h}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t}-E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{v}\left)\right]\leqq R,\end{array}$(11) using coefficients still makes sense, but we omitted them for clarity. Although all the parameters can be trained end-to-end, we found that end-to-end training, including the parameters of Structured Prior ϕ is slightly unstable and often degrades novel scene generation. Thus, we introduced a separate training strategy, in which ϕ is updated after the training of the other parameters. The detail of this method is explained in the next section.

3.3. Structured prior

In this section, we explain the detail of the Structured Prior and corresponding separate training strategy.

First of all, the Structured Prior is defined as $p_{\varphi }({\mathbf{z}}_1^L,\dots ,{\mathbf{z}}_K^L\mid {\mathbf{z}}^g)$.

As we mentioned in Equation (6(6) $\begin{array}{rl}& {\mathbf{z}}_k^{l+1}={\mathbf{z}}_k^l+g_{\psi }({\mathbf{z}}_k^l,{\mathbf{z}}^g,{\mathrm{\nabla }}_{{\mathbf{z}}_k^l}{\mathcal{L}}_l^{IAI}),\end{array}$(6) ), the latent variables of each slot are iteratively updated by IAI in the inference process. However, it makes sampling from the updated posterior ${\mathbf{z}}_1^L,\dots ,{\mathbf{z}}_K^L$ difficult. Though IAI updates latent variables using reconstruction error and auxiliary inputs, the generative model can not use such information and need to blindly capture this refinement process without them.

Therefore, in order to model this process, we implemented Structured Prior as Transformer [25]. Firstly, initial K variables ${\mathbf{z}}_1^0,\dots ,{\mathbf{z}}_K^0$ are computed from ${\mathbf{z}}^g$ using the Sequential Encoder $q_{\psi }({\mathbf{z}}^0\mid {\mathbf{z}}^g)$. Then, Transformer takes the K latent variables as input and approximates updated ones ${\mathbf{z}}_k^L$. Note that the parameter of Sequential Encoder is shared with inference.

3.3.1. Separate training

As we mentioned above, we introduced a separate training method for the Structured Prior. In the first stage, WeLIS is trained without the parameters of Structured Prior ϕ. Namely, ϕ is frozen and ${\text{β}}_2$ in Equation (8(8) $\begin{array}{rl}\mathcal{L}& =E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{v}\left)\right]\\ & -{\text{β}}_{1}KL\left[q_{\psi }\right({\mathbf{z}}^g\mid \mathbf{X})\mid \mid p({\mathbf{z}}^g\left)\right]\\ & -{\text{β}}_2{E}_{q_{\psi }({\mathbf{z}}^g\mid \mathbf{X})}\left[KL\left[q_{\psi }\right(\mathbf{z}\mid {\mathbf{z}}^g)\mid \mid p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g\left)\right]\right].\end{array}$(8) ) is set to zero or the second term of Equation (11(11) $\begin{array}{rl}& \arg \underset{\psi ,\varphi }{min}KL\left[q_{\psi }\right({\mathbf{z}}^g\mid {\mathbf{x}}_{obs},{\mathbf{v}}_{obs})\mid \mid p({\mathbf{z}}^g\left)\right]\\ & +E_{q_{\psi }({\mathbf{z}}^g\mid \mathbf{X})}\left[KL\left[q_{\psi }\right(\mathbf{z}\mid {\mathbf{z}}^g)\mid \mid p_{\varphi }(\mathbf{z}\mid {\mathbf{z}}^g\left)\right]\right]\\ & \mathrm{s}\mathrm{u}\mathrm{c}\mathrm{h}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t}-E_{q_{\psi }(\mathbf{z},{\mathbf{z}}^g\mid \mathbf{X})}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z},\mathbf{v}\left)\right]\leqq R,\end{array}$(11) ) is excluded if we use GECO. Then, as a second stage, ϕ is updated while other parameters are frozen.

4. Experiments

We evaluated our model with three different datasets derived from the MulMON paper [18]. CLEVR Multi-View (CLEVR MV) is a multi-view version of the CLEVR dataset [49], and CLEVR Augmented (CLEVR AUG) extends CLEVR MV with more complex and various objects for additional difficulty. GQN Jaco is a simulation-based robot arm dataset introduced originally in [20]. The dimension of viewpoint vector in both CLEVR datasets is three since it always faces toward the center of the scenes, and is seven for GQN Jaco as defined in the original dataset. Ultimately, the viewpoints are supposed to be actively inferred by the agents to capture the scene accurately, they are fixed in this experiment. Figure 2 shows examples of these datasets. All the datasets are used in $64\times 64$ resolution ¹. Up to six views are given as observation during training, and we trained all the models with Adam optimizer for $300k$ iterations. Besides, we adapted the official implementation of MulMON for reproduction.

Figure 2. Ground truth samples of the three datasets: CLEVR MV, CLEVR AUG and GQN Jaco.

In the following sections, firstly, we evaluated inference quality, i.e. novel view synthesis and segmentation quantitatively and qualitatively. Then, we show WeLIS can also generate physically plausible novel scenes, in contrast to the previous works. Lastly, we conducted a few experiments to look into the global latent space ${\mathbf{z}}^g$.

4.1. Novel view synthesis and segmentation

In this section, we evaluate novel view synthesis and segmentation quality. Figure 3 shows quantitative evaluation of three models: WeLIS, WeLIS without NF (ablation) and MulMON. We used two metrics for the evaluation of novel view synthesis. Image quality is evaluated by Root Mean Squared Error (RMSE), and segmentation quality is evaluated by mean Intersection over Union(mIoU). Note that lower is better for RMSE, and higher is better for mIoU. Random four seeds are used to calculate these scores. As GQN Jaco does not provide ground truth segmentation, we instead show qualitative results in Figure 4. In addition, Figure 5 shows an evaluation about observation viewpoints, namely, this figure shows simple reconstruction and segmentation quality without novel view synthesis.

Figure 3. Quantitative evaluation of novel view synthesis. Lower is better for root mean squared error(RMSE), and higher is better for mean intersection over union (mIoU). mIoU is not available for GQN Jaco dataset since ground truth segmentation masks are not provided.

Figure 4. Novel view synthesis results on GQN Jaco with four difference scenes. These images are prediction of unobserved query views. The number of observation views is three (N = 3).

Figure 5. Quantitative evaluation of reconstruction and segmentation. This figure is about observed views, i.e. input images in contrast to 3 which is about unobserved views.

We can see that WeLIS outperforms MulMON in CLEVR AUG, GQN Jaco and CLEVR MV with a small number of observation views N. WeLIS is relatively stable in terms of N, which means that WeLIS is good at inferring the whole scene from limited, partial observation. Introducing global latent in addition to object-level latent variables leads to more precise modeling of the world, thus we consider that it helps inference with a small number of observations. As for GQN Jaco (Figure 4), WeLIS has cleaner segmentation and captures the balls more consistently.

In addition, we show some samples from WeLIS and MulMON in Figure 6 with their RMSE and mIoU to show how quantitative difference affects perceptual quality. There is a distinctive difference between mIoU and segmentation quality. Though the difference about RMSE is relatively unnoticeable to human eyes, we can see that WeLIS has slightly sharper images, especially for objects with a complex shape such as dolphins and ducks.

Figure 6. Quantitative scores of novel view synthesis and qualitative comparisons. The left most column in each group show ground truth (GT) images from an unobserved viewpoint. The second columns in each group show images predicted by the model (Pred.). The third columns show segmentation masks (Seg.). The number of observation views N is three in this figure.

4.1.1. Ablation studies (Novel view synthesis)

We validate each component introduced in WeLIS by ablation studies. Figure 7 shows ablation results of the components and training methods.

Figure 7. Ablation studies of each component: Global Encoder (PoE), GECO, separate training of Structured Prior. ‘observation’ is about observed view points and ‘query’ is about unobserved ones.

As for NF (gray lines), it is important for stable inference and training. Without NF, the model often falls into local minima that does not properly represent one object per slot. This can be confirmed by relatively large standard errors. However, when the training succeeded, the model achieved the almost same performance as the full model.

Also, we can see that other components and training methods also contribute to the result: separate training of Structured Prior, PoE encoder, and GECO.

4.2. Novel scene generation

In this section, we evaluate novel scene generation (random generation) results. Figure 8 shows component-by-component generation results. We can see that WeLIS successfully generates objects and segmentation masks, and as a result, generates physically plausible novel scenes from multiple viewpoints. On the other hand, MulMON generates blurred and spatially ambiguous images. Top row of Figure 9 shows more samples. As shown in Table 1, WeLIS is the first multi-view model that can do both object-centric inference and novel scene generation.

Figure 8. Component-by-component random generation results of WeLIS and the baseline. Each row corresponds to a different query viewpoint. The first column shows generated images, and the next seven columns show each slot (${\mathbf{x}}_k,k\in \{1,\dots ,K\}$), while last column shows generated segmentation masks.

Figure 9. Random generation results from WeLIS, MulMON(baseline) and ablated models. The top row shows results from WeLIS and MulMON, and the bottom row shows WeLIS without Structured Prior and without normalizing flow for ablation study. We generated these images by randomly sampling eight ${\mathbf{z}}_g$ for WeLIS, and by sampling each slot independently for MulMON. Note that the same set of ${\mathbf{z}}_g$ is used in the full model(top left) and in the ablation of the Structured Prior(bottom left).

4.2.1. Ablation studies (Novel scene generation)

The bottom row of Figure 9 shows ablation studies of Structured Prior and NF. Firstly, we look into Structured Prior. In this experiment, the results from the full model (top left) and ablation study of Structured Prior (bottom left) are generated from the same ${\mathbf{z}}_g$. Generated images without Structured Prior are severely blurred, however, we can see that the spatial arrangement of objects itself is similar to the full model. This indicates that Structured Prior only contributes to object-level generation quality, and spatial consistency is secured by the global latent variable.

Secondly, the bottom right images in Figure 9 show ablation results of NF. As you can see, the quality is almost the same with the full model even without NF, which indicates that NF is not essential for generation itself. However, as we mentioned in Section 4.1, NF is important for stable inference and training, thus it is also useful for novel scene generation in practice.

4.2.2. Quantitative evaluations

We also quantitatively evaluated novel scene generation by Fréchet Inception Distance(FID) score in Table 2. WeLIS outperforms MulMON in every dataset, and as shown in the above section, WeLIS without NF achieved similar quality to the full model, as long as the training succeeded. Scores without NF in CLEVR AUG and GQN Jaco are degraded because the training fails more frequently as datasets become difficult.

Table 2. Evaluation of FID score on three models: WeLIS(Ours), WeLIS without Normalizing Flow and MulMON.

Datasets	WeLIS (Ours)	WeLIS w/o NF	MulMON
CLEVR MV	125.8 $\pm 1.7$	$\mathbf{117.2}$ $\pm 5.9$	157.9 $\pm 8.4$
CLEVR AUG	$\mathbf{83.3}$ $\pm 1.7$	134.2 $\pm 25.8$${}^{\mathrm{a}}$	168.6 $\pm 5.0$
GQN Jaco	$\mathbf{251.7}$ $\pm 2.1$	262.1 $\pm 11.0$	273.7 $\pm 10.4$

Note: The numbers after ± stands for standard deviations. Lower is better.

${}^{\mathrm{a}}$If the case where training did not converged is excluded, the score becomes $90.0\pm 1.3$.

Poor generation quality of MulMON comes from mainly two reasons: insufficient modeling of spatial arrangement and difficulty in sampling actual posterior. In WeLIS, the former one is dealt with by introducing global latent ${\mathbf{z}}_g$ which represents spatial information of a whole scene, and the latter one is dealt with by a Structured Prior which estimates complex posterior inferred and updated by IAI. Effectiveness of introducing ${\mathbf{z}}_g$ is evaluated in Section 4.3 and that of Structured Prior is already investigated in Figure 9.

4.3. Applying perturbation to the global latent variable

To look into the representation obtained by the global latent variable ${\mathbf{z}}_g$, we show images generated from four different ${\mathbf{z}}_g$ (A-D) and their four neighborhoods ${\mathbf{z}}_g+\mathit{ϵ}$ in Figure 10. Here, $\mathit{ϵ}$ is a small perturbation randomly sampled from a standard normal distribution. The leftmost images in the each group is the one from each ${\mathbf{z}}_g$, and the other four images are the ones generated with different perturbations. Thus, each group (A-D) represents a ‘cluster’ around ${\mathbf{z}}_g$.

Figure 10. Random generation results from four different ${\mathbf{z}}_g$ (A–D) and four neighborhoods ${\mathbf{z}}_g+\mathit{ϵ}$, where $\mathit{ϵ}$ is Gaussian noise as a perturbation. Each row corresponds to a different ${\mathbf{z}}_g$. The left most columns of each group show the original ${\mathbf{z}}_g$, and other columns are added small perturbations $\mathit{ϵ}$. Therefore, each row represents a ‘cluster’ of each ${\mathbf{z}}_g$. We can see that images in the same row share a similar spatial structure. The schematic of the latent space on the right side is just for illustration purposes.

We can see that generated scenes from the same ${\mathbf{z}}_g$ have similar spatial structure, while their objects vary. This indicates that the latent space of ${\mathbf{z}}_g$ obtains the spatial configuration of a scene as we expected.

Moreover, we show latent traversal results in Figure 11. Each row shows different interpolation results from randomly sampled two ${\mathbf{z}}_g$: sample A and sample B. We can see that spatial arrangement changes continuously, but the property of individual components changes quickly. This result also indicates that the global latent variable holds information about the spatial arrangement but individual objects.

Figure 11. Traversal of global latent space.

4.4. Downstream tasks

In the above sections, we evaluated our method mainly by reconstruction and segmentation. Though they are inevitably important metrics, they can not evaluate how good the representation obtained by latent variables are. One of the important things for deep generative models is whether the obtained representation is useful for various unknown tasks, i.e. downstream tasks. Here, we evaluate the representation by counting the number of objects in an observed scene. The number of objects is predicted from inferred latent variables using a linear probe. It is a fully connected layer which takes K (objects) latent variables as input, and output one-hot vector which represents the number of objects in the scene. With this task, we can evaluate how much information the representations hold.

Table 3 shows the accuracy from each model. Since WeLIS has two types of latent variables, we tried three combinations: ${\mathbf{z}}_g$, $\mathbf{z}$, and concatenation of ${\mathbf{z}}_g$ and $\mathbf{z}$. In CLEVR MV dataset, ${\mathbf{z}}_g$ contributes to the accuracy the most, and In CLEVR AUG dataset, $\mathbf{z}$ performs well especially in combination with ${\mathbf{z}}_g$. This result also shows that ${\mathbf{z}}_g$ effectively captures global information of the scene comparing to the combination of single object representations $\mathbf{z}$.

Table 3. Accuracy of a downstream task.

Models	Datasets	Observations	Accuracy
MulMON	CLEVR MV	3	64.6
WeLIS(${\mathbf{z}}^g$)			73.0
WeLIS($\mathbf{z}$)			63.5
WeLIS(${\mathbf{z}}^g$ + $\mathbf{z}$)			71.5
MulMON	CLEVR MV	4	75.0
WeLIS(${\mathbf{z}}^g$)			76.5
WeLIS($\mathbf{z}$)			53.5
WeLIS(${\mathbf{z}}^g$ + $\mathbf{z}$)			76.0
MulMON	CLEVR MV	5	77.5
WeLIS(${\mathbf{z}}^g$)			76.0
WeLIS($\mathbf{z}$)			58.0
WeLIS(${\mathbf{z}}^g$ + $\mathbf{z}$)			76.0
MulMON	CLEVR AUG	3	65.4
WeLIS(${\mathbf{z}}^g$)			63.9
WeLIS($\mathbf{z}$)			74.5
WeLIS(${\mathbf{z}}^g$ + $\mathbf{z}$)			77.5
MulMON	CLEVR AUG	4	70.7
WeLIS(${\mathbf{z}}^g$)			68.8
WeLIS($\mathbf{z}$)			78.7
WeLIS(${\mathbf{z}}^g$ + $\mathbf{z}$)			78.3
MulMON	CLEVR AUG	5	71.8
WeLIS(${\mathbf{z}}^g$)			69.3
WeLIS($\mathbf{z}$)			79.3
WeLIS(${\mathbf{z}}^g$ + $\mathbf{z}$)			79.6

Note: ‘Observations’ stands for the number of input observation views, and ‘Accuracy’ stands for the prediction accuracy of the number of objects in a scene.

5. Conclusion

We introduced a new multi-view object-centric model: WeLIS, which can perform multi-view object-centric inference and sampling in 3D scenes. We introduced several components to WeLIS, in order to adapt a global latent variable and to enable inference and training with that model. The global latent variable improved inference quality especially when the observation is limited (Section 4.1) and also enabled the model to generate physically plausible novel scenes (Section 4.2). We also conducted ablation studies (Section 4.1.1 and 4.2.1) and looked into the representation obtained by global latent space (Section 4.3).

One of the future directions of this research is to sophisticate inference methods such as normalizing flow and IAI. There is room for improvement with an optimal choice of normalizing flow. In addition, we used simplified IAI without image-sized input which can reduce computational cost, however, there is a possibility that this sacrificed the ability to distinguish similar, ambiguous objects such as rounded cubes and spheres in CLEVR MV dataset.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Yuya Kobayashi

Yuya Kobayashi is a researcher in SONY Computer Laboratory. He received Ph.D in Engineering from The University of Tokyo. He has been engaged in Deep Generative Models, Compuitational Neuroscience, Non-invasive Brain Recordings.

Masahiro Suzuki

Masahiro Suzuki is a assistant professor in The University of Tokyo. He received Ph.D in Engineering from the University of Tokyo in 2018. His research topic is Artificial Intelligence and Deep Learning.

Yutaka Matsuo

Yutaka Matsuo is a Professor in the University of Tokyo. He was a director of The Japanese Society for Artificial Intelligence in 2020–2022. He received Ph.D in Engineering from The University of Tokyo in 2002. His specialization is Artificial Intelligence, Deep Learning, Web Engineering.

Notes

1 According to the supplemental material, MulMON uses CLEVR AUG in $128\times 128$ resolution, but we used all the datasets in $64\times 64$ for consistent comparison.