I-Design: Personalized LLM Interior Designer (2024)

1 Introduction

Our lives are intimately connected to the spaces we inhabit. Whether renting or buying, these spaces become the backdrop for our memories, hobbies, and time with loved ones. However, finding or creating the perfect space to match our lifestyle, aspirations, and needs is not always straightforward, and professional assistance can be considered a luxury. Even when seeking help from experts, the gap between what inhabitants truly desire and their ability to convey it in the language of professionals often leads to unsatisfactory results. This discrepancy can leave people with ill-fitting living spaces, impacting their physical and mental well-being. Designing interior spaces better suited for individual needs should be accessible to everyone.

Toward this, we tackle the challenging task of 3D Indoor Scene Synthesis (3DISS). Given a user’s unstructured textual description of preferences, we aim to deliver 3D design solutions that align with the latter.

Specialized generative models[21, 11, 49] and data-driven approaches[56, 48, 66] have demonstrated a remarkable ability to produce diverse and realistic interior layouts.Yet, their performance is governed by the closed-set and limited datasets used to train them, which inevitably is a biased and incomplete sample of the world. In addition, such methods that accept textual input from the user can only utilize structured text with predefined grammar and rules.Consequently, it is challenging to guide these models toward producing practical designs for real-world, unseen interiors that align with user preferences.

The 3DISS task requires reasoning abilities beyond any specialized data-driven model. This includes understanding design principles and concepts like object selection, styles, and spatial arrangement. However, a potential solution must succeed at several fundamental steps to become successful in 3DISS. Firstly, the underlying generator must interpret the abstract input proficiently, i.e. identifying the objects to incorporate into the scene to meet user preferences. Simultaneously, it should know about the typical items associated with specific room types, incorporating high-frequency and common-sense objects even if the user does not explicitly mention them. In addition, it requires an awareness of plausible spatial relationships among objects originating from an unrestricted vocabulary. Addressing these challenges requires a system with a comprehensive understanding of diverse human interior design preferences and an extensive database of 3D objects, encompassing their functionality, dimensions, and stylistic attributes.

Given recent technological advancements, the most viable option to overcome these obstacles is large language models (LLMs) [64, 1].LLMs are trained on internet-scale data and encode a “world model” that can be probed and interacted with using natural language.Still, they cannot use the language as humans do to solve complex tasks without resorting to specialized techniques, such as Chain-of-Thought[55].Technical aspects (e.g. limited context window length, number of parameters) and artifacts (e.g. hallucinations) additionally hamper practical applications.Attempts to produce structured output with LLMs without incorporating hard constraints, communication interfaces, and cross-checks are exciting at first glance but leave much room for improvement.LayoutGPT[16], a recent work that utilizes LLMs for 3DISS, directly predicts the absolute positions of objects in the scene. Although this technique may be effective when dealing with small-scale arrangements of a few objects, it proves inadequate in generating realistic scenes containing dozens of objects interlinked in intricate ways. Moreover, the single-step scene generation in LayoutGPT cannot provide interpretability regarding the resulting objects and their arrangement.

Recent research [63, 40] has also demonstrated that when multiple LLM agents with diverse responsibilities communicate with each other, they can collectively tackle complex tasks with which a single LLM instance may struggle. Hence, we utilize LLM agents to address the challenges posed by 3DISS, leading to the creation of scenes that are spatially more plausible and diverse. Furthermore, drawing inspiration from other works [2, 7, 66, 51], we employ scene graphs as scene representations since they offer a high-level abstraction by focusing on the objects and their relationships, which can be creatively developed with LLMs, refined through rule-based feedback, and visualized. Additionally, employing the scene graph representation as the interface between LLM agents and algorithms enables interpretable object arrangements.

To this end, we present I-Design, a personalized interior designer for 3D Indoor Scene Synthesis. Starting from the user specification of the design preferences in plain text, I-Design queries LLM agents to come up with room items, their properties, and relative relationships in the form of a scene graph. It solves for absolute object placement in the scene graph using the proposed backtracking algorithm, retrieves 3D assets according to the functional and stylistic specifications, and composes the final result in 3D.To evaluate the proposed design pipeline, and following in the footsteps of[60], we propose a novel evaluation protocol based on a vision-language model (VLM).Extensive quantitative and qualitative experiments show that I-Design outperforms existing methods in delivering high-quality 3D design solutions that align with abstract concepts in the user input.

2 Related Work

3 Method

Our method for proposing design solutions from user input in plain text (I-Design) is summarized in Fig.1.Initially, I-Design examines unstructured textual user input and transforms it into a viable design proposal, represented as a scene graph, through querying LLM agents (Sec.3.2).We then introduce the scene graph layout module for producing object placement proposals represented by a 2D room floor plan (Sec.3.3).Finally, we retrieve 3D assets from existing databases (Sec.3.4) and assemble the final design within a 3D environment, producing functional and stylistic design solutions that reflect user input (Sec.3.5).

3.2 LLM Multi-agents

Generating a 3D indoor scene based on unstructured user input is a challenging task that demands detailed planning.Two essential steps are shared among common methods: selecting the objects to populate the scene based on user input and arranging them in a meaningful and coherent configuration.We employ a multi-agent approach to tackle this complexity effectively, allocating the tasks involved in these steps across multiple LLM agents.As shown in Fig.2, interpreting and transforming user input into a functional, personalized scene graph involves five distinct agents: Interior Designer, Interior Architect, Engineer, Layout Corrector, and Layout Refiner.The Interior Designer and Architect propose a relative scene graph from the user input. We assume that the agents have appropriately considered contextually bonded objects while constructing the relative scene graph, justifying spatial separation and disconnection within the graph.However, the relative scene graph lacks guaranteed spatial consistency, allowing for connections between objects that may defy real 3D scene constraints.Therefore, our pipeline uses a Layout Corrector agent to rectify invalid and implausible connections, while the scene graph contains only relative relations.Finally, to increase the pipeline’s robustness when dealing with multiple objects assigned to the same parent within the scene graph, the Layout Refiner agent proposes relations between such objects, simplifying the subsequent layout stage. Next, we examine each of the agents in more detail.

Interior Designer:The agent receives inputs comprising the free-form user input $T_{user}$, the room dimensions $(l_{room},w_{room},h_{room})$, and the desired number of objects $n$, and proposes a selection of objects $\{\mathbf{o}_{i}\}$ tailored to the user’s preferences, while also ensuring that their functionality matches the room type.Specifically, for each object, the interior designer suggests the following properties$\mathbf{o}_{i}=\{\alpha_{i},m_{i},s_{i},r_{i},p_{i}\}$ including its name $\alpha_{i}$, material $m_{i}$, 3D size $s_{i}$, orientation $r_{i}$, and position $p_{i}$.Note that the agent may propose several identical instances.For example, there might be four instances of the “chair” type positioned around a table (see Fig.2).

Interior Architect:Its primary responsibility is to establish object-to-object connections, as well as object-to-room layout relations.In other words, its role is to establish the edge connections $e_{ij}$ between the object nodes.The agent is not constrained on the number of edges each object can have, allowing for diverse configurations where an object may have edges to room layout elements but none to other objects, and vice versa.Additionally, the Interior Architect creates information on the rotation $r_{i}$ of each object around the vertical axis.

Engineer:This agent transforms the relative scene graph into a JSON object structured according to a specified schema. Each entry in the JSON file encompasses the details for $\mathbf{o}_{i}=\{\alpha_{i},m_{i},s_{i},r_{i},p_{i}\}$ for the corresponding object. Moreover, the agent employs a JSON schema validator that assesses the validity of the generated file based on the provided schema. In cases of non-compliance, a modification from the Engineer is requireduntil the output aligns with the specified schema, ensuring a valid JSON representation.

Layout Corrector:The responsibility of this agent is to fix invalid connections in the graph, which involves removing spatial implausible edges and eliminating ambiguities between nodes.We preemptively examine three types of spatial implausibilities: (1) Checking for objects positioned beyond room boundaries (as exemplified in Fig.2), (2) Assessing spatially impossible object-to-object connections, and (3) Ensuring size compatibility between parent and child objects.The agent determines the relocation of objects experiencing spatial conflicts, either suggesting alternative edges for these objects or removing them entirely from the scene.

1. 1.

   To check room boundaries, we examine children nodes of objects allocated along walls or in room corners.For example, if object A is positioned behind object B, which is placed alongside a wall, this would result in object A being positioned out of bounds.

2. 2.

   Assessing the plausibility of object-to-object connections involves verifying whether adjacency relations between objects could be violated via conflicting edges in the scene graph. This requires checking if any object has been positioned between two adjacent objects in the scene graph.

3. 3.

   We operate under the assumption that, in a valid scene graph, the bounding box of the parent object is sufficiently spacious to accommodate its children.For example, if a table has two chairs positioned on its left side, it should be wide enough to accommodate them; likewise, a wall with multiple objects attached on it should have sufficient size to accommodate all of its children objects.The last check ensures this compatibility.

See Also

I-Design: Personalized LLM Interior Designer[PDF] I-Design: Personalized LLM Interior Designer | Semantic ScholarIf You Have Zero Interior Design Skills, Check Out These 27 Simple Pieces Of Decor

Layout Refiner:The agent aims to eliminate ambiguities between children nodes that share the same edge from the parent. A notable example of this phenomenon is observed with ornaments placed on a desk. As a desk typically accommodates several objects on its surface, a scene graph designating the desk as the parent with the preposition on as the edge fails to convey the relative orientation of the objects on the desk to each other. To eliminate this ambiguity, we establish edges between children nodes, ensuring a distinct ordering among them.The agent also verifies that the scene graph maintains its acyclic property. If the graph contains cycles, we must remove edges contributing to these cycles to preserve a distinct hierarchy within the scene graph.

3.3 Scene Graph Layout

3.3.1 Computing Cluster Dimensions

In the final postprocessing phase, we compute cluster dimensions for each object to facilitate the positioning in the subsequent backtracking algorithm, thereby strategically constraining the search space. Leveraging that $\mathcal{G}$ is a Directed Acyclic Graph, we perform a topological sort on the nodes to establish a hierarchical ordering. This approach enables us to determine the clearance – the distance an object must spare in each direction to ensure the successful placement of its children within the scene.

Taking such precaution becomes crucial in scenes where the search space for each object is extensive, yet the solution space is relatively constrained. Without this precaution, the randomized nature of the backtracking algorithm may lead to extended processing times, particularly when dealing with many object relationships in the scene.

3.3.2 The Backtracking Algorithm

The backtracking algorithm serves to convert the relative representation of the scene graph $\mathcal{G}$ into object positions $\mathcal{P}$. The room is initially populated with the root nodes of the scene graph $\mathcal{G}$, representing the fundamental elements of the room layout, such as walls and ceiling. The positions of these root nodes remain fixed while other objects are arranged around them.Conceptually, the algorithm represents the plausible position of each object as a bounding box $\mathcal{B}_{i}$ and sampling a point $p_{i}$ for each $\mathbf{o}_{i}$ from $\mathcal{B}_{i}$.

The objects are placed after being topologically sorted, prioritizing nodes higher in the scene graph hierarchy for placement first. To avoid placement issues later in the algorithm, each object $\mathbf{o}_{i}$ is positioned alongside its children according to $\mathcal{G}_{i}$.The plausible positions bounding box for the objects is defined through predetermined placement functions $f(\mathbf{o}_{i},\mathbf{o}_{j})$ $\forall\mathbf{o}_{j},\mathbf{o}_{i}\in\mathcal{G}_{j}$.

The nodes are organized into depth groups, where a node’s depth $d=|\mathbf{o}_{i}\rightarrow\textbf{o}^{(r)}|$ represents its distance to the closest $\textbf{o}^{(r)}$. If an object $\textbf{o}_{i}$ with depth $d$ cannot be placed into the scene due to either an empty bounding box $\mathcal{B}_{i}=\varnothing$ or if $p_{i}$ consistently results in collisions with other objects, we remove $p_{j}$ for all objects with depth $\geq d$. Subsequently, we re-sample positions for objects with depth $d-1$. Once all objects with depth $d$ are positioned successfully, we increment the depth counter of the algorithm. An example process of scene graph processing is shown in Fig.3.

4 Experiments

4.2 Quantitative Evaluation

We compare I-Design with baseline approaches to quantitatively assess the design quality of the interior rooms generated by our proposed pipeline.

4.2.1 Metrics

We conduct quantitative evaluations using the following metrics:

Average Number of Proposed Objects (NObj): The diversity of the generated interior rooms can be partially assessed by considering the number of furniture pieces proposed and placed within the scene. Here, we compute the average number of proposed objects separately for bedrooms and living rooms.

Out-of-Boundary Rates (OOB): The feasibility of a generated layout can be quantified by examining the frequency of objects extending beyond room boundaries, as outlined in[16]. In this context, if any bounding boxes are outside the designated room boundaries within a scene, that scene is deemed invalid. OOB values are determined by calculating the ratio of invalid scenes to the total number of generated scenes.

Bounding Box Loss (BBL):The viability of a generated layout can also be measured by evaluating the degree of overlap between proposed furniture bounding boxes. This is calculated as the average volume of bounding box intersections across generated scenes.

GPT-4V ratings:As highlighted in[60], GPT-4V[1] has been shown to serve as a human-aligned evaluator for 3D content. Inspired by this concept, we employ GPT-4V as an evaluator to assess the visual quality of synthesized rooms based on their renderings. The rating criteria encompass aspects of functionality and activity-based alignment (Func.), layout and furniture (Layout.), color scheme and material choice (Scheme.), as well as overall aesthetic and atmosphere (Atmos.). The integer ratings range from 0, the minimum, to 10, the maximum. The rating process is conducted unary, where each scene is evaluated individually. Two renderings from different viewpoints are horizontally concatenated and passed along with the user input to the evaluator for rating to quantify their alignment with the generated results.Each evaluation is performed three times in parallel by setting the corresponding parameter n using the GPT-4V API.The resulting grades are averaged in every category, and their standard deviations are also reported to account for the grade spread and allow for pairwise grade comparisons.

4.2.2 Baseline

We treat LayoutGPT[16] as a baseline for comparison. LayoutGPT[16] utilizes LLM models, including GPT-4[1], to generate room layouts in a CSS-like format through few-shot learning with examples sourced from the 3D-FRONT[18] dataset.Here, for a fair comparison, we have extended the original LayoutGPT pipeline[16] by enabling it to retrieve and place objects from Objaverse[12].

4.2.3 Experimental Settings

As LayoutGPT[16] only supports “Bedroom” and “Living room” room types, our comparison is limited to these types and variations of room dimensions.We ensure that the prompts given to our pipeline contain only this information for a fair comparison.Besides, we let LayoutGPT[16] retrieve objects from Objaverse[12] and use the same settings for rendering.We generate ten living room and ten bedroom scenes with the same dimensions for each method. The quantitative comparison results are provided in Tab.1.

Method	Room Type	NObj$\uparrow$	OOB$\downarrow$	BBL$\downarrow$	GPT-4V Criteria$\uparrow$
					Func.$\uparrow$	Layout.$\uparrow$	Scheme.$\uparrow$	Atmos.$\uparrow$	Avg.$\uparrow$
LayoutGPT[16]	Bedroom	5.5	51.06	14.09	4.8$\,\pm$0.4	4.8$\,\pm$0.8	4.6$\,\pm$0.8	4.9$\,\pm$0.4	4.8
	LivingRoom	6.9	64.15	1.06	4.8$\,\pm$1.3	4.8$\,\pm$0.8	4.8$\,\pm$1.4	4.6$\,\pm$0.8	4.8
	Avg.	6.2	57.6	7.58	4.8	4.8	4.7	4.8	4.8
Ours (I-Design)	Bedroom	12.7	0.0	0.34	5.2$\,\pm$0.4	5.5$\,\pm$0.2	5.6$\,\pm$0.7	5.5$\,\pm$0.2	5.5
	LivingRoom	23.6	0.0	0.31	5.8$\,\pm$2.6	5.6$\,\pm$2.1	5.9$\,\pm$1.1	5.7$\,\pm$1.3	5.8
	Avg.	18.2	0.0	0.33	5.5	5.6	5.8	5.6	5.7

Besides comparing our approach with the baseline, we aim to demonstrate our ability to appropriately process prompts covering diverse aspects. For this purpose, we generate ten prompts for each potential factor influencing an individual’s decisions regarding interior designs, encompassing functionality, layout, color scheme, and overall atmosphere. We compare the rooms synthesized by tailored prompts focusing on specific aspectsTab.2.Refer to the Appendix for a complete list of prompts used for both tables.

4.2.4 Results

Prompt type	GPT-4V Criteria
	Func.	Layout.	Scheme.	Atmos.	Avg.
Atmospheric	4.9$\,\pm$0.7	4.9$\,\pm$0.6	4.3$\,\pm$0.6	4.4$\,\pm$0.5	4.9
Scheme	4.6$\,\pm$0.5	4.8$\,\pm$0.6	5.0$\,\pm$0.5	4.8$\,\pm$0.5	5.0
Layout	6.2$\,\pm$0.6	6.0$\,\pm$0.7	5.2$\,\pm$0.7	5.6$\,\pm$0.6	5.8
Functional	7.0$\,\pm$0.7	6.5$\,\pm$0.7	5.8$\,\pm$0.8	6.2$\,\pm$0.7	6.4

Compared to LayoutGPT[16], as illustrated in Tab.1, our approach stands out in proposing larger furniture sets and more physically plausible layouts, exhibiting zero OOB and minimal BBL values. For evaluations with GPT-4V, our method outperforms LayoutGPT[16] across all metrics, indicating better visual quality of the synthesized rooms and alignment with user preferences.

As outlined in Tab.2,the performance of I-Design varies depending on the type of prompt input.Our method yields the best results in all four grading aspects when incorporating functional descriptions.Functionality prompts provide cues regarding the room type and potential activities occurring within the space.The usage of atmosphere and scheme information somewhat diminishes the quality of synthesis.We attribute this to our pipeline’s lack of consideration for the materials and textures of the walls, ceiling, and floor and the absence of an object re-texturing step.This can significantly impact the room’s overall ambiance and can be considered a promising future research direction.

4.3 User Study

Tab.3 contains the results of a user study aimed at reinforcing ourGPT-4V evaluation.Out of 20 scenes created by our method and another 20 by LayoutGPT[16], we randomly sampled 5 scenes from the competitor pool for each sample and thus composed 100 pairs.Each pair depicts either a bedroom or a living room, each comprising a render from our method and one from LayoutGPT.Subjects were instructed[47] to vote on the most realistic room in each pair. The order of the methods is randomized during the display to prevent subject bias. Each participant was assigned 20 pairs to evaluate, and two control questions with predefined correct answers were included for verification purposes, as shown in Fig.LABEL:fig:userstudy. We collected 1254 answers for bedroom pairs and 660 for living room pairs. The gathered votes were converted into Bradley-Terry preference scores[5] and probabilities.The results affirm our findings with the GPT-4V evaluator, with our method outperforming LayoutGPT for both room types.

Method	Room Type	Bradley-Terry score$\uparrow$	Prob.$\uparrow$
LayoutGPT	Bedroom	0.12	0.42
	LivingRoom	0.17	0.38
Ours	Bedroom	0.40	0.58
	LivingRoom	0.69	0.62

5 Conclusion

Computing-assisted technologies have been widely used to simplify the interior design process for individuals without expertise. Current methods for generating interior rooms, whether through generative models or prior learning, often suffer from limitations such as inconsistencies between views and restrictions on furniture sets or room types, which are confined to the dataset’s provided domain. We aim to eliminate these constraints and provide users with a user-friendly interface that streamlines the construction phase of the interior room. To realize this vision, we presented I-Design.

By leveraging LLM multi-agent architecture, our framework interprets user preferences expressed through text, transforming unstructured text into a structured representation. A furniture set that aligns with the user’s requirements is inferred. Subsequently, spatial relationships among furniture are established, with common sense knowledge integrated into the agents’ decision-making process. The resulting scene graph provides a visual representation of the proposed design. Crucially, our framework ensures interpretability through a transparent breakdown of each step involved, facilitating an accessible design process.

Our framework still encounters limitations, notably termination problems where the pipeline fails to find a solution for object placements. This issue arises when handling many objects in a relatively small scene. Spatial conflicts may persist despite efforts in postprocessing to rectify them, or there may not be enough space for furniture placements.Asset retrieval also poses challenges, as the quality of retrieved assets may vary, and they can even come with invalid default orientations. Additionally, discrepancies in object sizes compared to proposed dimensions may occur, leading to artifacts when resizing them.

A promising future work direction is further exploring object placement with an automated, learning-based approach, moving away from the current backtracking algorithm that relies on trial and error. Another direction would be to adopt a generative approach to replace or complement the object retrieval process, thereby improving the diversity and flexibility of the overall pipeline.

References

• [1]Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., etal.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
• [2]Armeni, I., He, Z.Y., Gwak, J., Zamir, A.R., Fischer, M., Malik, J., Savarese, S.: 3d scene graph: A structure for unified semantics, 3d space, and camera. In: IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5664–5673 (2019)
• [3]Bautista, M.A., Guo, P., Abnar, S., Talbott, W., Toshev, A., Chen, Z., Dinh, L., Zhai, S., Goh, H., Ulbricht, D., etal.: GAUDI: A neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems (NeurIPS) 35, 25102–25116 (2022)
• [4]Blender Online Community: Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam (2018), http://www.blender.org
• [5]Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika 39(3/4), 324–345 (1952)
• [6]Cai, S., Chan, E.R., Peng, S., Shahbazi, M., Obukhov, A., VanGool, L., Wetzstein, G.: Diffdreamer: Towards consistent unsupervised single-view scene extrapolation with conditional diffusion models. In: ICCV (2023)
• [7]Chang, A., Monroe, W., Savva, M., Potts, C., Manning, C.D.: Text to 3d scene generation with rich lexical grounding. arXiv preprint arXiv:1505.06289 (2015)
• [8]Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., etal.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
• [9]Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B.F., Fu, J., Shi, Y.: AutoAgents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288 (2023)
• [10]Ching, F.D., Binggeli, C.: Interior design illustrated. John Wiley & Sons (2018)
• [11]Chung, J., Lee, S., Nam, H., Lee, J., Lee, K.M.: LucidDreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384 (2023)
• [12]Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13142–13153 (2023)
• [13]Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Ehsani, K., Salvador, J., Han, W., Kolve, E., Kembhavi, A., Mottaghi, R.: ProcTHOR: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems (NeurIPS) 35, 5982–5994 (2022)
• [14]Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems (NeurIPS) 34, 8780–8794 (2021)
• [15]Fang, C., Hu, X., Luo, K., Tan, P.: Ctrl-Room: Controllable text-to-3d room meshes generation with layout constraints. arXiv preprint arXiv:2310.03602 (2023)
• [16]Feng, W., Zhu, W., Fu, T.j., Jampani, V., Akula, A., He, X., Basu, S., Wang, X.E., Wang, W.Y.: Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305.15393 (2023)
• [17]Fridman, R., Abecasis, A., Kasten, Y., Dekel, T.: SceneScape: Text-driven consistent scene generation. Advances in Neural Information Processing Systems (NeurIPS) 36 (2024)
• [18]Fu, H., Cai, B., Gao, L., Zhang, L.X., Wang, J., Li, C., Zeng, Q., Sun, C., Jia, R., Zhao, B., etal.: 3D-FRONT: 3d furnished rooms with layouts and semantics. In: IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10933–10942 (2021)
• [19]Fu, H., Jia, R., Gao, L., Gong, M., Zhao, B., Maybank, S., Tao, D.: 3D-FUTURE: 3d furniture shape with texture. International Journal of Computer Vision 129, 3313–3337 (2021)
• [20]Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS) 33, 6840–6851 (2020)
• [21]Höllein, L., Cao, A., Owens, A., Johnson, J., Nießner, M.: Text2Room: Extracting textured 3d meshes from 2d text-to-image models. arXiv preprint arXiv:2303.11989 (2023)
• [22]Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., Tan, H.: LRM: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 (2023)
• [23]Huang, I., Krishna, V., Atekha, O., Guibas, L.: Aladdin: Zero-shot hallucination of stylized 3d assets from abstract scene descriptions. arXiv preprint arXiv:2306.06212 (2023)
• [24]Jin, Q., Yang, Y., Chen, Q., Lu, Z.: GeneGPT: Augmenting large language models with domain tools for improved access to biomedical information. ArXiv preprint arXiv:2304.09667 (2023)
• [25]Kar, A., Prakash, A., Liu, M.Y., Cameracci, E., Yuan, J., Rusiniak, M., Acuna, D., Torralba, A., Fidler, S.: Meta-Sim: Learning to generate synthetic datasets. In: IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4551–4560 (2019)
• [26]Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Repurposing diffusion-based image generators for monocular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
• [27]Lee, K.T., Im, J.B., Park, S.J., Kim, J.H.: Conceptual framework to support personalized indoor space design decision-making: A systematic literature review. Buildings 12(6), 716 (2022)
• [28]Li, W., Saeedi, S., McCormac, J., Clark, R., Tzoumanikas, D., Ye, Q., Huang, Y., Tang, R., Leutenegger, S.: InteriorNet: Mega-scale multi-sensor photo-realistic indoor scenes dataset. In: British Machine Vision Conference (BMVC) (2018)
• [29]Lin, B.Y., Fu, Y., Yang, K., Brahman, F., Huang, S., Bhagavatula, C., Ammanabrolu, P., Choi, Y., Ren, X.: Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. arXiv preprint arXiv:2305.17390 (2023)
• [30]Lin, C., Mu, Y.: Instructscene: Instruction-driven 3d indoor scene synthesis with semantic graph prior. arXiv preprint arXiv:2402.04717 (2024)
• [31]Liu, J., Xiong, W., Jones, I., Nie, Y., Gupta, A., Oğuz, B.: Clip-layout: Style-consistent indoor scene synthesis with semantic furniture embedding (2023)
• [32]Liu, M., Shi, R., Kuang, K., Zhu, Y., Li, X., Han, S., Cai, H., Porikli, F., Su, H.: OpenShape: Scaling up 3d shape representation towards open-world understanding. Advances in Neural Information Processing Systems (NeurIPS) 36 (2024)
• [33]Liu, M., Xu, C., Jin, H., Chen, L., VarmaT, M., Xu, Z., Su, H.: One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems (NeurIPS) 36 (2024)
• [34]Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., etal.: Wonder3D: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 (2023)
• [35]Merrell, P., Schkufza, E., Li, Z., Agrawala, M., Koltun, V.: Interactive furniture layout using interior design guidelines. ACM transactions on graphics (TOG) 30(4), 1–10 (2011)
• [36]Paschalidou, D., Kar, A., Shugrina, M., Kreis, K., Geiger, A., Fidler, S.: ATISS: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems (NeurIPS) 34, 12013–12026 (2021)
• [37]Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
• [38]Purkait, P., Zach, C., Reid, I.: SG-VAE: Scene grammar variational autoencoder to generate new indoor scenes. In: European Conference on Computer Vision (ECCV). pp. 155–171. Springer (2020)
• [39]Qi, S., Zhu, Y., Huang, S., Jiang, C., Zhu, S.C.: Human-centric indoor scene synthesis using stochastic grammar. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5899–5908 (2018)
• [40]Qian, C., Cong, X., Liu, W., Yang, C., Chen, W., Su, Y., Dang, Y., Li, J., Xu, J., Li, D., Liu, Z., Sun, M.: Communicative agents for software development. arXiv preprint arXiv:2307.07924 (2023)
• [41]Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021)
• [42]Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10684–10695 (2022)
• [43]Shen, L., Li, X., Sun, H., Peng, J., Xian, K., Cao, Z., Lin, G.: Make-it-4d: Synthesizing a consistent long-term dynamic scene video from a single image. In: ACM International Conference on Multimedia. pp. 8167–8175 (2023)
• [44]Shen, Y., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y.: HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. arXiv preprint arXiv:2303.17580 (2023)
• [45]Song, C.H., Wu, J., Washington, C., Sadler, B.M., Chao, W.L., Su, Y.: LLM-Planner: Few-shot grounded planning for embodied agents with large language models. In: IEEE/CVF International Conference on Computer Vision (CVPR). pp. 2998–3009 (2023)
• [46]Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene completion from a single depth image. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1746–1754 (2017)
• [47]Subjectify.us: Crowd-sourced subjective quality evaluation platform (Accessed: Jan 2024)
• [48]Tang, J., Nie, Y., Markhasin, L., Dai, A., Thies, J., Nießner, M.: Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303.14207 (2023)
• [49]Tang, S., Zhang, F., Chen, J., Wang, P., Furukawa, Y.: MVDiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv (2023)
• [50]Tseng, H.Y., Li, Q., Kim, C., Alsisan, S., Huang, J.B., Kopf, J.: Consistent view synthesis with pose-guided diffusion models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16773–16783 (2023)
• [51]Wang, K., Lin, Y.A., Weissmann, B., Savva, M., Chang, A.X., Ritchie, D.: PlanIT: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions on Graphics (TOG) 38(4), 1–15 (2019)
• [52]Wang, T., Kanakis, M., Schindler, K., VanGool, L., Obukhov, A.: Breathing new life into 3d assets with generative repainting. In: Proceedings of the British Machine Vision Conference (BMVC). BMVA Press (2023)
• [53]Wang, X., Yeshwanth, C., Nießner, M.: Sceneformer: Indoor scene generation with transformers. In: 2021 International Conference on 3D Vision (3DV). pp. 106–115. IEEE (2021)
• [54]Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., etal.: Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997 (2023)
• [55]Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., etal.: Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837 (2022)
• [56]Wei, Q.A., Ding, S., Park, J.J., Sajnani, R., Poulenard, A., Sridhar, S., Guibas, L.: LEGO-Net: Learning regular rearrangements of objects in rooms. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 19037–19047 (2023)
• [57]Weiss, T., Litteneker, A., Duncan, N., Nakada, M., Jiang, C., Yu, L.F., Terzopoulos, D.: Fast and scalable position-based layout synthesis. IEEE Transactions on Visualization and Computer Graphics 25(12), 3231–3243 (2018)
• [58]Wen, Z., Liu, Z., Sridhar, S., Fu, R.: AnyHome: Open-vocabulary generation of structured and textured 3d homes. arXiv preprint arXiv:2312.06644 (2023)
• [59]Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A.H., White, R.W., Burger, D., Wang, C.: AutoGen: Enabling next-gen LLM applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155 (2023)
• [60]Wu, T., Yang, G., Li, Z., Zhang, K., Liu, Z., Guibas, L., Lin, D., Wetzstein, G.: GPT-4V (ision) is a human-aligned evaluator for text-to-3d generation. arXiv preprint arXiv:2401.04092 (2024)
• [61]Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1912–1920 (2015)
• [62]Xu, K., Stewart, J., Fiume, E.: Constraint-based automatic placement for scene composition. In: Graphics Interface. vol.2, pp. 25–34 (2002)
• [63]Yang, H., Yue, S., He, Y.: Auto-GPT for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224 (2023)
• [64]Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., Hu, X.: Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data (2023)
• [65]Yu, H.X., Duan, H., Hur, J., Sargent, K., Rubinstein, M., Freeman, W.T., Cole, F., Sun, D., Snavely, N., Wu, J., etal.: WonderJourney: Going from anywhere to everywhere. arXiv preprint arXiv:2312.03884 (2023)
• [66]Zhai, G., Örnek, E.P., Wu, S.C., Di, Y., Tombari, F., Navab, N., Busam, B.: CommonScenes: Generating commonsense 3d indoor scenes with scene graphs. arXiv preprint arXiv:2305.16283 (2023)
• [67]Zhai, G., Örnek, E.P., Wu, S.C., Di, Y., Tombari, F., Navab, N., Busam, B.: CommonScenes: Generating commonsense 3d indoor scenes with scene graphs. Advances in Neural Information Processing Systems (NeurIPS) 36 (2024)
• [68]Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: IEEE/CVF International Conference on Computer Vision (ICCV). pp. 3836–3847 (2023)
• [69]Zheng, J., Zhang, J., Li, J., Tang, R., Gao, S., Zhou, Z.: Structured3D: A large photo-realistic dataset for structured 3d modeling. In: European Conference on Computer Vision (ECCV) (2020)

6 Additional Qualitative Results

In this section, we present additional qualitative outcomes. Fig.6 shows a range of living room renders generated using generic prompts.Fig.7 highlights our method’s capability in handling various prompt types.Fig.8 illustrates the conversion of a generated scene graph into a 3D representation.

7 Implementation of the Multi-agent Pipeline

As stated in Sec.3.2, the responsibility of the Multi-agent pipeline includes suggesting scene objects, determining their relative placements, and structuring this information into a JSON object that conforms to a specified schema. There are a few motivations for distributing these tasks to multiple LLM instances.

The main reason for following a “Divide & Conquer” approach is the observed performance increase when each LLM instance focuses on solving a more trivial sub-problem by attending to the outputs of the previous agents during communication in an attempt to solve the entirety of the problem iteratively. Adding a feedback loop to the communication process further helps erase unwanted behavior from the agents and tackle hallucination issues.

Additionally, task distribution between agents addresses another challenge: not surpassing the output token limit of the LLM when dealing with a large number of objects. Currently, GPT-4 supports an output token limit of 4000 tokens, which constrains the number of objects and information that an LLM instance can handle in a single iteration. Furthermore, we observed that as LLM outputs approach the token limit, the attention to user and system prompts drastically decreases, resulting in object suggestions or placements that do not align with user preferences. Distributing the information processing among multiple agents mitigates this problem and allows more objects to be successfully generated.

In our implementation, the Interior Designer and Interior Architect agents generate and process all objects together in one pass. This design choice enables the agents to attend to each object effectively. We additionally explored alternative approaches wherein objects would be suggested and placed incrementally as the scene is completed. However, our experiments revealed repetitive object suggestion patterns with these alternative schemes, with some objects repeatedly proposed multiple times. This suggests that as the number of objects in the scene increases, the agents may struggle to comprehend the semi-completed scenes effectively. All GPT-4 agents utilize the “JSON mode,” which exclusively restricts their output to the JSON format. This approach further reduces the number of tokens spent in each iteration.

As the Engineer’s task solely involves fitting the outputs of the Interior Designer and Architect agents into a specified schema, the Engineer agent can, in contrast, generate structured outputs for each object individually. This step is independent of each object and does not require consideration of other objects in the scene. Due to the aforementioned limitations regarding large output chunks, we opt for an iterative structuring process for the Engineer agent.

8 Implementation of the GPT-4V Evaluator

In Sec.4.2, we presented a grading-oriented evaluation approach leveraging the GPT-4V model. This system enables the measurement and comparison of abstract concepts like atmosphere and color schemes using renders from the scenes. As reflected in Sec.4.3, the findings from the user study align with the GPT-4V evaluation, demonstrating the effectiveness of our LLM-based evaluation method.

In our current experimental setup, we encode two separate renders of a scene from opposing corners and prompt the GPT-4V evaluator to grade the scene in four categories: Atmosphere, Scheme, Layout, and Functionality. The evaluator is executed three times in total, and the mean grade is calculated to reduce the variability in grades across different runs.

To stimulate a “chain of thought” process[55], we instructed the GPT-4V evaluator to comment on the scenes regarding various grading aspects before generating the actual grades. This approach enables the model to consider its comments throughout the grading process.

9 Discussion

In this section, we delve into the existing limitations of our pipeline and potential future research directions.

One limitation stems from the LLM’s inability to consider multiple factors simultaneously (including the rotation of the parent object, placement of other children, availability of space, and so forth) when integrating an object into a scene, particularly as the scene becomes more complex.Thus, as the scene graph grows, the likelihood of the LLM suggesting unreasonable relational edges among objects increases. Likewise, as the number of objects in the scene increases, LLMs struggle to monitor the remaining available space, which may result in an overflow of objects in certain parts of the scene.

Another issue arises due to the discrepancy between the bounding box scale ratios inferred by the LLM and the actual asset retrieved. A laptop is a good example of such a case since closed and open laptops have very different bounding box ratios. If the agent infers a closed laptop, but the retrieved asset is an open laptop, the renders may result in visually implausible outcomes with distortions because of the resizing process. A similar discrepancy might also arise for object orientations. The canonical orientation of an asset might differ from the canonical orientation inferred by agents. For example, determining the front side of a desk can be subjective. The front side might be considered where a person would typically place a chair, or it could be perceived as the opposite side.

To maintain a consistent and immersive atmosphere, cohesive texture becomes essential. However, the original textures of retrieved objects from the dataset cannot be guaranteed to align seamlessly with user preference. Achieving precise control over texture and geometry-related features remains challenging despite extracting assets from large-scale datasets guided by text embeddings to ensure semantic alignment. Future work may consider incorporating a re-texturing step to enhance the coherence of the overall atmosphere.

Lastly, we place objects in the scene with a bounding box assumption, meaning that the objects are assumed to have quadratic surfaces. While this approach simplifies the spatial representation of 3D objects and prevents collisions, it can lead to spatial inconsistencies when placing child objects, especially when placing a child object on top of a parent object. This limitation may result in artifacts such as “floating” objects. For further work, we encourage exploring an extra step that utilizes mesh surfaces instead of bounding boxes for precise placement.

10 System Prompts for Agents

Creating the scene graph involves various agents, each playing a distinct role within the overall generation process. These roles are imparted to the agents through “system prompts,” shaping their responses to user prompts and instilling them with individual characteristics. These prompts help define each agent’s functionality and guide them in generating structured outputs. The system prompts for each agent are provided verbatim below. The JSON schemas provided to each agent (highlighted with {json_schema}) are included in the supplementary files and the project webpage.

11 Room Synthesis Prompt Generation for Evaluation

Our proposed pipeline aims to serve as a personalized interior design assistant capable of offering tailored design suggestions based on diverse user inputs, reflecting various preferences and requirements. Our motivation is to construct prompts that reflect the authentic comments real people might make when designing their rooms.

In the realm of interior design, the significance of both aesthetics and functionality is consistently emphasized[10]. A deeper exploration of personalized interior design, as articulated in[27], identifies four pivotal aspects central to decision-making: indoor space components, human tendencies, technology, and spatial evaluation. Indoor space components encompass factors such as materials, layout, and arrangement, while human tendencies consider individual inclinations influenced by past experiences and personalities. Spatial evaluation accentuates productivity, underlining the importance of creating environments conducive to efficient work.

Drawing from this framework, we establish our GPT-4V evaluation criterion and generate prompts by varying key elements, including functionality, layout, color scheme with material choice, and overall atmosphere. These prompts are generated utilizing GPT-4. For instance, in generating prompts related to functionality, we instruct GPT-4 to provide descriptions for different room types, incorporating varied functionalities and dimension specifications. Similarly, for prompts assessing layout, color scheme, and overall atmosphere alignment, we maintain fixed room types and dimensions while altering layout, color scheme, or atmosphere. We generate ten prompts for each alignment consideration. Below are the prompts for generating room preferences and a comprehensive list of prompts used in the evaluationsTab.1 and Tab.2 of the main paper.