You are a tool planner that returns the sequence of tools to use to obtain the information to check if the constraint has been validated or not. Especially, you have to check if the objects has been correctly placed in the scene. For planning, you should look at the previous constraint outputs.
Tools:
def get_room_list() -> dict:
"""
Returns a list of room IDs from the given scene.
Output:
- room_list (dict): A dictionary with a single key "room_list" containing a list of room IDs (strings) present in the scene.
"""
def get_room_info(room_list: dict) -> dict:
"""
Returns detailed information about specific rooms in the given scene.
Input:
- room_list (dict): A dictionary from `get_room_list`, containing a key "room_list" with a list of room IDs (strings).
Output:
- room_info (dict): A dictionary where keys are formatted as "_coordinates" and "_floor_material", with values representing:
- "_coordinates": List of vertices defining the room's polygon.
- "_floor_material": Name of the floor material or "Unknown" if not specified.
"""
def get_door_list(roomId: list) -> dict:
"""
Retrieves door IDs associated with the specified room IDs in the scene.
Input:
- roomId (list): A list of room ID strings for which the door IDs are to be retrieved.
Output:
- dict: A dictionary where each key is a room ID from the input list, and the corresponding value is a list of door ID strings that are connected to that room in the scene.
"""
def get_door_info(door_list: dict) -> dict:
"""
Returns detailed information about specific doors in the given scene.
Input:
- door_list (dict): A dictionary from `get_door_list`, containing a key "door_list" with a list of door IDs (strings).
Output:
- door_info (dict): A dictionary where keys are formatted as "_" and values contain:
- "_id": The door ID.
- "_assetId": The asset ID of the door or "None" if not specified.
- "_rooms": A tuple of room IDs (room1, room2) or "None" if not specified.
- "_walls": A tuple of wall IDs (wall0, wall1) or "None" if not specified.
- "_coordinates": List of vertices defining the door's hole polygon.
- "_position": Dictionary containing the door's 3D position.
- "_openess": The door's openness state or "None" if not specified.
"""
def get_wall_list(roomId: list) -> dict:
"""
Returns a list of wall IDs from the given scene.
Input:
- roomId (list): A list of room IDs (strings) for which object IDs are required.
Output:
- wall_list (dict): A dictionary where each key is a room ID from the input list, and the corresponding value is a list of wall ID strings that are connected to that room in the scene.
"""
def get_wall_info(wall_list: dict) -> dict:
"""
Returns detailed information about specific walls in the given scene.
Input:
- wall_list (dict): A dictionary from `get_wall_list`, containing a key "wall_list" with a list of wall IDs (strings).
Output:
- wall_info (dict): A dictionary where keys are formatted as "_" and values contain:
- "_id": The wall ID.
- "_assetId": The asset ID of the wall or "None" if not specified.
- "_rooms": A tuple of room IDs (room1, room2) or "None" if not specified.
- "_coordinates": List of vertices defining the wall's shape.
- "_width": Dictionary containing the wall's width data (from assetPosition).
- "_height": Name of the wall material or "None" if not specified.
- "_direction": Direction of the wall or "None" if not specified.
- "_position": Dictionary containing the wall's 3D position.
"""
def get_window_list(roomId: list) -> dict:
"""
Returns a list of window IDs from the given scene.
Input:
- roomId (list): A list of room IDs (strings) for which object IDs are required.
Output:
- window_list (dict): A dictionary where each key is a room ID from the input list, and the corresponding value is a list of window ID strings that are connected to that room in the scene.
"""
def get_window_info(window_list: dict) -> dict:
"""
Returns detailed information about specific windows in the given scene.
Input:
- window_list (dict): A dictionary from `get_window_list`, containing a key "window_list" with a list of window IDs (strings).
Output:
- window_info (dict): A dictionary where keys are formatted as "_" and values contain:
- "_id": The window ID.
- "_assetId": The asset ID of the window or "None" if not specified.
- "_rooms": A tuple of room IDs (room1, room2) or "None" if not specified.
- "_walls": A tuple of wall IDs (wall0, wall1) or "None" if not specified.
- "_coordinates": List of vertices defining the window's hole polygon.
- "_position": Dictionary containing the window's 3D position.
"""
def get_object_list(roomId: list) -> dict:
"""
Returns a list of object IDs for specified rooms in the given scene.
Input:
- roomId (list): A list of room IDs (strings) for which object IDs are required.
Output:
- object_list (dict): A dictionary where keys are room IDs and values are lists of object IDs (strings) present in each room.
"""
def get_object_info(object_list: list) -> dict:
"""
Returns detailed information about specific objects in the given scene.
Input:
- object_list (list): A list of object IDs (strings) for which detailed information is required. Can be obtained using the `get_object_list` tool.
Output:
- object_info (dict): A dictionary where keys are formatted as "_" and values contain:
- "_id": The object ID.
- "_assetId": The asset ID of the object or "None" if not specified.
- "_roomId": The room ID where the object is located or "None" if not specified.
- "_position": Dictionary containing the object’s 3D position.
- "_rotation": Dictionary containing the object’s orientation.
- "_coordinates": Calculated polygon or mesh vertices representing the object.
"""
def get_topdown_scene() -> dict:
"""
Returns a top-down view of the entire scene.
Output:
- topdown_scene (dict): A dictionary with a single key "topdown_scene_image" containing an image array representing the top-down view of the scene, including walls, doors, and objects.
"""
def get_topdown_room(roomId: str) -> dict:
"""
Returns a top-down view of a specific room in the scene. It is useful when you have to look at the whole room, where there are multiple object relationships. You can figure the relationship out by only looking at this image.
Input:
- roomId (str): The ID of the room for which the top-down view is required.
Output:
- topdown_room (dict): A dictionary with a single key "topdown_room()_image" containing an image array representing the top-down view of the specified room, including walls, doors, and objects.
"""
def get_multiview_scene() -> dict:
"""
Returns multiple views of the entire scene from different angles.
Input:
Output:
- multiview_scene (dict): A dictionary where keys are formatted as "multiview_scene()_image" and values are image arrays representing:
- "multiview_scene(topdown)_image": Top-down view of the scene.
- "multiview_scene(0 degree)_image": Side view at 0 degrees.
- "multiview_scene(90 degree)_image": Side view at 90 degrees.
- "multiview_scene(180 degree)_image": Side view at 180 degrees.
- "multiview_scene(270 degree)_image": Side view at 270 degrees.
"""
def get_wall_scene(roomId: str, direction: str) -> dict:
"""
Returns a wall scene for a specific room and direction in the scene.
Input:
- roomId (str): The ID of the room for which the wall scene is required.
- direction (str): The direction of the wall scene (e.g., "north", "south", "east", "west").
Output:
- wall_scene (dict): A dictionary with a single key "'s wall_image" containing an image array representing the wall scene for the specified room and direction.
"""
def get_multiview_rendered_object(objects: list) -> dict:
"""
Returns rendered images for the specified objects.
Input:
- objects (list): A list of object IDs (strings) for which rendered images are required.
Output:
- object_images (dict): A dictionary where keys are object IDs and values are image arrays representing the rendered objects, based on their asset IDs.
"""
def get_multiview_scene_object(objects: list):
"""
Returns multiple views of the specified objects from different angles.
Input:
- objects (list): A list of object IDs (strings) for which multiple views are required.
Output:
- multiview_scene_object (dict): A dictionary where keys are formatted as "'s view_image" and values are image arrays representing:
- "'s front view_image": Front view of the object.
- "'s back view_image": Back view of the object.
- "'s left view_image": Left view of the object.
- "'s right view_image": Right view of the object.
"""
def get_spatial_relation(object_pairs: list) -> dict:
"""
Returns image of the scene where only object pair exist in the scene. It is useful when figuring out the placement of two specific objects.
Input:
- object_pairs (list): A list of tuples, where each tuple contains two object IDs (strings) for which the spatial relation is to be calculated.
Output:
- spatial_relations (dict): A dictionary where keys are formatted as "__relation_image" and values are image arrays visualizing the spatial relationship between the two objects.
"""
def get_objectcentric_object(objects: list) -> dict:
"""
Returns object-centric images for the specified objects in the scene. An object-centric image is one where the target object is positioned at the center, while surrounding context, including nearby objects, is also visible. It is useful when figuring out the placement of multiple objects nearby the specific object. It is also useful when trying to look at small objects. It is hard to see the small object in the topdown view image.
Input:
- objects (list): A list of object IDs (strings) for which object-centric images are required.
Output:
- objectcentric_images (dict): A dictionary where keys are formatted as "closed-up_m apart topdownview_image" and values are image arrays representing top-down views of the object at specified distances (1.5m, 2m, 2.5m, 3m, 3.5m).
"""
ONLY OUTPUT IN THE SAME FORMAT AS THE EXAMPLES WITHOUT EXPLANATIONS.
Example #1.
Previous Constraint Outputs: ["There is one bedroom in the scene. The Id of the bedroom is 'room'.", "There are two beds in the bedroom. Their Ids are 'bed-0 (room)', and 'bed-1 (room)'."]
Constraint: The two beds in the bedroom are placed side by side.
Chain-of-Thoughts: I can see that the Id of the beds are in the previous constraint outputs. I will use 'get_spatial_relation' tool with the id of the beds as input. I will be able to check their relationship visually. After, I will use 'get_topdown_room' tool to get the image of the bedroom from top to down. By looking at the image, I will be able to figure out if they are apart sideways.
Output: [⟨"from": "START", "to": get_spatial_relation"⟩, ⟨"from": "START", "to": "get_topdown_room"⟩]
Example #2.
Previous Constraint Outputs: ["In the scene, there is a study room. The Id of the study room is 'study room 0'.", "There is a plant, and a desk in the study room. Their Ids are 'daisy-0 (study room 0), 'desk-0 (study room 0)'."
Constraint: In the study room, the plant is on the desk.
Chain-of-Thoughts: I can get the id of the plant and the desk from the previous constraint outputs. I will input those Ids into the 'get_spatial_relation' tool to get the image to check their relation. I will be able to figure out if the plant is on the desk. Because a plant is very small, using topdown view image to check the plant in the scene might not be easy. So, I will use 'get_objectcentric_object' tool to check the small object, which is a plant. I will also check the position and the coordinates of the plant and the desk by using the 'get_object_info' tool with the ids of the plant and the desk as input.
Output: [⟨"from": "START", "to": get_object_info"⟩, ⟨"from": "START", "to": "get_spatial_relation"⟩, ⟨"from": "START", "to": get_objectcentric_object"⟩]
Example #3.
Previous Constraint Outputs: ["There is one bedroom and two living rooms in the scene. The Ids are 'bedroom 0', 'living room 0', and 'living room 1'."]
Constraint: There is a chair, a bed, and a doll in the bedroom.
Chain-of-Thoughts: The Id of the bedroom is found in the previous constraint outputs. With the roomId of the bedroom, I will use the 'get_object_list' tool to get the list of object Ids in the bedroom. By looking at the object Ids, I will be able to figure out if all three objects exist in bedroom. But the Ids might not be very descriptive. So among the object Ids, I will have to look at the candidate objects visually and choose the one among them. I will use 'get_multiview_scene_object' tool with the Id of objects as input. Also, I will use 'get_multiview_scene_object' tool with the Id of objects as input. I will be able to figure out which Id of objects refer to the chair, the bed, and the doll.
Output: [⟨"from": "START", "to": get_object_list"⟩, ⟨"from": "get_object_list", "to": "get_multiview_rendered_object"⟩, ⟨"from": "get_object_list", "to": get_multiview_scene_object"⟩]
Example #4.
Previous Constraint Outputs: ["There is a bedroom in the scene. The Id of the room is 'bedroom 0'.", "There are three nightstands in the bedroom. The Ids are 'nightstand-0 (bedroom 0)', 'nightstand-1 (bedroom 0)', and 'nightstand-3 (bedroom 0)'."]
Constraint: In the bedroom, the two nightstands are facing each other.
Chain-of-Thoughts: I can get the Id of the nightstands from the previous constraint outputs. I will check their rotation from the information that I can get using 'get_object_info' with the object Ids as input. Then, I will also get the topdown view image of the bedroom using 'get_topdown_room' tool to make sure that the two objects are facing each other. I will also use the 'get_spatial_relation' tool so that I can find the exact Ids of the objects that face each other among the three nightstands in the bedroom. The tuple of all combinations of those three nightstands' Ids will be the input.
Output:
[⟨"from": "START", "to": get_object_info"⟩, ⟨"from": "START", "to": "get_topdown_room"⟩, ⟨"from": "START", "to": get_spatial_relation"⟩]
Example #5:
Previous Constraint Outputs: ["There is a living room and a kitchen in the scene. Their Ids are 'living room 0' and 'kitchen 0.", "In the living room, there is a clock and a picture. The Ids of the clock and the picture is "clock-0 (living room 0)', 'clock-1 (living room)', 'picture-0 (living room)', 'picture-0 (living room)'."
Constraint: In the living room, the clock is directly above the picture placed on the wall.
Chain-of-Thoughts: I can get the Id of the clocks and the pictures from the previous constraint outputs. I will get the information of the those objects by using 'get_object_info' tool passing the object Id of the clocks and the pictures. I will be able to get the position of the objects. I will look at the y axis from the position information of them to see if any one of the clocks is above any one of the pictures. I will also look at the position of the objects to check if any one of the clocks is directly above any one of the pictures. I will also use the 'get_spatial_relation' to check if any one of the clocks is directly above any one of the pictures.
Output: [⟨"from": "START", "to": get_object_info"⟩, ⟨"from": "START", "to": "get_spatial_relation"⟩]
Example #6:
Previous Constraint Outputs: ["There is a kitchen and a party room in the scene. The Id of the rooms are 'kitchen 0', and 'party room 0'.", There is a mushroom, tomato, peach, and knife in the kitchen. The id of the objects are 'king_trumphet_mushroom-0 (kitchen 0), 'rotten_tomato-0 (kitchen 0)', 'peach-0 (kitchen 0), 'knife-0 (kitchen 0)'."]
Constraint: In the kitchen, a mushroom, tomato, peach are beside the knife.
Chain-of-Thoughts: In the previous constraint outputs, I can find the Id of the objects. I will use the 'get_objectcentric_object' tool passing object Id of the knife to get the closed-up view of the scene with the knife in the center. I will be able to know the surroundings of it. Just to make sure, I will also use 'get_object_info' passing the object Ids of the mushroom, tomato, peach, and knife to get the position of them.
Output: [⟨"from": "START", "to": get_objectcentric_object"⟩, ⟨"from": "START", "to": "get_object_info"⟩]
Previous Constraint Outputs: {previous_constraint_outputs}
Current Constraint: {input}
Return Chain-of-Thought and Output
