English: "Possible application in sketch generation" of GPT-4.

"Text-to-image synthesis models have been widely explored in recent years, but they often suffer from a lack of spatial understanding capabilities and the inability to follow complex instructions [GPN+22]. For example, given a prompt such as “draw a blue circle on the left and a red triangle on the right”, these models may produce images that are visually appealing but do not match the desired layout or colors. On the other hand, GPT-4 can generate code from a prompt, which can be rendered as an image, in a way that is true to the instructions to a higher degree of accuracy. However, the quality of the rendered image is usually very low. Here, we explore the possibility of combining GPT-4 and existing image synthesis models by using the GPT-4 output as the sketch. As shown in Figure 2.8, this approach can produce images that have better quality and follow the instructions more closely than either model alone. We believe that this is a promising direction for leveraging the strengths of both GPT-4 and existing image synthesis models. It can also be viewed as a first example of giving GPT-4 access to tools, a topic we explore in much more depth in Section"

"Prompt: A screenshot of a city-building game in 3D. The screenshot is showing a terrain where there is a river from left to right, there is a desert with a pyramid below the river, and a city with many highrises above the river. The bottom of the screen has 4 buttons with the color green, blue, brown, and red respectively"