[{"_1":2,"_176":-5,"_177":-5},"loaderData",{"_3":4,"_13":14,"_20":21,"_155":156},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","routes/_layout",{"_15":-5,"_16":17,"_18":17,"_19":11},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","routes/_layout.r.$namespace.$reponame",{"_22":23,"_24":-5,"_25":11,"_26":11,"_27":28,"_29":30,"_31":-5,"_32":11,"_33":34,"_85":86,"_110":111,"_148":-5,"_149":11,"_150":17,"_151":11,"_152":-5,"_153":154},"canonicalUrl","https://hub.docker.com/r/jinaai/dalle-flow/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","jinaai","reponame","dalle-flow","cloudBuildSource","isAutobuildsAvailable","repo",{"_35":36,"_37":38,"_40":41,"_42":43,"_44":11,"_45":46,"_47":48,"_49":50,"_52":30,"_27":28,"_53":54,"_55":39,"_56":57,"_58":59,"_60":61,"_62":-5,"_63":64,"_65":66,"_67":11,"_68":69,"_70":71,"_76":11,"_77":78,"_82":28,"_83":84},"categories",[],"content_types",[39],"image","date_registered","2022-06-21T17:42:36.994753Z","description","A Human-in-the-Loop workflow for creating HD images from text","is_private","last_updated","2022-10-27T09:44:41.414766Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[51],"application/vnd.docker.container.image.v1+json","name","pull_count",3447,"repository_type","star_count",10,"status",1,"status_description","active","source","collaborator_count",0,"full_description","

\n $\"DALL·E$ \n
\nA Human-in-the-loop^? workflow for creating HD images from text\n

\n\n

\n\n $\"Open$ \n $\"Docker$ \n\n

\n\n\n\n\nDALL·E Flow is an interactive workflow for generating high-definition images from text prompt. First, it leverages [DALL·E-Mega](https://github.com/borisdayma/dalle-mini), [GLID-3 XL](https://github.com/Jack000/glid-3-xl), and [Stable Diffusion](https://github.com/CompVis/stable-diffusion) to generate image candidates, and then calls [CLIP-as-service](https://github.com/jina-ai/clip-as-service) to rank the candidates w.r.t. the prompt. The preferred candidate is fed to [GLID-3 XL](https://github.com/Jack000/glid-3-xl) for diffusion, which often enriches the texture and background. Finally, the candidate is upscaled to 1024x1024 via [SwinIR](https://github.com/JingyunLiang/SwinIR).\n\nDALL·E Flow is built with [Jina](https://github.com/jina-ai/jina) in a client-server architecture, which gives it high scalability, non-blocking streaming, and a modern Pythonic interface. Client can interact with the server via gRPC/Websocket/HTTP with TLS.\n\n**Why Human-in-the-loop?** Generative art is a creative process. While recent advances of DALL·E unleash people's creativity, having a single-prompt-single-output UX/UI locks the imagination to a _single_ possibility, which is bad no matter how fine this single result is. DALL·E Flow is an alternative to the one-liner, by formalizing the generative art as an iterative procedure.\n\n## Usage\n\nDALL·E Flow is in client-server architecture.\n- [Client usage](#Client)\n- [Server usage, i.e. deploy your own server](#Server)\n\n\n## Updates\n\n- ⚠️ **2022/10/26** To use CLIP-as-service available at `grpcs://api.clip.jina.ai:2096` (requires `jina >= v3.11.0`), you need first get an access token from [here](https://console.clip.jina.ai/get_started). See [Use the CLIP-as-service](#use-the-clip-as-service) for more details.\n- 🌟 **2022/9/25** Automated [CLIP-based segmentation](https://github.com/timojl/clipseg) from a prompt has been added.\n- 🌟 **2022/8/17** Text to image for [Stable Diffusion](https://github.com/CompVis/stable-diffusion) has been added. In order to use it you will need to agree to their ToS, download the weights, then enable the flag in docker or `flow_parser.py`.\n- ⚠️ **2022/8/8** Started using CLIP-as-service as an [external executor](https://docs.jina.ai/fundamentals/flow/add-executors/#external-executors). Now you can easily [deploy your own CLIP executor](#run-your-own-clip) if you want. There is [a small breaking change](https://github.com/jina-ai/dalle-flow/pull/74/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R103) as a result of this improvement, so [please _reopen_ the notebook in Google Colab](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n- ⚠️ **2022/7/6** Demo server migration to AWS EKS for better availability and robustness, **server URL is now changing to `grpcs://dalle-flow.dev.jina.ai`**. All connections are now with TLS encryption, [please _reopen_ the notebook in Google Colab](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n- ⚠️ **2022/6/25** Unexpected downtime between 6/25 0:00 - 12:00 CET due to out of GPU quotas. The new server now has 2 GPUs, add healthcheck in client notebook.\n- **2022/6/3** Reduce default number of images to 2 per pathway, 4 for diffusion.\n- 🐳 **2022/6/21** [A prebuilt image is now available on Docker Hub!](https://hub.docker.com/r/jinaai/dalle-flow) This image can be run out-of-the-box on CUDA 11.6. Fix an upstream bug in CLIP-as-service. \n- ⚠️ **2022/5/23** Fix an upstream bug in CLIP-as-service. This bug makes the 2nd diffusion step irrelevant to the given texts. New Dockerfile proved to be reproducible on a AWS EC2 `p2.x8large` instance.\n- **2022/5/13b** Removing TLS as Cloudflare gives 100s timeout, making DALLE Flow in usable [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n- 🔐 **2022/5/13** New Mega checkpoint! All connections are now with TLS, [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n- 🐳 **2022/5/10** [A Dockerfile is added! Now you can easily deploy your own DALL·E Flow](#run-in-docker). New Mega checkpoint! Smaller memory-footprint, the whole Flow can now fit into **one GPU with 21GB memory**.\n- 🌟 **2022/5/7** New Mega checkpoint & multiple optimization on GLID3: less memory-footprint, use `ViT-L/14@336px` from CLIP-as-service, `steps 100->200`. \n- 🌟 **2022/5/6** DALL·E Flow just got updated! [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb)\n - Revised the first step: 16 candidates are generated, 8 from DALL·E Mega, 8 from GLID3-XL; then ranked by CLIP-as-service.\n - Improved the flow efficiency: the overall speed, including diffusion and upscaling are much faster now!\n\n\n## Gallery\n\n $\"a$ $\"A$ $\"an$ $\"Eternal$ $\"another$ $\"The$ $\"A$ $\"A$ $\"a$ $\"oil$ $\"Colossus$ $\"landscape$ $\"an$ $\"An$ $\"dinosaurs$ $\"fantasy$ $\"GPU$ $\"a$ $\"Paddington$ $\"a$ $\"A$ $\"an$ $\"an$ $\"city,$ $\"an$ $\"a$ $\"a$ $\"a$ $\"a$ $\"cozy$ $\"an$ $\"the$ $\"A$ $\"a$ $\"Bionic$ $\"The$ $\"painting$ $\"a$ $\"fantasy$ $\"ocean$ $\"An$ $\"antique$ $\"a$ $\"an$ $\"a$ $\"a$ $\"A$ $\"golden$ \n\n## Client\n\n $\"Open$ \n\nUsing client is super easy. The following steps are best run in [Jupyter notebook](./client.ipynb) or [Google Colab](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb). \n\nYou will need to install [DocArray](https://github.com/jina-ai/docarray) and [Jina](https://github.com/jina-ai/jina) first:\n\n```bash\npip install \"docarray[common]>=0.13.5\" jina\n```\n\nWe have provided a demo server for you to play:\n> ⚠️ **Due to the massive requests, our server may be delay in response. Yet we are _very_ confident on keeping the uptime high.** You can also deploy your own server by [following the instruction here](#server).\n\n```python\nserver_url = 'grpc://dalle-flow.jina.ai:51005'\n```\n\n\n### Step 1: Generate via DALL·E Mega\n\nNow let's define the prompt:\n\n```python\nprompt = 'an oil painting of a humanoid robot playing chess in the style of Matisse'\n```\n\nLet's submit it to the server and visualize the results:\n\n```python\nfrom docarray import Document\n\ndoc = Document(text=prompt).post(server_url, parameters={'num_images': 8})\nda = doc.matches\n\nda.plot_image_sprites(fig_size=(10,10), show_index=True)\n```\n\nHere we generate 24 candidates, 8 from DALLE-mega, 8 from GLID3 XL, and 8 from Stable Diffusion, this is as defined in `num_images`, which takes about ~2 minutes. You can use a smaller value if it is too long for you. \n\n\n

\n\n

\n\n### Step 2: Select and refinement via GLID3 XL\n\nThe 24 candidates are sorted by [CLIP-as-service](https://github.com/jina-ai/clip-as-service), with index-`0` as the best candidate judged by CLIP. Of course, you may think differently. Notice the number in the top-left corner? Select the one you like the most and get a better view:\n\n```python\nfav_id = 3\nfav = da[fav_id]\nfav.embedding = doc.embedding\nfav.display()\n```\n\n

\n\n

\n\nNow let's submit the selected candidates to the server for diffusion.\n\n```python\ndiffused = fav.post(f'{server_url}', parameters={'skip_rate': 0.5, 'num_images': 36}, target_executor='diffusion').matches\n\ndiffused.plot_image_sprites(fig_size=(10,10), show_index=True)\n```\n\nThis will give 36 images based on the selected image. You may allow the model to improvise more by giving `skip_rate` a near-zero value, or a near-one value to force its closeness to the given image. The whole procedure takes about ~2 minutes.\n\n

\n\n

\n\n### Step 3: Select and upscale via SwinIR\n\nSelect the image you like the most, and give it a closer look:\n\n```python\ndfav_id = 34\nfav = diffused[dfav_id]\nfav.display()\n```\n\n

\n\n

\n\n\nFinally, submit to the server for the last step: upscaling to 1024 x 1024px.\n\n```python\nfav = fav.post(f'{server_url}/upscale')\nfav.display()\n```\n\nThat's it! It is _the one_. If not satisfied, please repeat the procedure.\n\n

\n\n

\n\nBtw, DocArray is a powerful and easy-to-use data structure for unstructured data. It is super productive for data scientists who work in cross-/multi-modal domain. To learn more about DocArray, [please check out the docs](https://docs.jina.ai).\n\n## Server\n\nYou can host your own server by following the instruction below.\n\n### Hardware requirements\n\nDALL·E Flow needs one GPU with 21GB VRAM at its peak. All services are squeezed into this one GPU, this includes (roughly)\n- DALLE ~9GB\n- GLID Diffusion ~6GB\n- Stable Diffusion ~7GB (batch_size=1 in `config.yml`, 512x512, slower) or ~14GB (batch_size=4 in `config.yml`, 512x512, slightly faster)\n- SwinIR ~3GB\n- CLIP ViT-L/14-336px ~3GB\n\nThe following reasonable tricks can be used for further reducing VRAM:\n- SwinIR can be moved to CPU (-3GB)\n- CLIP can be delegated to [CLIP-as-service free server](https://console.clip.jina.ai/get_started) (-3GB)\n\n\nIt requires at least 50GB free space on the hard drive, mostly","has_starred","hub_user","jinadev","immutable_tags_settings",{"_72":11,"_73":74},"enabled","rules",[75],".*","is_automated","permissions",{"_79":17,"_80":11,"_81":11},"read","write","admin","user","storage_size",72801133098,"orgData",{"_87":88,"_89":90,"_91":92,"_93":28,"_94":95,"_96":97,"_98":99,"_100":101,"_102":103,"_104":105,"_106":107,"_108":107,"_109":17},"id","a907c72c3c424a209789c2fa209f40ee","uuid","a907c72c-3c42-4a20-9789-c2fa209f40ee","type","organization","orgname","full_name","Jina AI","location","Global","company","Jina AI Limited","profile_url","https://oss.jina.ai","date_joined","2020-02-03T14:37:22.038389Z","gravatar_url","https://www.gravatar.com/avatar/b51ae23313f6bce542b82244fb7df972?s=80&r=g&d=mm","gravatar_email","","badge","is_active","recentTags",[112],{"_113":39,"_114":115,"_116":117,"_118":119,"_87":120,"_121":122,"_45":137,"_138":115,"_139":28,"_140":51,"_52":141,"_142":143,"_144":128,"_145":137,"_146":61,"_147":17},"content_type","creator",9198541,"digest","sha256:4bef531767e511c93ff1bb956b95fcc950fed1b8731d572ad5832a3092b5db2f","full_size",9607760398,238971206,"images",[123],{"_124":125,"_116":117,"_126":107,"_127":128,"_129":130,"_131":132,"_133":107,"_134":-5,"_135":119,"_58":61,"_136":-5},"architecture","amd64","features","last_pulled","2026-09-12T12:41:39.01566161Z","last_pushed","2022-10-27T09:44:42Z","os","linux","os_features","os_version","size","variant","2022-10-27T09:44:41.297563Z","last_updater","last_updater_username","media_type","latest","repository",17416640,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"routes/_layout.r.$namespace.$reponame.tags",{"_33":157,"_164":165,"_153":174,"_175":11},{"_35":158,"_37":159,"_40":41,"_42":43,"_44":11,"_45":46,"_47":48,"_49":160,"_52":30,"_27":28,"_53":54,"_55":39,"_56":57,"_58":59,"_60":61,"_62":-5,"_63":64,"_65":66,"_67":11,"_68":69,"_70":161,"_76":11,"_77":163,"_82":28,"_83":84},[],[39],[51],{"_72":11,"_73":162},[75],{"_79":17,"_80":11,"_81":11},"tags",{"_166":59,"_167":-5,"_168":-5,"_169":170},"count","next","previous","results",[171],{"_113":39,"_114":115,"_116":117,"_118":119,"_87":120,"_121":172,"_45":137,"_138":115,"_139":28,"_140":51,"_52":141,"_142":143,"_144":128,"_145":137,"_146":61,"_147":17},[173],{"_124":125,"_116":117,"_126":107,"_127":128,"_129":130,"_131":132,"_133":107,"_134":-5,"_135":119,"_58":61,"_136":-5},[],"canDeleteTags","actionData","errors"]

jinaai/dalle-flow

Tags for jinaai/dalle-flow