Sakana AI drops image models to generate Japan’s traditional ukiyo-e artwork

Keep in mind Sakana AI? Nearly a yr in the past, the Tokyo-based startup made a hanging look on the AI scene with its high-profile founders from Google and a novel automated merging-based method to creating high-performing fashions. Right this moment, the corporate introduced two new image-generation fashions: Evo-Ukiyoe and Evo-Nishikie.

Obtainable on Hugging Face, the fashions have been designed to generate photos from textual content and picture prompts. Nevertheless, there’s an fascinating and distinctive catch: as an alternative of dealing with common picture technology in numerous types, these fashions are laser-focused on Japan’s widespread historic artwork type ukiyo-e. It flourished between the seventeenth and nineteenth centuries, and Sakana hopes to convey it again to fashionable content material customers utilizing the ability of AI.

The transfer comes as the most recent localization effort within the AI area — one thing that has grown over the previous yr, with corporations in nations like South Korea, India and China constructing fashions tailor-made to their respective cultures and dialects. 

What to anticipate from the brand new Sakana AI fashions?

Courting again to the early 1600s, Ukiyo-e – or “footage of the floating world” – developed as a preferred artwork in Japan specializing in topics like historic scenes, landscapes, sumo wrestlers, and so forth. The style revolved round monochrome woodblock prints however ultimately graduated to full-color prints or “nishiki-e” with a number of woodblocks. Its reputation declined within the nineteenth as a consequence of a number of elements, together with the rise of digital pictures.

Now, with the discharge of the 2 image-generation fashions, Sakana desires to convey the historic art work again into widespread tradition. The primary one – Evo-Ukiyoe – is a text-to-image providing that generates photos intently resembling ukiyo-e, particularly when prompted with textual content inputs describing components generally present in ukiyo-e artwork reminiscent of cherry blossoms, kimono or birds. It may well even generate ukiyo-e-style artwork with issues that didn’t exist again then, like a hamburger or laptop computer, however the firm factors out that typically the outcomes could veer off observe — not resembling ukiyo-e in any respect.

The mannequin is predicated on Evo-SDXL-JP, which Sakana developed utilizing its novel evolutionary model merging technique on high of Stability AI’s SDXL and different open diffusion fashions. The corporate mentioned it used LoRA (Low-Rank Adaptation) to fine-tune Evo-SDXL-JP on a dataset of over 24,000 carefully-captioned ukiyo-e artworks acquired by way of a partnership with the Artwork Analysis Heart (ARC) of Ritsumeikan College in Kyoto. 

“We curated this knowledge with a variety of topics, overlaying together with complete artwork and face-centered ones, from the digital photos of ukiyo-e within the ARC assortment. We additionally targeted on multi-colored nishiki-e with lovely colours whereas contemplating variety,” the corporate wrote in a weblog submit.

The second mannequin, Evo-Nishikie, is an image-to-image providing that colorizes monochrome Ukiyo-e prints. Sakana says it could actually add coloration to historic e-book illustrations that have been printed in a single coloration of ink or give solely new seems to current multi-colored Nishikie prints. All of the consumer must do is present the supply picture and possibly pair it with a set of directions describing the weather to be coloured.

Sakana mentioned it introduced this mannequin to life by performing ControlNet coaching on Evo-Ukiyoe, utilizing mounted prompts and situation photos.

Aim for additional analysis and growth

Whereas the fashions solely assist prompting in Japanese and are within the very early levels, Sakana hopes the work to show AI conventional “Japanese magnificence” will unfold the enchantment of the nation’s tradition worldwide and discover purposes in schooling and new methods of having fun with classical literature.

At the moment, the corporate is offering each fashions and the related code to get began on Hugging Face. The Python script included within the repository and LoRA weights can be found beneath the Apache 2.0 license.

“This mannequin is offered for analysis and growth functions solely and must be thought of as an experimental prototype. It’s not supposed for business use or deployment in mission-critical environments. Use of this mannequin is on the consumer’s personal danger, and its efficiency and outcomes should not assured,” the corporate notes on Hugging Face.

So, far Sakana AI has raised $30 million in funding from a number of buyers, together with by Lux Capital, which has invested in pioneering AI corporations like Hugging Face, and likewise Khosla Ventures, identified for investing in OpenAI approach again in 2019.