FRIDA, a robotic arm with a paintbrush taped to it, makes use of synthetic intelligence to collaborate with people on artworks. Ask FRIDA to color an image, and it will get to work placing brush to canvas.

“There’s this one portray of a frog ballerina that I feel turned out actually properly,” stated Peter Schaldenbrand, a Faculty of Laptop Science Ph.D. pupil within the Robotics Institute working with FRIDA and exploring AI and creativity. “It’s actually foolish and enjoyable, and I feel the shock of what FRIDA generated primarily based on my enter was actually enjoyable to see.”

FRIDA, named after Frida Kahlo, stands for Framework and Robotics Initiative for Growing Arts. The challenge is led by Schaldenbrand with RI school members Jean Oh and Jim McCann, and has attracted college students and researchers throughout CMU.

Customers can direct FRIDA by inputting a textual content description, submitting different artworks to encourage its fashion, or importing {a photograph} and asking it to color a illustration of it. The crew is experimenting with different inputs as properly, together with audio. They performed ABBA’s “Dancing Queen” and requested FRIDA to color it.

“FRIDA is a robotic portray system, however FRIDA will not be an artist,” Schaldenbrand stated. “FRIDA will not be producing the concepts to speak. FRIDA is a system that an artist may collaborate with. The artist can specify high-level objectives for FRIDA after which FRIDA can execute them.”

The robotic makes use of AI fashions just like these powering instruments like OpenAI’s ChatGPT and DALL-E 2, which generate textual content or a picture, respectively, in response to a immediate. FRIDA simulates how it will paint a picture with brush strokes and makes use of machine studying to judge its progress as it really works.

FRIDA’s ultimate merchandise are impressionistic and kooky. The brushstrokes are daring. They lack the precision sought so usually in robotic endeavors. If FRIDA makes a mistake, it riffs on it, incorporating the errant splotch of paint into the tip end result.

“FRIDA is a challenge exploring the intersection of human and robotic creativity,” McCann stated. “FRIDA is utilizing the type of AI fashions which were developed to do issues like caption photos and perceive scene content material and making use of it to this creative generative drawback.”

FRIDA faucets into AI and machine studying a number of instances throughout its creative course of. First, it spends an hour or extra studying the best way to use its paintbrush. Then, it makes use of giant vision-language fashions educated on large datasets that pair textual content and pictures scraped from the web, comparable to OpenAI’s Contrastive Language-Picture Pre-Coaching (CLIP), to grasp the enter. AI methods use these fashions to generate new textual content or photos primarily based on a immediate.

Different image-generating instruments comparable to OpenAI’s DALL-E 2, use giant vision-language fashions to provide digital photos. FRIDA takes {that a} step additional and makes use of its embodied robotic system to provide bodily work. One of many largest technical challenges in producing a bodily picture is lowering the simulation-to-real hole, the distinction between what FRIDA composes in simulation and what it paints on the canvas. FRIDA makes use of an concept generally known as real2sim2real. The robotic’s precise brush strokes are used to coach the simulator to replicate and mimic the bodily capabilities of the robotic and portray supplies.

FRIDA’s crew additionally seeks to handle a few of the limitations in present giant vision-language fashions by regularly refining those they use. The crew fed the fashions the headlines from information articles to offer it a way of what was occurring on this planet and additional educated them on photos and textual content extra consultant of numerous cultures to keep away from an American or Western bias. This multicultural collaboration effort is led by Zhixuan Liu and Beverley-Claire Okogwu, first-year RI grasp’s college students, and Youeun Shin and Youngsik Yun, visiting grasp’s college students from Dongguk College in Korea. Their efforts embody coaching knowledge contributions from China, Japan, Korea, Mexico, Nigeria, Norway, Vietnam and different nations.

As soon as FRIDA’s human consumer has specified a high-level idea of the portray they wish to create, the robotic makes use of machine studying to create its simulation and develop a plan to make a portray to attain the consumer’s objectives. FRIDA shows a coloration pallet on a pc display for a human to combine and supply to the robotic. Computerized paint mixing is at present being developed, led by Jiaying Wei, a grasp’s pupil within the Faculty of Structure, with Eunsu Kang, school within the Machine Studying Division.

Armed with a brush and paint, FRIDA will make its first strokes. Sometimes, the robotic makes use of an overhead digicam to seize a picture of the portray. The picture helps FRIDA consider its progress and refine its plan, if wanted. The entire course of takes hours.

“Folks marvel if FRIDA goes to take artists’ jobs, however the principle aim of the FRIDA challenge is sort of the alternative. We wish to actually promote human creativity by means of FRIDA,” Oh stated. “For example, I personally wished to be an artist. Now, I can really collaborate with FRIDA to precise my concepts in portray.”

Extra details about FRIDA is offered on its web site. The crew will current its newest analysis from the challenge, “FRIDA: A Collaborative Robotic Painter With a Differentiable, Real2Sim2Real Planning Atmosphere” on the 2023 IEEE Worldwide Convention on Robotics and Automation this Might in London. FRIDA resides within the RI’s Bot Intelligence Group (BIG) lab within the Squirrel Hill neighborhood of Pittsburgh.