Training AI to Paint with Code

Beautiful. I am reminded of a paper I was inspired by a long time ago [0], (okay it's 2018 so I guess just 8 years ago, but it feels like longer, from the before-times), that demonstrated learning brush strokes. At the time there was already a lot of work on GANs, but these are pixel-based methods, and I was really interested in the idea of how to derive descriptive methods of scene generation/understanding. I found this work really interesting because it combined RL and GAN techniques in a creative way. I miss that kind of research. Now of course VLMs have shown that you can mix modalities in generalized sequence-to-sequence problems and it doesn't surprise me that this kind of thing is possible, but it's so nice to see it done well using modern techniques. [0] https://proceedings.mlr.press/v80/ganin18a.html

Thank you for sharing this! did not expect to find my work on HN randomly haha

A fun diversion and general capabilities test that I like to do is to ask the models to "programmatically assign color and alpha values to a png to generate an image. it should be a painting or whatever subject interests you in the moment. 800x800" or something along those lines, and they'll make a "generative" art piece for you. For example first time I tried this Claude Fable used Python to generate a transparent PNG of a quartz crystal with inclusions that were variable depending on the seed number. So anyway yeah it's surprisingly easy to get the frontier models nowadays to make cool art!

I think this actually might be one of the best ways to train people to use AI. I can see this honing people's prompting abilities and expressiveness, along with constraints and desired outcome. Wild the possibilities

Really awesome! Been thinking about how to get LLMs to do generative art (yes, the pre-AI definition of generative art). Love to see this approach and results!