Image editing model training is fascinating. One method for training image editing models involves using a second model to apply the inverse of the change you want the model to learn. Typically, the task you’re asking the second model to perform is easy, whereas the inverse task is difficult.
For example, you might ask the second model to cover the person’s face with a black square; a VLM model notes that the person is a man with brown hair and round glasses. Then, during training, the resulting image is presented along with the prompt, “Remove the black square from the man’s face. He has brown hair and round glasses.”
The model now learns how to remove black squares and replace them with a man’s face with brown hair and round glasses.
Since the training data is easily synthesized using existing models, you can generate enormous amounts of it - often very cheaply. For specialized editing tasks, this technique is really powerful. Build your training set for your special purpose task, fine tune an existing image editing model such as Qwen Image Edit to produce a new checkpoint or LoRA (often a LoRA is more than good enough) and then you have a special purpose model to perform whatever narrow editing task you need it to perform on your image data.
Are these models built atop models that already understand natural language?
If the commands all follow the same syntax, it's easy to imagine how you can generate a good training set.
But how to they fully grasp natural language to be able to perform tasks worded unexpectedly, which would be easy to parse, if they understood natural language?
"But how to they fully grasp natural language to be able to perform tasks worded unexpectedly, which would be easy to parse, if they understood natural language?"
A Large Language Model. Pardon me for spelling out the full acronym, but it is what it is for a reason.
I think a lot of the whiz-bang applications of LLMs have drowned it out, but LLMs are effectively the solution to the long-standing problem of natural language understanding, and that alone would be enough to make them a ground-breaking technology. Taking English text and translating it with very high fidelity into the vector space these models understand is amazing and I think somewhat underappreciated.
Yes, the newer image and video editing models have an LLM bolted onto them. The rich embeddings from the LLM are fed into a diffusion transformer (DiT) alongside a tokenized version of the input image. These two streams “tell” the model what to do.