0

How would one use the Dall-E text-to-image API's image and mask parameters in Unity C#?

For background, something like the following works for the other parameters like prompt (full code on GitHub):

string apiMode = "generations";
string apiUrl = "https://api.openai.com/v1/images/" + apiMode;

UnityWebRequest www = UnityWebRequest.Post(apiUrl, "");
www.SetRequestHeader("Content-Type", "application/json");
www.SetRequestHeader("Authorization", "Bearer " + key);

string jsonString = JsonConvert.SerializeObject(aiParams, Formatting.None, serializerSettings);

www.uploadHandler = new UploadHandlerRaw(System.Text.Encoding.UTF8.GetBytes(jsonString));
www.downloadHandler = new DownloadHandlerBuffer();
yield return www.SendWebRequest();

When using apiMode "edits" or "variations" though, as per Dall-E's documentation, it will return an API error suggesting to switch to content-type "multipart/form-data". How would one use this and add the binary data for image or mask, given some byte[] for the png? Thanks!

Rubén
  • 34,714
  • 9
  • 70
  • 166
Philipp Lenssen
  • 8,818
  • 13
  • 56
  • 77

1 Answers1

1

UnityWebRequest.Post -> overload that takes List<IMultipartFormSection> will automatically use this content type header and there you can provide MultipartFormFileSection and MultipartFormDataSection sections.

The example is unfortunately useless but it would probably look somewhat like e.g.

 var jsonString = JsonConvert.SerializeObject(aiParams, Formatting.None, serializerSettings);
 var jsonBytes = Encoding.UTF8.GetBytes(jsonString);
 var formParts = new List<IMultipartFormSection>();
 formParts.Add(new MultipartFormDataSection("data", jsonBytes, "application/json"));
 formParts.Add(new MultipartFormFileSection("image", yourFileContentBytes, "YourFileName.png", "image/png"));

 using (UnityWebRequest www = UnityWebRequest.Post("https://www.my-server.com/myform", formParts))
 {
     www.SetRequestHeader("Authorization", "Bearer " + key);
     yield return www.SendWebRequest();

     if (www.result != UnityWebRequest.Result.Success)
     {
         Debug.Log(www.error);
     }
     else
     {
         Debug.Log("Form upload complete!");
     }
 }
derHugo
  • 83,094
  • 9
  • 75
  • 115
  • 1
    Thanks! Strange, Dall-E now replies with "'image' is a required property", even though I've used `formData.Add(new MultipartFormFileSection("image", aiParams.image, "image.png", "image/png"));` and added that formData in the Post request, and the `byte[]` image is filled with content. (I've had the same Dall-E reply in the past when I simply didn't have any `image` param.) – Philipp Lenssen Nov 07 '22 at 11:35
  • Sorry, ignore the previous, I'm now further ahead after rewriting more parts to your pattern. I'm now at the Dall-E return message "Additional properties are not allowed ('data' was unexpected)", so I guess now I have to figure out under what name to pass the json (I picked "data"), and as last resort, pass the values one by one as formParts. – Philipp Lenssen Nov 07 '22 at 11:44
  • 1
    @PhilippLenssen usually afaik `data` and `files` would be the most common one .. I'm actually impressed that the Dall-E API is so verbose and tells you exactly what they expect :D Glad you could figure it out – derHugo Nov 07 '22 at 12:25
  • I ended up only using JSON serialization when using their non-image API endpoints, and listing the few parameters one by one otherwise ([new GitHub code for reference](https://github.com/JPhilipp/AIConnectors/blob/main/ImageAI/DallE/ImageAIDallE.cs)). The code is less streamlined than I would like because it tries to have a somewhat unified approach when switching between Dall-E, StableDiffusion and Replicate.com-StableDiffusion, even though all 3 APIs are really doing it differently. In any case, all works now, thanks! – Philipp Lenssen Nov 07 '22 at 12:29
  • 1
    @PhilippLenssen I'm getting the "image is required property" response too in my POSTMAN and API call via Swift. Any leads on how you debugged that? – AnupamChugh Nov 16 '22 at 11:48
  • @AnupamChugh You can find my resulting and working Unity Dall-E API code [here on GitHub](https://github.com/JPhilipp/AIConnectors). Maybe that can help you. Note `image` should only be needed if you use the API modes `edits` or `variations`, but not when using `generations`, so try the latter if that's the one you want. – Philipp Lenssen Nov 16 '22 at 15:24