Jsoup HTTP POST with payload

Question

I am trying to make this HTTP request via jsoup as given here:

http://api.decarta.com/v1/[KEY]/batch?requestType=geocode

And here is my code for that:

String postUrl = postURLPrefix + apiKey + "/batch?requestType=geocode";
String response = Jsoup.connect(postUrl).timeout(60000).ignoreContentType(true)
        .header("Content-Type", "application/json;charset=UTF-8")
        .method(Connection.Method.POST)
        .data("payload", jsonPayload.toString())
        .execute()
        .body();

jsonPayload.toString() gives this:

{
  "payload": [
    "146 Adkins Street,Pretoria,Pretoria,Gauteng",
    "484 Hilda Street,Pretoria,Pretoria,Gauteng",
    "268 Von Willich Street,Centurion,Centurion,Gauteng",
    ...
  ]
}

Which is a perfectly valid JSON.

However, jsoup each time returns HTTP status code 400 (malformed).
So, how do I send proper HTTP POST with JSON payload using jsoup if this is possible at all? (Please note that it's payload and not an ordinary key-value pair in URL)

Santosh Hegde · Answer 1 · 2017-01-04T15:28:59.963

7

Use latest JSOUP library.

If you are using maven, then add below entry to pom.xml

<dependency>
        <groupId>org.jsoup</groupId>
        <artifactId>jsoup</artifactId>
        <version>1.10.2</version>
</dependency>

And below code will solves your problem.

String postUrl=postURLPrefix+apiKey+"/batch?requestType=geocode";
            System.out.println(postUrl);
            String response= Jsoup.connect(postUrl).timeout(60000).ignoreContentType(true)
                    .method(Connection.Method.POST)
                    .requestBody("payload",jsonPayload.toString())
                    .execute()
                    .body();

edited Jan 04 '17 at 15:28

answered Jan 04 '17 at 15:00

Santosh Hegde

3,420
10
35
51

1

Based on the latest 1.11.3 version, we need to use ".method(Method.POST)", and ".requestBody(jsonPayload.toString())". – Veeresh Devireddy Nov 10 '18 at 18:42
.header("Accept", "application/json") .header("Content-Type", "application/json") is also needed! – VISWESWARAN NAGASIVAM Dec 15 '19 at 07:54

Alkis Kalogeris · Accepted Answer · 2014-12-13T20:54:07.960

What you need is to post raw data. That functionality has been implemented but it hasn't been added yet. Check this pull request https://github.com/jhy/jsoup/pull/318 . Do you really need to use jsoup for this? I mean you could use HttpURLConnection (this is what jsoup uses underneath) to make the request and then pass the response to jsoup as a string.

Here is an example of HttpURLConnection taken (but simplified and added json/raw data) from www.mkyong.com

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;

public class Main {

    public static void main(String[] args) {

        try {

            String url = "http://www.google.com";

            URL obj = new URL(url);
            HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
            conn.setReadTimeout(5000);
            conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
            conn.addRequestProperty("User-Agent", "Mozilla");
            conn.addRequestProperty("Referer", "google.com");

            conn.setDoOutput(true);

            OutputStreamWriter w = new OutputStreamWriter(conn.getOutputStream(), "UTF-8");

            w.write("SOME_JSON_STRING_HERE");
            w.close();

            System.out.println("Request URL ... " + url);

            int status = conn.getResponseCode();

            System.out.println("Response Code ... " + status);

            BufferedReader in = new BufferedReader(new InputStreamReader(
                    conn.getInputStream()));
            String inputLine;
            StringBuffer html = new StringBuffer();

            while ((inputLine = in.readLine()) != null) {
                html.append(inputLine);
            }

            in.close();
            conn.disconnect();

            System.out.println("URL Content... \n" + html.toString());
            System.out.println("Done");

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

This is just an example. It can and should be cleaned up. The error handling is non-existent. I left it like this on purpose in order to be as simple as possible. — Alkis Kalogeris, Dec 13 '14 at 20:55
Fantastic minimal answer for how to natively do this without use of a library like JSoup. Worked perfectly for me the first time I tried it. +1 — Muhammad Abdul-Rahim, Mar 23 '16 at 22:52

Mahozad · Answer 3 · 2021-12-03T11:30:19.900

This is how I did that with jsoup v1.14.3 and Kotlin (also applies to Java).
It sends two parameters name and age as part of a JSON object (assuming the server accepts that):

val document = Jsoup.connect("the/target/url")
    .userAgent("Mozilla")
    .header("content-type", "application/json")
    .header("accept", "application/json")
    .requestBody("""{"name": "Katy", "age": 43}""")
    .ignoreContentType(true)
    .post()

You can also send the parameters in url-encoded format (assuming the server accepts that):

val document = Jsoup.connect("the/target/url")
    .userAgent("Mozilla")
    .header("content-type", "application/x-www-form-urlencoded")
    .header("accept", "application/json")
    .data("name", "Katy")
    .data("age", "43")
    // OR
    //.data(mapOf("name" to "Katy", "age" to "43"))
    .ignoreContentType(true)
    .post()

score 0 · Answer 4 · answered Sep 10 '22 at 01:23

Example

    private fun token(email: String, password: String): String {
        val url = "https://asdf.com/tokens"

        val headers = mapOf(
            "Accept" to "application/json",
            "Content-Type" to "application/json",
            "Authorization" to "null"
        )
        val json = mapOf(
            "auth_type" to "CREDENTIAL",
            "credential_type_payload" to mapOf(
                "email" to email,
                "password" to password,
            ),
        ).toJson()

        return CrawlUtil.post(url, headers, json)["token"] as String
    }

CrawlUtil

import org.jsoup.Connection
import org.jsoup.Jsoup

object CrawlUtil {
    private const val TIMEOUT = 5 * 60 * 1000
    private const val MAX_BODY_SIZE = 0
    private const val IGNORE_CONTENT_TYPE = true

    private fun connection(
        url: String,
        headers: Map<String, String>,
        json: String,
        data: Map<String, String>? = emptyMap()
    ): Connection {
        var connection = Jsoup.connect(url)

        headers.forEach {
            connection = connection.header(it.key, it.value)
        }

        data!!.forEach {
            connection = connection.data(it.key, it.value)
        }

        return connection
            .timeout(TIMEOUT)
            .maxBodySize(MAX_BODY_SIZE)
            .ignoreContentType(IGNORE_CONTENT_TYPE)
            .requestBody(json)
    }

    fun get(
        url: String,
        headers: Map<String, String>,
        json: String,
        data: Map<String, String>? = emptyMap()
    ): Map<String, Any> {
        return connection(url, headers, json, data)
            .get()
            .text()
            .toObject()
    }

    fun post(
        url: String,
        headers: Map<String, String>,
        json: String,
        data: Map<String, String>? = emptyMap()
    ): Map<String, Any> {
        return connection(url, headers, json, data)
            .post()
            .text()
            .toObject()
    }
}

ExtensionUtil

inline fun <reified T> String.toObject(): T {
    return JacksonUtil.toObject(this)
}

JacksonUtil

import com.fasterxml.jackson.core.type.TypeReference
import com.fasterxml.jackson.databind.DeserializationFeature
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.SerializationFeature
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule
import com.fasterxml.jackson.datatype.jsr310.deser.LocalDateDeserializer
import com.fasterxml.jackson.datatype.jsr310.deser.LocalDateTimeDeserializer
import com.fasterxml.jackson.module.afterburner.AfterburnerModule
import com.fasterxml.jackson.module.kotlin.KotlinModule
import java.time.LocalDate
import java.time.LocalDateTime
import java.time.format.DateTimeFormatter

object JacksonUtil {
    val objectMapper: ObjectMapper = ObjectMapper()
        .registerModule(KotlinModule.Builder().build())
        .registerModule(JavaTimeModule())
        .registerModule(AfterburnerModule())
        .configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false)
        .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
        .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
        .registerModule(
            JavaTimeModule().apply {
                addDeserializer(LocalDate::class.java, LocalDateDeserializer(DateTimeFormatter.ISO_DATE))
                addDeserializer(
                    LocalDateTime::class.java,
                    LocalDateTimeDeserializer(DateTimeFormatter.ISO_DATE_TIME)
                )
            }
        )

    fun toJson(obj: Any?): String {
        if (obj == null) {
            return ""
        }

        return objectMapper.writeValueAsString(obj)
    }

    fun merged(a: MutableMap<String, Any>, b: Any): Map<String, Any> {
        return objectMapper
            .readerForUpdating(a)
            .readValue(toJson(b))
    }

    inline fun <reified T> toObject(s: String): T {
        return objectMapper.readValue(s, object : TypeReference<T>() {})
    }
}

https://seunggabi.tistory.com/entry/Jsoup-GET-POST-crawling

Jsoup HTTP POST with payload

4 Answers4

Example

CrawlUtil

ExtensionUtil

JacksonUtil

Linked