cURL works while accessing GitHub /search/users API but using restTemplate.exchange returns zero users

Tags: , , , ,



I have defined a RestTemplate backed HttpClient to call Github API to search users

My method is this

public List<User> fetchPublicInformation(String firstName, String lastName, String location) {

    final HttpHeaders headers = new HttpHeaders();
    if (token != null && token.length() != 0) {
        headers.set("Authorization", "bearer " + token);
    }
    headers.set("'User-Agent'", "request");
    HttpEntity<String> entity = new HttpEntity<String>(headers);

    synchronized (this) {

        StringBuilder uri = new StringBuilder(GITHUB_SEARCH + "users?q=fullname:");
        if (!firstName.isEmpty()) {
            uri.append(firstName.trim().toLowerCase());
        } else {
            firstName = " ";
        }
        if (!lastName.isEmpty()) {
            uri.append(" " + lastName.trim().toLowerCase());
        } else {
            lastName = " ";
        }
        if (location != null && !location.isEmpty()) {
            uri.append("+location:" + location.trim().toLowerCase());
        }
        System.out.println(uri.toString());
        ResponseEntity<GitHubUsersResponse> response = null;
        response = template.exchange(uri.toString(), HttpMethod.GET, entity, GitHubUsersResponse.class);
        return response.getBody().getItems();
    }
}

This method hits the URI

https://api.github.com/search/users?q=fullname:shiva tiwari+location:bangalore

and return [] as Items(part of response Body)

while if I use the same URI with cURL it gives me four responses.

I am unable to find my fault.

Answer

investigating with OP in the comments, we found out that he was not using the same url of java with curl, so he was getting different results.

he was running this command:

$ curl https://api.github.com/search/users?q=fullname:shiva tiwari+location:bangalore
{
  "total_count": 1230,
  "incomplete_results": false,
  "items": [
    {
      "login": "Shiva108",
      "id": 13223532,
...

which produces an output that contains several objects inside the “items” array while with java code he was getting an empty “items” array.

the space character ' ' in the url is crucial! the shell uses spaces to separate arguments of commands and they need to be properly escaped when they are inside arguments.

the url OP was using with curl was in fact just https://api.github.com/search/users?q=fullname:shiva, the last part was being interpreted as another argument to curl (and produced also an error curl: (3) URL using bad/illegal format or missing URL) while java was using the full url including the surname and location filter.

a literal space ' ' inside an url is also an illegal character and needs to be encoded using + or %20 (see Percent-encoding), in fact, if we use quotes ' to escape the space in the shell, we get “400 Bad Request”:

$ curl -v 'https://api.github.com/search/users?q=fullname:shiva tiwari+location:bangalore'
...
< HTTP/1.0 400 Bad Request
...

but with proper space encoding we get the same result as java (empty “items” array):

$ curl 'https://api.github.com/search/users?q=fullname:shiva+tiwari+location:bangalore'
{
  "total_count": 0,
  "incomplete_results": false,
  "items": [

  ]
}
$

(I’m pretty sure java code handles space encoding automatically)



Source: stackoverflow