nodejs mongodb and android studio slow fetch speed

I have a system which, pulls data from my server and stores it to a mobile SQL database via android studio. It works but it is painful slow like 30mins. I have around 86000 records in my database and want to pull all of them out of the server. What is the best way to do this?

Presently I get the count from the server and then query the server database until i find each ID and then send that result back to my mobile app.

app.post("/get_data", function(req, res)
{
    var Id_request = req.body.Id_request;//The requested ID

    var query = {Val_String : Id_request};//Query value
    //console.log(query);
    //Data.find({}, function(err, result) {//Finds all data
    Data.findOne(query, function(err, result) {//Finds all data
    if (err) {
      //res.status(400).send(err);
      console.log("Sending error");
      res.status(200).send();
    } else {
      return res.json(result);
    }
  });
});

I use a recersive function in my pull request for each ID

private void call_method()
    {
        HashMap<String, String> map = new HashMap<>();
        map.put("Id_request", Integer.toString(data_pointer));//The ID value
        Call<Fetch_result> call = retrofitInterface.executeGet_data(map);//Run the post
        call.enqueue(new Callback<Fetch_result>() {
            //call.enqueue(new Callback<Fetch_result>() {
            @Override
            public void onResponse(Call<Fetch_result> call, Response<Fetch_result> response) {
                if (response.code() == 200)//Successful login
                {
                    D1= response.body().getT1_String();
                    D2= response.body().getT2_String();
                  
                    data_pointer = data_pointer + 1;
                    boolean result = BLE_DB.addData_Downloaded(D1,D2);//Add data
    
                    if(data_pointer<= Total_entries) {//Call method again
                        call_method();//Recursive here
                    }else if (data_pointer > Total_entries){
                        Utils.toast(getApplicationContext(),"All data received");
                    }

                } else if (response.code() == 404) {
                    Utils.toast(getApplicationContext(), "Get data fail");
                }
            }

            @Override
            public void onFailure(Call<Fetch_result> call, Throwable t) {
                Utils.toast(getApplicationContext(), "Get data error");
            }
        });
    }

How Can I speed this up or do it differently to speed it up?

Answer

Try to fetch as much data as possible at once ( limit the amount of queries you do ). It is hard to tell you how since I don’t know your monogDB collection.
Try to do this with as little requests as possible. If you can return all the fetched data at once, this will save you some time.
JSON may be very slow when doing it on 86000 documents
Consider caching the data for future users

Right now i suspect what is limiting you is the fact that you are doing 86000 queries to the db… If you can get the entire mongoDB collection it may be a bit faster( look at notes )

Notes: https://docs.mongodb.com/manual/reference/method/db.collection.find/#db-collection-find ( omit the query parameter will result in fetching the entire collection )

Advertisement

Answer