What are the best practices for calculating statistics in REST API applications

I am trying to make a service that will calculate statistics for each month.

I did smth like this:

public Map<String, BigDecimal> getStatistic() {
        List<Order> orders = orderService.findAll(Sort.by(Sort.Direction.ASC, "creationDate")).toList();
        SortedMap<String, BigDecimal> statisticsMap = new TreeMap<>();
        MathContext mc = new MathContext(3);
        for (Order order : orders) {
            List<FraudDishV1Response> dishesOfOrder = order.getDishIds()
                    .stream()
                    .map(dishId -> dishV1Client.getDishById(dishId))
                    .collect(Collectors.toList());
            BigDecimal total = calculateTotal(dishesOfOrder);
            String date = order.getCreatedDate().format(DateTimeFormatter.ofPattern("yyyy-MM"));
            statisticsMap.merge(date, total, (a, b) -> a.add(b, mc));
        }
        return statisticsMap;
    }

But it takes a long time if there are lots of etries in the database. Are there any best practices for working with statistics in REST API applications?

And also I’d like to know if it is a good way to save the statistics in a separate repository? It will save time for calculating statistics, but during creating a record in the database, you will also have to update the statistics db.

Answer

Well, I did’t stop and made several solutions step by step…

Step 1: Use streams. Before that, calculating statistics for 10,000 OrderEntities records took 18 seconds. Now it has accelerated to 14 seconds.

Step 2: Using parallelStream instead of streams. Parallel streams accelerated the calculation of statistics to 6 seconds! I was even surprised.

public SortedMap<String, BigDecimal> getStatisticsByParallelStreams() {
        List<OrderEntity> orders = new ArrayList<>();
        orderService.findAll(Sort.by(Sort.Direction.ASC, "createdDate")).forEach(orders::add);

        MathContext mc = new MathContext(3);
        return orders.stream().collect(Collectors.toMap(
                order -> order.getCreatedDate().format(DateTimeFormatter.ofPattern("yyyy-MM")),
                order -> calculateTotal(order.getDishIds()
                        .parallelStream()
                        .map(dishId -> dishV1Client.getDishById(dishId))
                        .collect(Collectors.toList())),
                (a, b) -> a.add(b, mc),
                TreeMap::new
        ));
    }

Step 3: Optimizing requests to another microservice. I connected the JProfiler to the app and I have found out that I offen do extra requests to the another microservice. After it firstly I made a request to receive all Dishes, and then during calculating statistics, I use a recieved List of Dishes. And thus I speeded it up to 1.5 seconds!:

public SortedMap<String, BigDecimal> getStatisticsByParallelStreams() {
        List<OrderEntity> orders = new ArrayList<>();
        orderService.findAll(Sort.by(Sort.Direction.ASC, "createdDate")).forEach(orders::add);

        List<FraudDishV1Response> dishes = dishV1Client.getDishes();

        MathContext mc = new MathContext(3);
        return orders.stream().collect(Collectors.toMap(
                order -> order.getCreatedDate().format(DateTimeFormatter.ofPattern("yyyy-MM")),
                order -> calculateTotal(order.getDishIds()
                        .parallelStream()
                        .map(dishId -> getDishResponseById(dishes, dishId))
                        .collect(Collectors.toList())),
                (a, b) -> a.add(b, mc),
                TreeMap::new
        ));
    }