Netty writeAndFlush() incomplete

My tcp client request netty server, and netty server use writeAndFlush() return 393718 bytes. But the client only receive 262142 bytes. I use “tcpdump -A” to apturing packets, less than 393718 too. So I think the properble is in netty writeAndFlush() function?

Here is the code

tcp server:

public static void main (String args[]) {

        int processorsNumber = Runtime.getRuntime().availableProcessors() * 3;
        ThreadFactory threadFactory = new DefaultThreadFactory("work thread pool");
        EventLoopGroup bossLoopGroup = new NioEventLoopGroup(1);
        EventLoopGroup workLoopGroup = new NioEventLoopGroup(processorsNumber, threadFactory, SelectorProvider.provider());

        try {
            ServerBootstrap sb = new ServerBootstrap();
            sb.group(bossLoopGroup , workLoopGroup)
                    .channel(NioServerSocketChannel.class)
                    .option(ChannelOption.SO_BACKLOG, 128)
                    .childOption(ChannelOption.SO_KEEPALIVE, true)
                    .childHandler(new ChannelInitializer<NioSocketChannel>() {
                        @Override
                        protected void initChannel(NioSocketChannel ch) throws Exception {
                            //ch.pipeline().addLast(new LineBasedFrameDecoder(1000));
                            ch.pipeline().addLast(new TcpServerHandler());
                        }
                    });

            sb.bind(new InetSocketAddress("0.0.0.0", 18881)).sync().channel().closeFuture().sync();
        } catch (InterruptedException e) {
            logger.error("netty error", e);
            e.printStackTrace();
        } finally {
            bossLoopGroup.shutdownGracefully();
            workLoopGroup.shutdownGracefully();
        }
    }

server handler:

public void channelRead(ChannelHandlerContext ctx, Object msg) {

        ByteBuf in = (ByteBuf) msg;
        String cmd = in.toString(CharsetUtil.UTF_8);
        logger.info(cmd);

        String retCode = "";
        String file = "E:\sz\app.log";
        try {
            FileReader fr = new FileReader(file);
            BufferedReader br = new BufferedReader(fr);
            String buffer = null;
            while ((buffer = br.readLine()) != null) {
                retCode += buffer;
            }
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

        logger.info("return：" + retCode);
        byte[] result = retCode.getBytes();
        logger.info("======================="+result.length);
        ByteBuf resultBuf = Unpooled.buffer(result.length);
        resultBuf.writeBytes(result);
        ctx.writeAndFlush(resultBuf);

        ctx.close();
    }

tcp client:

public static void main(String[] args) {
        StringBuilder response = new StringBuilder();
        try {
            Socket socket = new Socket();
            InetSocketAddress address = new InetSocketAddress("127.0.0.1", 18881);
            socket.connect(address, 3000);
            socket.setSoTimeout(10 * 1000);

            OutputStream clientRequest = socket.getOutputStream();
            String cmd = "cmd";
            clientRequest.write(cmd.getBytes());
            clientRequest.flush();

            InputStream clientResponse = socket.getInputStream();
            int maxLen = 1024;
            byte[] contextBytes = new byte[maxLen];
            int readLen;
            while ((readLen = clientResponse.read(contextBytes, 0, maxLen)) != -1) {
                response.append(new String(contextBytes, 0, readLen));
            }

            clientRequest.close();
            clientResponse.close();
        } catch (IOException e) {
            logger.error("tcp error", e);
            e.printStackTrace();
        }
        String result = response.toString();
        System.out.println(result);
        System.out.println(result.getBytes().length);
    }

Answer

You close the connection directly after writing, in a async framework, this is asking for trouble.

What is happening here, is that a new incoming message arrives in your channelRead by a Netty thread. This message is then progressed through your handler, and finally, it’s being sent back using writeAndFlush, however, this call of sending the data back happens partly in the background, and so the code continues on forward, then it sees the call to close the connection, it happily does that at this point, since you told it.

What happens at this point is a race condition between the TCP layer trying to send the data, and the TCP layer receiving the call to close the connection, because of the buffers used in different parts of the layers, it happens so that the limit is at 262142 bytes, however this limit may fluctuate if unknown aspects change of the connection.

A solution for this problem is waiting till Netty says the write is successful, and only then close it. Since we don’t want to wait in a async application, this means we must register a listener, luck is on our side this case, and netty provides us with a listener that just closes the channel for us. You can use it like this:

ctx.writeAndFlush(resultBuf).addListener(ChannelFutureListener.CLOSE);