Top 3 Server Configuration Issues and Possible Solutions

If you are reading this chances are that you are somehow involved in web server application development. Either you are a QA, developer, someone from the operations or a little bit of everything. It doesn’t matter to what kind of server architecture or programming language you are exposed to, dealing with HTTP is almost always a must. It’s arguably the most used application layer communications protocol on the web right now.

While writing core web services for API testing product Apimation the author has encountered a number of odd server behaviors which can lead to some very serious issues in production. Here is a list of TOP 3 nastiest, unwanted behaviors and their possible causes:

1. Unexpected EOFs on client side

End of file can occur when server unexpectedly closes connection socket. Some part of the response on client side is never received. So the question is: why server sometimes does not allow client to receive the full content and closes the connection? The answer is directly related to connection management on server side. In HTTP/1.X connections can be categorized in several models: short lived, persistent and pipeline connections. Short lived and persistent connections from client side are managed with Connection and Keep-Alive headers. If Connection header is set to close, connection policy will be short lived which means the connection opening procedure will be done each time the request is sent and it will be closed after the response has been received. If connection needs to be reused for more than one request Connection header can be set to something other than “close” and Keep-Alive header is used to set idle connection timeouts. In HTTP 1.1 persistent connection are there by default and usually developers do not add Connection and Keep-Alive headers. However, it might be a problem if server application or HTTP gateway server forces the connections to be closed after the response is received. In this case the preconditions for EOF issue is in place.

So imagine client is opening a TCP connection with the server and sending “request A”. Upon receiving “response A” client is fast enough to send out following “request B”, but on the server side, due to the connection force close policy, it terminates the connection which in turn fails “response B” reception on the client side.

To deal with EOF issues be sure to check any connection management policies on all server side applications that are involved in dealing with incoming requests. Configurations on gateway servers, code side server configurations and such.

2. HTTP streaming errors

This is similar to the previous issue but mostly occurs when client is sending a large body request. It happens during the request being sent out. Depending on http server implementation the error messages vary, but almost always will be thrown as HTTP streaming errors. Reason for this is also due to connection being closed. What is the cause this time?

When configuring client or the server it is possible to set request or response timeout settings respectively. These timeout settings can be set to control when the sending or receiving side should quit, in case some parts of HTTP messages take too long to get to the other side.

Server side configuration settings used for this is “ReadTimeout”, “ReadHeaderTimeout” etc. If read timeouts on server side are OK, be sure to check with client side teams for write timeouts which could be the culprit. If the issue still persists check that there are no limitations on maximum amount of bytes that can be received or sent.

3. Connection reset by peer

This is one of the most unwanted server behaviors. Client is refused to connect to the server and consequently send out any requests. This is due to the server not being able to receive any new connections. Once these kind of issues are encountered it is almost impossible to solve them without getting rid of active connections or in the worst case – regularly restarting the service.

Reasons for this can be found both in the system and server application limitations. Unix system limitations regarding new connections can be found in file descriptor limits. If file descriptor limits are reached, server will not be able to open any new connections. Check any articles related to that to know which system settings need to be changed.

Another reason for connection resets is the application connection pool limitations. Be sure to check HTTP server or web framework settings related to connection pool.

If the issue still is occurring, with all the limits set to high, it might help to go over timeout and idle connection configurations. If clients never close a persistent connection, server could be left with hanging connections. Which is always a big risk when getting connection reset issues. Hanging connections could also pose a risk to memory leaks.

Conclusion

Whenever you’re dealing with HTTP servers, make sure you know everything about all the inner moving parts. If it is a framework or a library that you are using, be sure to check if there are any open issues that might create a problem for your web service. Always thoroughly read library and framework documentation regarding custom HTTP server configurations. Do not skip parts that you do not understand yet, learn them instead. Only choose a well maintained and popular library for HTTP server communications. Never rely on HTTP clients to behave well, so make sure to cover that risk on the server side. Be sure that your service is always 200 OK and never 503 Unavailable!