Swarm of XeBees

• Tweet
• Sharebar

• Tweet

The problem:

In our project we use four load-balanced tomcat nodes(with Sticky-Session enabled). Everything works fine but there is just one thing that bugs me. Whenever we do a production deployment, we remove one node, deploy on that and, if everything works fine, we remove second node from the load-balancer and deploy on that and so on. The problem is that whenever we remove a node during deployment, a session expired is thrown to all the users working on the node we just removed, as he/she is redirected to another node.

It is not a huge problem in our case. But this could be serious in e-commerce domain. For example if a user is filling his/her shopping cart and the session expires for no reason.

The configuration:

I deployed a simple Spring App in two Tomcat instances. Apache Webserver as the load balancer. Apache's mod_jk is used for load balancing using sticky-sessions. A single Memcached Server is used as an external key value server for storing sessions. I used xMemcachedClient to interact with Memcahed Server, because it is easily available as a pom dependency and is provided by Google.

The solution:

Load-Balancing:

I used this guide to load-balance my application using Apache Webserver and mod_jk. Below is my config:

LoadModule jk_module modules/mod_jk.so
JkWorkersFile conf/jkworkers.properties
JkLogFile logs/mod_jk.log
JkLogLevel "info"
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"
JkMount /app* loadbalancer

And below is the content of my jkworkers.properties file:

# Define list of workers that will be used for mapping requests
worker.list=loadbalancer,status

# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8009
worker.node1.host=localhost
worker.node1.type=ajp13
worker.node2.ping_mode=A
worker.node1.lbfactor=1 

# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=7009
worker.node2.host=localhost
worker.node2.type=ajp13
worker.node2.ping_mode=A
worker.node2.lbfactor=1

# Load-balancing behavior
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=1
# Status worker for managing load balancer
worker.status.type=status

Managing Sessions:

To manage session failover I used the following approach:

• Both the tomcats hold sessions locally on their respective jvm.
• Additionally all the sessions(of both tomcats) are stored in Memcached server.(Cache expiry in memcached can be configured, I have configured it to be 1day)
• Suppose a user is browsing on node1. In general, he will be directed to node1 as sticky-sessions are enabled and his session can be fetched from the local memory. In case of node1 fails, the next request will go to the other tomcat instance i.e. node2. node2 is asked for a session he does not know. So, node2 will fetch the session from the Memcached Server and store it in its local jvm. Now, node2 is responsible for this session.

Now lets see some code.

package in.xebia.vijay.controllers;
//import statements
@Controller
public class HomeController {
	
	private static final String SEPARATOR = "|";
	private static final Logger logger = LoggerFactory.getLogger(HomeController.class);
	@Resource
	private SessionService sessionService;
	
	@RequestMapping(value = "/login", method = RequestMethod.POST)
	public String home(Model model, User user,HttpSession session, @CookieValue("JSESSIONID") String jsessionId) {
		logger.info("In home!");
		sessionService.set(session.getId()+SEPARATOR+"user", user, session);
		User userFromSession = (User) sessionService.get(jsessionId+SEPARATOR+"user", session);
		model.addAttribute("userName", userFromSession.getUserName());
		return "home";
	}
	@RequestMapping(value = "/check", method = RequestMethod.GET)
	public String check(Model model, HttpSession session, @CookieValue("JSESSIONID") String jsessionId) {
		logger.info("In Check!");
		User userFromSession = (User) sessionService.get(jsessionId+SEPARATOR+"user", session);
		model.addAttribute("userName", userFromSession.getUserName());
		return "check";
	}
	@RequestMapping(value = "/", method = RequestMethod.GET)
	public String login() {
		logger.info("In login!");
		return "login";
	}
}

The lines to note in above code are 14,15 and 22:

• In line 14, I am adding the current user to session using SessionService. Point to note is that instead of doing something like :

  sessionService.set("user", user, session);

  I have written :

  sessionService.set(session.getId()+SEPARATOR+"user", user, session);

  I am "deliberately" appending the SessionId just to ensure that if a particular tomcat node fails, the other tomcat can retrieve failed node's sessions using the cookie JSESSIONID stored in the client header.

• And that is the reason, in lines 15 as well as 22, I am fetching session attribute from the SessionService using the JSESSIONID retrieved from the cookie.

All the other code is quite self explanatory. Now lets have a look at the SessionServiceImpl.java

package in.xebia.vijay.impl;
//import statements
@Service
public class SessionServiceImpl implements SessionService {

	@Autowired
	private MemcachedClient memcachedClient;
	private static int EXPIRY = 24 * 60 * 60;

	@Override
	public <T> void set(String key, T value, HttpSession session) {
		try {
			session.setAttribute(key, value);
			memcachedClient.set(key,EXPIRY,value);
		} catch (TimeoutException e) {
			e.printStackTrace();
		} catch (InterruptedException e) {
			e.printStackTrace();
		} catch (MemcachedException e) {
			e.printStackTrace();
		}
	}

	@SuppressWarnings("unchecked")
	@Override
	public <T> T get(String key, HttpSession session) {
		try {
			if(session.getAttribute(key)!=null)
				return (T) session.getAttribute(key);
			else if(memcachedClient.get(key)!=null){
				String[] keyPart = key.split("|");
				String newKey = session.getId()+"|"+keyPart[1];
				session.setAttribute(newKey, memcachedClient.get(key));
				memcachedClient.set(newKey, EXPIRY, memcachedClient.get(key));
				memcachedClient.delete(key);
				return memcachedClient.get(newKey);
			}
			else
				return null;
		} catch (TimeoutException e) {
			e.printStackTrace();
		} catch (InterruptedException e) {
			e.printStackTrace();
		} catch (MemcachedException e) {
			e.printStackTrace();
		}
		return null;
	}

}

Points to be noted :

• In the implementation of method

  public  void set(String key, T value, HttpSession session); 

  we add the attribute to HttpSession as well as the Memcahced Server, so that in case of any node failure the other node can fetch the values from Memcached Server.

• In the implementation of method

  public  T get(String key, HttpSession session); 

  we first check the local session if the attribute is found, well and good simply return with the value. Else we check the Memcahed Server for the key provided if some value is found we store that attribute in the current session and replace the value in Memcached Server with new key(which is appended with the session id of current tomcat).

Below I am attaching the bean definition for xMemcachedClient:

<beans:bean id="memcachedClient" class="net.rubyeye.xmemcached.XMemcachedClient">
		<beans:constructor-arg type="java.lang.String"  value="localhost"/>
		<beans:constructor-arg  type="int" value="11211"/>
</beans:bean>

Nothing special just passing the host and port to initialize the memCachedClient.

In case you want to run the code on your own machine here is the link to Github Repository.

Conclusion:

In this blog, we learned how to manage session-failover on load-balanced tomcat nodes with sticky-session enabled using Memcached Server.
Happy reading.

PS : The Spring App created is quite naive and is only for the purpose of demonstration. You can obviously have a different implementation using the same concept.

Filed under: memcached, Spring