OPW Pencils Down

The summer has almost ended and so has my OPW internship @ Apache Mesos .
I had a great time working on my project and it was a good learning experience for me.

In this blog post I will summarize my work on the project, generically called ‘Slave unregistration in Mesos‘. I will also talk about the valuable things I have learned and my overall impression. This will be quite a long post, so I will brake it into sections, in case you want to skip reading some parts:
1. What I have done (technical aspect)
2. What I have learned (non-technical)
3. Thanks (non-technical)

Technical aspect – What I have done during the summer

The initial purpose of the project was to drain (immediately), on demand, mesos slaves. Draining immediately a slave implies killing all its underlying tasks/jobs. The draining should start when the slave receives a SIGUSR1 signal. Before shutting down, the slave should unregister with the master so that the master knows as soon as possible that the slave won’t be available any more. Continue reading

Advertisements

‘drain now’ mechanism

My OPW project has several parts (I talked about them on my previous post here). I started my work with the first part which is called “the drain now” mechanism. This mechanism aims to drain the slaves when SIGUSR1 signal is sent to them.

‘draining’ in this context means killing an entity (process) and all that it’s running underneath that entity (all the processes that were started by that entity). So ‘draining the slave’ means killing the slave processes and all the jobs/tasks the slave is running.

For the first part of the mechanism, I implemented a signal handler which is triggered when SIGUSR1 signal is issued. The handler kills everything that runs underneath the slave and after that shuts down the slave.

For the second part of the mechanism, I adjusted the first part by  sending an unregister request message from the slave to the master, before shutting down the slave. This is because normally, the master waits up to the health checking delay (~75 seconds) until it considers the tasks that were running on the slave as lost.  With this change, the master will mark the tasks as lost (and do all the necessary things when a task becomes lost) as soon as it will receive the unregister request message. In addition, the master will remove the slave from its lists.

At a first glance all this seems pretty easy to do, but things tend to get a bit complicated when you’re working on a big project and lots of companies relay their infrastructure on it (including Twitter!!). Every piece of code that you add has to be perfect so that the application remains efficient and without bugs. So my patch had a couple of review iterations until it was ready to be committed. My mentor (Ben Mahler) helped me a lot with reviewing all my code and giving me tips.

So as a learned lesson : ‘keep it simple’. When you have something to do, even if it looks easy, think about it twice, maybe there’s even an easier way you could do it.

Thanks for reading and have a great day 🙂 !

“Slave unregistration in Mesos” – brief description

The project I will be working on is called “Slave unregistration in Mesos”. Before going into more detail about it, I will briefly explain Mesos’ architecture, so that my project will make more sense.

It is said that a picture values a million words, so here’s a picture which describes Mesos’ architecture (source here).

architecture3

 

As you can see in the picture, Mesos has a distributed architecture, with several entities:

Master
There is only one leading master per cluster; for high availability there can be more masters but only one is active

Slaves
These are hosts in the datacenter and one slave daemon is running on each host; these hosts are the ones on which all the tasks are run

Zookeeper
It is responsible for the leader election between the masters and also is used by Mesos slaves to discover the master

Frameworks
These are applications that are doing analytic things (Hadoop, Spark, Aurora, MPI) which run their tasks on top of Mesos.
Multiple frameworks can run on the same Mesos cluster because Mesos provides good resource isolation through Linux containers.
A framework consists of two parts:

  • scheduler
    – is responsible for scheduling jobs/tasks
    – it receives resource offers (memory, cpu, disk) from the master, which are available in the cluster, and it will use them to launch tasks
  • executor
    – is responsible to execute the tasks the scheduler wants to launch on the slave
    – it is launched by the slave when a task is launched by the scheduler on the slave

Initial motivation of the project

Sometimes a Mesos cluster has hundreds or thousands of slave machines. From time to time, some of them need maintenance (eg. upgrade the system). Up until now, when an operator wanted to do some maintenance work on a slave, he manually connected on that slave and killed the slave process. This approach will however, leave all tasks running on the machine. So a mechanism is needed that will drain the slave (kill all the tasks and the slave daemon).

Here comes in hand the first part of my project which consists of two parts:

  1.  the ability to kill all the tasks that were running within the slave and after that shutdown the slave daemon, on demand.
  2. the problem with (1) is that the master will wait up to the health checking delay (~75 seconds) to notify the framework that the tasks were lost. This is why, before shutting down, the slave will send an unregistration message to the master.

Basically the two items above represent the ‘drain now’ mechanism, because the slave will be drained immediately, when the signal is triggered.

The problem with the ‘drain now’ mechanism is that it basically kills all the jobs that were running on the slave which sometimes is not desired. This is because some of them are long term jobs which take a lot of time and resources to be rescheduled and get to the point where they had been before the slave was shutted down. Therefore the followings will also be implemented:

  1.  the ability to deactivate slaves. This means that the master will no longer send resource offers belonging to that slave. As a result, frameworks can no longer launch new tasks on the slave. The tasks that had been launched before the slave was deactivated will continue running.
  2. the ability to ‘inverse offers’ from frameworks, which means that the master is requesting the frameworks to return the resources they were using,  in a particular amount of time; if the resources are not returned in time, they will be forcible revoked.

So this is the description of my project. Please stay tuned for updates.

 

Thanks for reading and have a great day 🙂 !

 

Init

Hello everybody! This is a technical blog and it will  mainly be about my progress on a project I am currently working on.

ImageAt the end of April (2014) I found out that I was accepted at ‘Outreach Program for Women‘ (short OPW). I was very happy (and still am) about this because now, I have the opportunity to contribute to the open source community. I have done some contributions to several open source applications before, but they were small fixes and I always wanted to do something significant, with greater impact.

First, let me introduce what OPW is. In short terms, it’s kinda’ like GSOC but for women. In long terms, it’s an internship program organized by the GNOME Foundation, where participants can contribute to open source applications. You can participate only if you are a woman; if not, there’s GSOC which offers a lot of opportunities as well. Also, OPW doesn’t restrict you in being a student so feel free to apply no matter what age you are.

Basically the purpose of this program is to encourage women to contribute to open source. In the past, the number of females that were actively involved in FOSS communities was very low, so they definitely needed more support and encouragement from organizations.

If you are not a programmer, don’t worry, OPW offers non programming projects as well, that include user experience design, graphic design, documentation, web development, marketing, translation, etc.

For more information, please check out their website here.

The project I will be working on is for Apache Mesos, a cluster management framework. Mesos is used by many companies(including Twitter) that have large clusters and want a better and easier management for them.

For more information, please check out their website here.

So, this is all for the introduction post. For more information about my progress on the project, please stay tuned.

Thanks for reading and have a great day 🙂 !