The summer has almost ended and so has my OPW internship @ Apache Mesos .
I had a great time working on my project and it was a good learning experience for me.
In this blog post I will summarize my work on the project, generically called ‘Slave unregistration in Mesos‘. I will also talk about the valuable things I have learned and my overall impression. This will be quite a long post, so I will brake it into sections, in case you want to skip reading some parts:
1. What I have done (technical aspect)
2. What I have learned (non-technical)
3. Thanks (non-technical)
Technical aspect – What I have done during the summer
The initial purpose of the project was to drain (immediately), on demand, mesos slaves. Draining immediately a slave implies killing all its underlying tasks/jobs. The draining should start when the slave receives a SIGUSR1 signal. Before shutting down, the slave should unregister with the master so that the master knows as soon as possible that the slave won’t be available any more.
Usually operators need to drain a slave, when they need to do some maintenance work on it (i.e. like upgrading the kernel). Sometimes draining immediately is not a good practice, because as I said above, all tasks running within the slave will be instantly killed. Some tasks might be very important or some rules might exist (outside of mesos) which will be violated if a task gets killed. This is why the drain now mechanism which I implemented initially, was not enough for doing maintenance.
A very important thing I have to mention is that mesos does not currently have a build in mechanism for maintenance (i.e. something like notifying frameworks that some slaves will undergo maintenance so that they don’t schedule new tasks on those slaves and if possible, move the existing tasks on other slaves). Operators have their own framework specific tools, which they use before doing maintenance on slaves. This does not scale well because on a multi-framework system, you need custom tools for each framework.
Since the SRE engineers are the ones that do all the maintenance, we had a meeting with two of them. We discussed about their current methods of preparing a slave machine for maintenance and what they would need from mesos in order to make their lives easier.
The conclusion was that we need to add support for the following:
1. “deactivate slaves” -> stop sending resource offers for the deactivated slaves. The result is that frameworks will not launch new tasks on those slaves.
2. send “inverse offers” to frameworks -> the master is requesting that the frameworks return resources in a particular amount of time. If the resources are not returned in time, they are are forcibly revoked.
So I started working on “deactivate slaves”. This had two parts:
1. creating a HTTP endpoint which should receive HTTP requests to activate/deactivate certain slaves and should pass them to the back end application (master) to do the necessary work.
2. the real activation/deactivation part which had to consider that the master might reboot, so the state of the slave (activated/deactivated) had to be written on the disk.
Usually, when working on a large application, if you want to add a more complex thing, you have to write a design documentation first and after that share it with the community. This is because two (more) brains think better than one so you need to get feedback from others as soon as possible (they might expose problems you didn’t consider).
My mistake was that I didn’t do that. I almost finished the implementation for the “deactivate slaves” part when I started to write the fist design document, called ‘Maintenance Primitives’. Here’s the link to it.
So, after sharing the document with a couple of engineers and getting some comments, we had two more meetings. The conclusion of those meetings was that ‘slave deactivation’ is not really ok. This is because, if a framework really needs to run a task on a slave and the only resources left are the ones from a deactivated slave, you have a problem (you can not run a new task on a deactivated slave). So we decided to have a different approach:
1. do not deactivate slaves; instead, schedule maintenance on them, by inserting some sort of metrics in offers on which the schedulers (frameworks) can sort upon. This way, offers belonging to slaves scheduled for maintenance will have, let’s say a lower metric, and will be less preferred. So the consequence will be that frameworks will launch new tasks on the slaves scheduled for maintenance only if there’s no other option.
2. add “inverse offers” support; also here we’ll need to deactivate slaves so eventually, my patches (for ‘deactivate slave’) will be useful.
3. unschedule maintenance -> make the slave active again and send offers with no metric.
I wrote another design document for this. I didn’t share it with the community yet because it still needs some adjustments, but I will link it here, if you want to get more details about the ‘maintenance primitives’.
So the next step is to share the document with the community and SRE engineers and get feedback from everybody so that we can adjust the document and have a final version. Afterwards the implementation part should begin.
Other than this, I worked on minor fixes, issues and code refactoring. Here’s all my reviews.
What I have learned from this experience
- Before starting something, get feedback as soon as possible from people. Don’t forget that more brains think better than one!
- Be persevering / Keep trying whatever you want to accomplish in life! Before this OPW internship I applied at GSOC two times (once I was very close to be accepted!) and one time at OPW. Didn’t matter for me, I said to myself that I will keep trying until they will accept me and that’s what I did! Doesn’t matter how much you fail, in the end, when you’ll win, you’ll forget about all those failures and just enjoy the win. And this experience proved me that my philosophy was right!
- The only stupid question is the one not asked! Most of the time I ask something after I spent at least a couple of hours searching the answer and didn’t find it. This is because I’m afraid people might think I’m not well prepared and I’m asking trivial things. This is not true, nobody thinks anything of that, and even if they do think, who cares?! The most important thing is that you do your work in the least amount of time (so spending hours and hours for finding something, instead of asking a question and find the answer in 10 seconds, is not the right choice!).
- Be ok (accept) if you don’t understand something / don’t know something. If you participate in a discussion and you don’t understand what people are talking about don’t blame yourself because of that! Instead, ask them to give you some background . It’s not your fault you don’t know! You can’t know everything in the world, right?!
This is the last and the most important section. I want to thank my mentor Ben Mahler who has coordinated and helped me during the internship. I don’t want this to look cheesy but I really appreciate that he had the patience to review all my patches, my documents, read my emails and answer all my questions. I can tell that every comment and every review was made with the same patience and earnestness. There were times when I was asking myself “What does this guy think after reviewing my 20 lines patch for the 20th time ?!” haha :-)) . He was very supportive and encouraged me all the time and never put any kind of pressure on me.
I so much appreciate people that are very dedicated to what they are doing and Ben is that kind of person.
I would also like to thank the GNOME Foundation for creating this program in order to encourage women from the tech industry. This is such a tough field for women because the proportion men vs. women is so unbalanced (much more men than women). So a lot of women feel like they don’t belong here, that they are not smart enough, that this is a place for guys and so on. All these thoughts basically discourage them to enter into the tech field and even if they do enter, those kind of thoughts will keep them from standing out and excel.
Alex's tech soup 