Latest CryptDB

Hi everyone,

I was thinking it would be better for us to discuss the current published version of CryptDB rather than the one we have on our class page. This is not much different from the one we already have except that it also discusses security at the proxy and the user levels which is interesting too. Except for this, everything the new paper covers is the same as that of the one we have on our page. Please refer to the following link to gain access to the latest copy. I would be presenting this paper to the class on Thursday. Please kindly make a note of it.

http://people.csail.mit.edu/nickolai/papers/raluca-cryptdb.pdf

-Uday

Mesos: A middle man between cluster & frameworks

Motivation
In industry, there is a requirement to run multiple instances of Hadoop(production, test) on same data set. Mesos framework fulfills this kind of requirements. Mesos makes easy to share clustered resources like, CPU, RAM or hard disk space across distributed frameworks.

As shown in below figure, Mesos acts a common resource sharing layer to all frameworks.

Short overview of Mesos design
Mesos master allocates resources on slave nodes to cluster frameworks. Mesos allocates resources at the level of tasks using allocation module. Framework can reject/accept the resource offers. Slave nodes also provide performance isolation between frameworks using platform isolation modules (ex: Linux container).  To use Mesos framework, cluster frameworks need to implement Executor, Scheduler interfaces.

pros
- This framework enables to use cluster resources efficiently.
- The design of Mesos framework is flexible enough to add new frameworks

cons
1) Performance of the job can't be guaranteed as resource allocation may delay the whole execution.
2) Mesos provides resource isolation using external OS libraries(ex: linux container). This may have some additional overhead.
3) It would have better if Mesos handles resource allocation efficiently instead of letting each framework to work on.
4) Mesos can be much more capable in providing common concerns like security etc across frameworks but the paper has not discussed any details.
5) Synchronization issues may arise when data is shared across different frameworks. Paper hasn’t addressed this issue.

Current state of Mesos
Mesos project is being incubated in Apache. Mesos has got recognition from big companies like Twitter, Facebook and yahoo.
-Twitter uses apache mesos to run analytics job.
-Facebook uses mesos to run production and test hadoop instances on same data

Similar frameworks
NextGen MapReduce handles process execution and resource scheduling (this system is called YARN)  in generic layer and MapReduce is being implemented on top of this.
In one sentence, difference between YARN and mesos is, mesos is a Meta framework scheduler whereas YARN is an application scheduler.

References:
1) http://www-users.cselabs.umn.edu/classes/Fall-2012/csci8980-2/papers/mesos.pdf
2) http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html

Everyone's Email Address

Please respond to this post by providing your email address so that everyone could be added as the blog author

Good presentations and blogs

Guys, I am enjoyed *all* the presentations and blogs thusfar, and the discussions raised in class. If you haven't spoken out in class yet, please try to do so. I wanted to call your attention to two exemplary presentation/blogs. Presentation: Prabhu's talk was excellent. He motivated the problem carefully, described the interesting technical innovations in the paper, and gave critical comments. He also gave a snappy talk that fell well within the 30 minutes. There is no substitute for knowing the paper inside-and-out. Learn from it. Blog: Ankit's blog was a beautiful piece of work. It contained a nice discussion of the paper, draw comparisons to things we had already read, and give a slew of pro and con statements. Wow!

Talk

Check out this talk on Monday: https://mail.google.com/mail/?shva=1#inbox/139fda3ae17ce48c