Dec 232015
 

When I started piano lessons about 7 months ago, I started to find interesting parallels between writing code and playing piano. It shouldn’t be hard to imagine that one could be learning a programming language like a musician learns an instrument as well. There are a set of basic tools and principals. In piano, there is your basic syntax of reading music, tools such as arpeggios and scales, and principals like circle of fifths.

The first thing I did when I wanted to move into C++ development was write an open source library in C++ to better grasp the tools and principals of the language. By the time I was finished, I had exposure to a lot of the basic tools from the standard library and basic principals like type deduction, polymorphism, templates, and more “language generic” problems such as design patterns, separation of concern, clean coding, and unit testing.

As part of my journey to expose myself to as much piano as I could, I started watching a lot of youtube videos of pianists. Even if you don’t play piano or an instrument, I would suggest listening to this video, at least from minute 2:15 and onwards.

 

There is an incredible amount of parallels to learning a programming language but the following stuck with me:

Incorporating scales into your daily practice routine is the most efficient shortcut to technical mastery and brilliance, and it’s an indissoluble part of every great pianists journey.

– Ilinca Vartic

Now if Ilinca was a programmer maybe she would have worded it like this:

Incorporating functions from the standard library into your coding projects is the most efficient shortcut to technical master and brilliance, and it’s an indissoluble part of every great programmers journey.

Not only do you gain fundamental knowledge of the language you’re using, but you don’t have a write unit tests for it! At some point we all realize our wheel probably isn’t as good as the wheel from the standard library. Though there are many facets of becoming an expert at a programming language, I think having intimate knowledge of it’s standard library is important. It’s starting to become more important to me at least.

Dec 112015
 

A friend of mine who works for a seismic processing startup was chatting with me about a problem he had about with sending off commands to various machines which are supposed to process some data. Essentially wanting the ability to queue those machines for processing. Since the nodes don’t need to communicate any information, it’s not necessary to use anything fancy like MPI, though he didn’t have any built in mechanism to manage jobs. He was able to accomplish this task with a clever, though crude, combination of bash, cron, and top. Of course curiosity got the best of me wondering how this might be implemented and shortly thereafter a python cluster manager was born.

Since I’ve been focused on C++ and CUDA lately, I thought it would be a nice refresher to write a small python package which provides a more robust solution to this scheduling problem. It gave me a chance to review some simply threading and socket communications using the Python standard library along with experiment with a couple other packages. Moreover, the more I worked on it, the more I wanted to to do with it. I only wanted to dedicate one weekend of casual coding but this took about 3 such weekends to put together. I give you, clustermuster:

https://github.com/bee-rock/clustermuster

Though the server itself required ssh authentication with it’s various nodes, I haven’t included a secure authentication mechanism for sending commands to the manager apart from requiring an appropriate schema. Using it within your own network would be okay, provided you trust everyone on your network. If you plan on doing something like this in production, I’m sure there are libraries out there that would accommodate your needs. To be honest, the socket library that comes with the standard Python library is very easy to use incorrectly. Though it’s a great exercise to figure out how to use and to play with, the next time I write an application using sockets in Python, I would certainly consider using the Tornado framework, http://www.tornadoweb.org/en/stable/.

I’ll likely do a demo with it in an upcoming post.