I signed up with XS4ALL last week. It wasn't the cheapest ISP in Amsterdam, but they have managed to stay true to their original "hacker culture". It's good to see that their owner KPN also tries to challenge this blockage.
When I lived with a former Airbus engineer, he explained that these systems were running 80486 processors due to their relability (in the sense that nothing unexplainable would ever happen). That being said, he also told stories of how the complexity caused software bugs to manifest themselves that often disappeared before their causes were determined.
Well, if "CERN is considering ØMQ as a control infrastructure for Large Hadron Collider." and they manage petabytes of data, also "There are trading firms using ØMQ as a secret weapon in low-latency arms race." I think it could be well suited for machine learning.
It can be used to stitch together low latency and high throughput distributed systems. Think of it as transport lego blocks, it lets you connect different parts of your system. They support a number of common messaging patterns : request-response, publish-subscribe, etc.
But it stops there (and that is the beauty of it). It doesn't prescribe or make assumptions about the types of messages that are sent. Those are just byte blobs as far it it is concerned. You can choose to represent your messages as json, protobuf, xml, raw video frames.
So can you use it for HPC? It is hard to tell, sure, it is like asking if hammer can be used to be fix a car. Yeah, it can, in some cases, but it is just a good versatile tool, that can be used for more than that.
That depends more on whether your algorithm is easily distributed. If it is, zeromq will be good for you.
I am currently using it to transfer model parameters learned on machine #1 to machine #2 where real time prediction takes place. The predictions are then sent out via 0mq to a robot controller.
The great thing of building systems with 0mq is that you just add another component without having to think too much about 'but then I will have to implement a protocol for it and HTTP will probably to slow and the request/reply pattern doesn't really fit there blabla'.
Looks very interesting! I cannot find any details about its release though, even though the article talks about September 19. Will it be open source? proprietary?
I used it to build some simple topologies to parallelize collecting statistics over huge data sets. I never got around to doing dynamic programming using a 0mq topology, but I wrote up some fun plans for it.