Thursday, 28 February 2013

Connecting Streambase to Redis

I've been working on writing my first input and output adapters to Redis from Streambase
The standard Java library seems to be Jedis and there is a good  example of use of Jedis for the pub sub
approach. It's pretty easy to use.

I've only got a windows server to put the Redis server on, but following Microsoft's OpenTech effort
I go a Redis 2.4 server up and running fairly easily

The behaviour I want is a cache of the last value.
e.g something publishes values to Redis
A client connects, it can get the last value published and then any updates

An approach is each publish event in a Redis pipeline sets a key's value and also publishes to a channel.
A client on connecting subscribes to the channel and picks up the key's value, thereby getting any updates.

The subscription for Jedis is a blocking subscription, but it's not clear how safely to add further subscriptions to the same connection One approach is the first subscription is to a control channel, all further requests are sent via Redis to the subscription and in the OnMessage routine you had further subscriptions
This seems elegant though a bit wasteful of trips etc. (i.e. you have to have the check for a control message in the OnMessage handler, and the addition of a subscription requires a trip to Redis and back) But as one subscribes and unsubscribes rarely it's not a problem.

For the initial release I've just pSubscribed to the * pattern to pick up all the messages - I can filter them later in Streambase. The sample input and output adapter is pushing happily 4900 messages and sets a second even using the ide which is probably more than enough for what I'm trying to do, even without any clumping of the messages to pipeline them or send them with protocol buffers.
I'll submit it to the Streambase component exchange, but it's also on github
I've also written Excel addins to publish and receive - the use case here was to have cached monitoring centrally of Streambase - the current Streambase add in for excel is in the process of being rewritten and you'd still need to implement the caching somehow.









No comments:

Post a Comment