Thursday, 28 February 2013

Excel addin to and from Redis Implementation

I've completed a working version of the Excel add-ins, both the RTD receive Excel add in and the publish addin to connect Excel to Redis now working.
That enables any excel files to share a value, and also pick up the latest value if they can connect to the
Redis server. The point of this for me was that they'd monitor some control values from Streambase,
which are also posted to Redis.

Takeaways - Redis is really nice and easy to access - most of my testing was via Ruby and the Redis library  which are very simple to use. Add in Express hides a lot of the idiosyncrasies of writing Excel add-ins, in this instance with Delphi. The OmniThead libraries are great, I still find it relatively easy to get things working with Delphi and Streambase makes it pretty simple to write event servers. Again another nice Redis library, Jedis, makes it a pleasure.

I used the OmniThread libraries to fire off the thread and their non locking queues
I'm using the Add In express vcl libraries which are good and their support is excellent.
Running on my four non hyper threaded cores (xeon 5150 2.66 GHz) machine
it seems happy to send and receive over 4000 updates a second - given Microsoft's stats there's clearly some scope for improvement, but that may be enough for the current project.
I've put my code on GitHub and a compiled versions on Mega
Further work could include some auditing (who sent the updates), sample handling of other data structures,
more robust error handling and notifications (keys and values with spaces in them don't really work)
I've also forked Nils Achterholt's delphi-redis adding a couple of methods for the publication and subscription
 

Connecting Streambase to Redis

I've been working on writing my first input and output adapters to Redis from Streambase
The standard Java library seems to be Jedis and there is a good  example of use of Jedis for the pub sub
approach. It's pretty easy to use.

I've only got a windows server to put the Redis server on, but following Microsoft's OpenTech effort
I go a Redis 2.4 server up and running fairly easily

The behaviour I want is a cache of the last value.
e.g something publishes values to Redis
A client connects, it can get the last value published and then any updates

An approach is each publish event in a Redis pipeline sets a key's value and also publishes to a channel.
A client on connecting subscribes to the channel and picks up the key's value, thereby getting any updates.

The subscription for Jedis is a blocking subscription, but it's not clear how safely to add further subscriptions to the same connection One approach is the first subscription is to a control channel, all further requests are sent via Redis to the subscription and in the OnMessage routine you had further subscriptions
This seems elegant though a bit wasteful of trips etc. (i.e. you have to have the check for a control message in the OnMessage handler, and the addition of a subscription requires a trip to Redis and back) But as one subscribes and unsubscribes rarely it's not a problem.

For the initial release I've just pSubscribed to the * pattern to pick up all the messages - I can filter them later in Streambase. The sample input and output adapter is pushing happily 4900 messages and sets a second even using the ide which is probably more than enough for what I'm trying to do, even without any clumping of the messages to pipeline them or send them with protocol buffers.
I'll submit it to the Streambase component exchange, but it's also on github
I've also written Excel addins to publish and receive - the use case here was to have cached monitoring centrally of Streambase - the current Streambase add in for excel is in the process of being rewritten and you'd still need to implement the caching somehow.









Friday, 15 February 2013

Delphi object references

I was wondering whether a selection of objects in a program were in fact the same object as they were supposed to be.
As far as I understand an object points to the actual instance of the object, i.e. the value of an object is a reference to the memory where the actual object is.
A simple console app to clarify this is below, giving the results

Object1     >7036359053610256
Object1 loc >4341464
Object2    >7036359021690880
Object2 loc>4341468
set Object2 to Object1
Object2    >7036359053610256
Object2 loc>4341468

Object1 and Object2 have two different addresses. 4341464 and 4341468
However on setting object1 to object2 the contents at those memory addresses is the same
7036359053610256 which points to the exact same instance of an object.





program testObjRef;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils,windows;
var
object1 : TObject;
object2 : TObject;


  function GetObjectReference(thisObject : TObject) : string;
  var
  p: Pointer;
  begin
    p := @thisObject;
    Result := intToStr(uint64(p^));
  end;


begin
  try
    { TODO -oUser -cConsole Main : Insert code here }
  object1 := TObject.Create;
  writeln('Object1     >' + GetObjectReference(object1));
  writeln('Object1 loc >' + IntToStr(uint64(@object1)));
  writeln('Object2    >' + GetObjectReference(object2));
  writeln('Object2 loc>' + IntToStr(uint64(@object2)));
  writeln('set Object2 to Object1');
  object2 := object1;
  writeln('Object2    >' + GetObjectReference(object2));
  writeln('Object2 loc>' + IntToStr(uint64(@object2)));



  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.