With conference coming up soon in October (see you there we hope?) we had to carefully select our presentations to ensure that they remained educational and weren't just product showcases. One of the subjects that we therefore decided not to cover was creating scaleable distributed applications using our proprietary engine farm. Whilst the techniques involved are interesting we'd ultimately be promoting our proprietary technology and if you wanted a sales presentation you'd have asked for one, right? So we decided to create a blog article about it instead! The results are what you're now reading.

What is an engine farm?
An engine farm is a program that is designed to spin up and spin down a variable number of OpenEngines on a server to deal with incoming and outgoing transactions. As load volumes increase, more engines are brought into play and as demand decreases engines are retired. The farm also has to deal with engines crashing - be it through programmer error or operating system intervention, and has the ability to kill hung engines so that license count usage does not creep up until the system becomes disabled. The Sprezz engine farm was designed to be pluggable as we deal with a number of different clients (see later for a brief description of these) and as such it is currently dealing wih SFTP, FTP, HTTP, HTTPS and obviously SOAP for XML. It’s a native Windows Service controlled (normally) by a control panel app.

What is it used for?
Well the answer is, ultimately, whatever you want to use it for. In this section we'll look at a few of the uses that Sprezzatura's clients have put the engine farm to. The clients will have to remain anonymous for reasons of both commercial confidentiality and national security.

In one application the farm is used for maintaining a phone billing system and front end billing web site. Whilst one engine is downloading call data records using SFTP and then parsing them to create billing records another set of engines might be servicing web site requests to top up individuals phone accounts using HTTPS to do the credit card transactions.

In another application, localised engine farms at branch offices collect transactions from the local network and securely transmit these to another server farm which maintains a centralised database. This database is in turn published to the web from a secure site that is in turn run by a server farm. Data entered on the local systems is live on the national database and available for query within a second or two of entry.

Finally in another application an engine farm sitting on a local server consumes XML requests generated by an external entity and in turn publishes the requested XML back to the requester. As volumes increase more engines ensure that the system remains responsive.

Why not just use the OESocketServer?
Well naturally everyone is going to feel that their solution is the best solution but there's more to our answer than that. The major bullet points that we believe distinguish the Sprezz engine farm are as follows :-

• When the Sprezz engine farm was developed the socket server was not available
• The Sprezz engine farm runs as a native Windows service so configuration is easier as there is no reliance on the Java runtime being installed nor on using a 3rd party service host to run as a service.
• The Sprezz engine farm talks directly to engines via revCAPI not via a Java Native Interface so there is no performance loss from string marshalling
• As clients are written per installation it is possible to create highly secure connections without needing to configure a secure tunnel
• The control panel application makes configuration more understandable

What does it look like?
Well that's the million dollar question! What do you WANT it to look like? In all cases as the engine farm is a service it doesn't look like anything. The only thing that you'll see is the configuration window - be this an application or a control panel component. For the purposes of this blog post we'll use the phone billing application as the example.

Whilst the core engine farm remains the same throughout, the front end is modified per application. In this instance the configuration window for the engine farm is an executable, in other instances it is a control panel app.

When it's running as said previously it's a native Windows Service - so the only sign that it is running is an entry in the services list :-




In the Phone Billing app the configuration window looks like this :-


In the first subtab we specify how to log into the engine, and the procedure name utilised by the farm to pull down and price call records (SFTP) and the procedure name used by the farm to do the credit card billing (HTTPS).



In the second subtab we actually configure information about the engine farm itself. The prompts are relatively self explanatory but to deal with them in order :-

Engine Folder - where the engines should be launched from
Show Engines - whether the individual engines should be visible
Persistent Engines - whether the engines should "stay alive" after they have been launched. If not checked the engine will be closed as soon as it has completed the requested task.
Idle Timeout - if the engines are persistent how long must they be idle before they are closed?
Min Engines - the minimum number of engines to have alive at any one point
Max Engines - the maximum number of engines to have alive at any one point. This prevents the engine farm from taking ALL of the engines if there are local network users too.
RCL Mode - as with the Universal Driver - how should communication with the engines take place?
TCPIP Start Port - for when TCPIP is selected over Named Pipe
Pipe Prefix - if Names Pipes is used, what prefix should be used for the pipe name?



The final tab contains less used information. Again the prompts are relatively self explanatory but :-

Init Proc - the routine to run when an engine is first started
Final Proc - the routine to run before an engine closes
Execution Timeout - if an engine has not responded after this many minutes, kill it as it is assumed to have died
GC Interval - time between internal housekeeping events
Bind Engines - this tells the engine manager to generate a unique ID for an engine so they can be identified easily.




In this particular application we need to log onto an SFTP server and if the logon fails walk through a list of alternate servers. This table is used to configure this.



In the event of errors the service will mail the error details to a nominated email address. This is configured here.




For trouble shooting the service can be asked to keep a log on what it gets up to (this is in addition to the routine updating of the event log with errors that happens normally). The log contains details of all aspects of the service's operations rather than just error conditions.

A redacted sample of an error log follows :-

26/10/2010 11:17:34 0000016 6040 : E:\revsoft\OpenInsight\pbs_resources\zzPBSSvc\v1.3\zzPBSSvc.exe - v1.3.0.22
: ===========================================================================
:
26/10/2010 11:17:34 0000016 6040 : zzPBSSvc loaded log configuration settings:
: -> LogDirectory - E:\revsoft\OpenInsight\pbs_resources\zzPBSSvc\v1.3
: -> LogFile - zzPBSSvc.log
26/10/2010 11:17:34 0000016 6040 : zzPBSSvc loaded OEM configuration settings:
: -> APIPath - E:\revsoft\OpenInsight
: -> AppID - PBS
: -> UserID - PBS
: -> Password -
: -> HandlerProc - PBS_ZZPBSSVC
: -> InitProc - PBS_ZZPBSSVC_INIT
: -> FinalProc -
: -> PersistantEngines - 1
: -> MinEngines - 2
: -> MaxEngines - 3
: -> UseTCP - 0
: -> TCPStartPort - 54321
: -> PipePrefix - OEM
: -> ShowEngines - 1
: -> IdleTimeout - 10
: -> ExecTimeout - 10
: -> GCInterval - 5
: -> CallAttachCmd - 0
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc loaded SFTP configuration settings:
26/10/2010 11:17:34 0000047 6040 : -> SFTP Server - redact
: -> SFTP UserName - redact
: -> SFTP Password - redact
: -> SFTP Unbilled Path - redact/unbilled
26/10/2010 11:17:34 0000047 6040 : --------------------------
26/10/2010 11:17:34 0000047 6040 : -> SFTP Server - redact
: -> SFTP UserName - redact
: -> SFTP Password - redact/unbilled
26/10/2010 11:17:34 0000047 6040 : --------------------------
26/10/2010 11:17:34 0000047 6040 : -> SFTP Frequency - 5
26/10/2010 11:17:34 0000047 6040 : -> SFTP Download Path - E:\revsoft\OpenInsight\pbs\CDR Files\
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc loaded EMail configuration settings:
: -> Email To - support@sprezzatura.com
: -> Email From - SPBS_Service@redact
: -> SMTP Server -
: -> SMTP UserName - post.redact.net
: -> SMTP Password -
: -> SMTP Secure - 0
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc starting OE Manager...
26/10/2010 11:17:36 0001813 6040 : zzPBSSvc started OE Manager OK
26/10/2010 11:17:36 0001813 6040 : zzPBSSvc main thread entering processing loop...
26/10/2010 11:17:36 0001875 4124 : zzPBSSvc thread reports main PBS thread has finished initialization
: -> initError is 0
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc beginning Regular Payment check...
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc processing regular payments...
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc processed regular payments OK
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc ending Regular Payment check
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc beginning CDR check...
:
26/10/2010 11:17:36 0002016 6040 : ** EXCEPTION! **********************************
: ** Swapping to backup SFTP Server redact [Connection lost]
:
:
26/10/2010 11:17:36 0002282 6040 : ** EXCEPTION! **********************************
: ** Exhausted SFTP backup servers1 [Connection lost]
:
26/10/2010 11:22:36 0302391 6040 : zzPBSSvc beginning CDR check...
:
26/10/2010 11:22:37 0302485 6040 : ** EXCEPTION! **********************************
: ** Swapping to backup SFTP Server redact [Connection lost]
:
:
26/10/2010 11:22:37 0303375 6040 : ** EXCEPTION! **********************************
: ** Exhausted SFTP backup servers1 [Connection lost]
:
26/10/2010 11:25:16 0462344 4124 : zzPBSSvc received STOP request...
26/10/2010 11:25:16 0462344 4124 : zzPBSSvc handled STOP request OK
26/10/2010 11:25:16 0462344 6040 : zzPBSSvc main thread exiting processing loop...
26/10/2010 11:25:16 0462344 6040 : zzPBSSvc stopping OE Manager...
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc stopped OE Manager OK
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc engines statistics:
: -> TotalRequestCount - 1
: -> PeakRequestCount - 1
: -> PeakRequestsWaitingCount - 1
: -> PeakEngineCount - 2
: -> CreateEngineFailureCount - 0
: -> TerminatedEngineCount - 0
: -> RetiredEngineCount - 0
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc main thread is exiting ...
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread waiting for main PBS thread to exit
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread reports main PBS thread has finished
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread reports main PBS thread has been released
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread exiting OK

In conclusion
Engine farms aren't for everybody but when you want a robust, failsafe, self recovering engine farm service which can process pretty much anything and scale to meet your needs we have the technology that has been field proven over the past several years and we are confident it can meet your requirements. For further information or to request a quotation contact sales@sprezzatura.com.

Labels: engine farm, ftp, http, https, sftp

In a previous blog on Latent and Resolved Selects, we talked about how processing through the Basic+ reduce function and select statements were quicker than using the RLIST command. As hinted at in the prior article, the speed increase comes with additional maintenance required by the developer. In this part of our series on Selection, we'll talk about this maintenance, and the steps you need to take.

There are three areas that the developer needs to be aware of. The first is file resizing. The second is readnext termination. The third is cursor status, which will be covered in the next in this series of postings.

File Resizing

As most of you know, Revelation's Linear Hash Filing System (LH) is more accurately known as "Linear hashing with two partial expansions, separate chaining, distributed control and variable length records". It's the "distributed control" portion we're interested in here, since that's the part that manages the file resizing.

Without going into a detailed technical discussion about how and why the system chooses to resize the file, we know it happens without requiring any specific action by the user. The system checks for a resize on certain file access. When a file resizes, it generally interacts with only one or two groups. The records in these groups can be split and moved anywhere across the entire file. This means, when you read a record from a table, you can potentially trigger a resize.

Since there is no way of knowing where the records in the group will move to, it's possible that the records will move to a group with a lower group number. This means it will be positioned earlier in the file than the current position.

This is important, because when the system is processing a non-indexed select without an active key list, it starts at group 0 and moves through each group, examining each record. If the file resizes while this is happening, a record can be moved to a group that has already been processed, meaning the record will be skipped. Of course, the opposite can happen as well, in that a record that has already been processed can be moved forward to a group that hasn't been processed yet. This record can then potentially show up twice.

To avoid this problem, the RLIST function increases the sizelock by two when issuing a select.

• A sizelock of 0 means the file will resize normally, growing or shrinking as required.
• A sizelock of 1 means the file will only grow when required. It will never shrink.
• A sizelock of 2 or more means the file will never resize.

Increasing the sizelock by two ensures that the file will not resize out from under you while the select is being processed.

Since you are managing the select and readnext yourself, and not RLIST, you need to manually set the sizelock. This can easily be accomplished using the FIX_LH routine.



When you have finished processing the list, you must decrement the sizelock back.



It's important to remember that you should not decrement the sizelock until after you have completely finish processing the file. That means when you have finished processing the readNext loop, not just after you have issued the select statement.

ReadNext Termination

The second item to be aware of is readnext termination. When working with a latent list, the system continues to process the select in a non-terminating loop. To avoid this, you must use the AT (ascending terminating) or DT (descending terminating) options when issuing the readnext statement.

The extended ReadNext syntax is


where the "by" option can be the following literals or numbers.

Literal	Number	Direction
AT	0	Ascending Terminating
AN	1	Ascending Non-Terminating
DT	2	Descending Terminating
DT	3	Descending Non-Terminating

Note: Do not put the "by" option in quotes. It's a statement, not a passed string.

If you know the sort mode at compile time, you can use either the literal or the number.
If you will only know the sort mode at runtime, you must use the number, which you can assign to a variable.




Cursor Status

As mentioned above, cursor status and working with multiple cursors will be covered in the next post in the Select Series.

Labels: latent, reduce, resolved, select, sizelock

One of the nice features that AREV has that is missing from OpenInsight is the ability to change the datafile of a databound window on the fly by adding the modifier DATAFILE=NewDataFileName onto the sentence being used to launch the window. This was frequently used when multiple files shared the exact same structure - for example a "current" file and the associated archive files.

As we've mentioned we're working on a large AREV->OI conversion and we really missed this facility so in true Sprezz style we decided that we'd better implement it.

This proved to be easier than even we had anticipated due to the similarity between OI Window Common and AREV Window Common. The workaround we implemented is only good for windows which only update one file, though it wouldn't take much to make it suitable for multi-file windows. The following code shows how to implement it in a create event although of course if you have a promoted event framework you might choose to do it there.

Labels: AREV Behaviour, JoinMap, Window Common

A recent question on the Revelation Forum asked how to detect an ENTER keypress when using an EditTable as this is not exposed by the standard OpenInsight CHAR event (ENTER is one of the keys that the EditTable normally considers to be "reserved" like TAB and the Arrow keys).

Despite this limitation it is actually possible to detect an ENTER keypress with WM_KEYDOWN window message and the WINMSG event but ONLY if the EditTable is set to "Protected", i.e.




To trap the ENTER key we first need to tell OpenInsight that we want to be notified of any WM_KEYDOWN window messages that are sent to the EditTable by using the OpenInsight QUALIFY_EVENT message. This is normally done in the form's CREATE event:




Once we've done this we can write a WINMSG event handler for the EditTable that responds to the VK_RETURN keycode:




(Note in these examples we are using some insert records from our WinAPI Library that you can find here)

Labels: EditTable, EditTable Cookbook, OpenInsight