With conference coming up soon in October (see you there we hope?) we had to carefully select our presentations to ensure that they remained educational and weren't just product showcases. One of the subjects that we therefore decided not to cover was creating scaleable distributed applications using our proprietary engine farm. Whilst the techniques involved are interesting we'd ultimately be promoting our proprietary technology and if you wanted a sales presentation you'd have asked for one, right? So we decided to create a blog article about it instead! The results are what you're now reading.

What is an engine farm?
An engine farm is a program that is designed to spin up and spin down a variable number of OpenEngines on a server to deal with incoming and outgoing transactions. As load volumes increase, more engines are brought into play and as demand decreases engines are retired. The farm also has to deal with engines crashing - be it through programmer error or operating system intervention, and has the ability to kill hung engines so that license count usage does not creep up until the system becomes disabled. The Sprezz engine farm was designed to be pluggable as we deal with a number of different clients (see later for a brief description of these) and as such it is currently dealing wih SFTP, FTP, HTTP, HTTPS and obviously SOAP for XML. It’s a native Windows Service controlled (normally) by a control panel app.

What is it used for?
Well the answer is, ultimately, whatever you want to use it for. In this section we'll look at a few of the uses that Sprezzatura's clients have put the engine farm to. The clients will have to remain anonymous for reasons of both commercial confidentiality and national security.

In one application the farm is used for maintaining a phone billing system and front end billing web site. Whilst one engine is downloading call data records using SFTP and then parsing them to create billing records another set of engines might be servicing web site requests to top up individuals phone accounts using HTTPS to do the credit card transactions.

In another application, localised engine farms at branch offices collect transactions from the local network and securely transmit these to another server farm which maintains a centralised database. This database is in turn published to the web from a secure site that is in turn run by a server farm. Data entered on the local systems is live on the national database and available for query within a second or two of entry.

Finally in another application an engine farm sitting on a local server consumes XML requests generated by an external entity and in turn publishes the requested XML back to the requester. As volumes increase more engines ensure that the system remains responsive.

Why not just use the OESocketServer?
Well naturally everyone is going to feel that their solution is the best solution but there's more to our answer than that. The major bullet points that we believe distinguish the Sprezz engine farm are as follows :-

• When the Sprezz engine farm was developed the socket server was not available
• The Sprezz engine farm runs as a native Windows service so configuration is easier as there is no reliance on the Java runtime being installed nor on using a 3rd party service host to run as a service.
• The Sprezz engine farm talks directly to engines via revCAPI not via a Java Native Interface so there is no performance loss from string marshalling
• As clients are written per installation it is possible to create highly secure connections without needing to configure a secure tunnel
• The control panel application makes configuration more understandable

What does it look like?
Well that's the million dollar question! What do you WANT it to look like? In all cases as the engine farm is a service it doesn't look like anything. The only thing that you'll see is the configuration window - be this an application or a control panel component. For the purposes of this blog post we'll use the phone billing application as the example.

Whilst the core engine farm remains the same throughout, the front end is modified per application. In this instance the configuration window for the engine farm is an executable, in other instances it is a control panel app.

When it's running as said previously it's a native Windows Service - so the only sign that it is running is an entry in the services list :-




In the Phone Billing app the configuration window looks like this :-


In the first subtab we specify how to log into the engine, and the procedure name utilised by the farm to pull down and price call records (SFTP) and the procedure name used by the farm to do the credit card billing (HTTPS).



In the second subtab we actually configure information about the engine farm itself. The prompts are relatively self explanatory but to deal with them in order :-

Engine Folder - where the engines should be launched from
Show Engines - whether the individual engines should be visible
Persistent Engines - whether the engines should "stay alive" after they have been launched. If not checked the engine will be closed as soon as it has completed the requested task.
Idle Timeout - if the engines are persistent how long must they be idle before they are closed?
Min Engines - the minimum number of engines to have alive at any one point
Max Engines - the maximum number of engines to have alive at any one point. This prevents the engine farm from taking ALL of the engines if there are local network users too.
RCL Mode - as with the Universal Driver - how should communication with the engines take place?
TCPIP Start Port - for when TCPIP is selected over Named Pipe
Pipe Prefix - if Names Pipes is used, what prefix should be used for the pipe name?



The final tab contains less used information. Again the prompts are relatively self explanatory but :-

Init Proc - the routine to run when an engine is first started
Final Proc - the routine to run before an engine closes
Execution Timeout - if an engine has not responded after this many minutes, kill it as it is assumed to have died
GC Interval - time between internal housekeeping events
Bind Engines - this tells the engine manager to generate a unique ID for an engine so they can be identified easily.




In this particular application we need to log onto an SFTP server and if the logon fails walk through a list of alternate servers. This table is used to configure this.



In the event of errors the service will mail the error details to a nominated email address. This is configured here.




For trouble shooting the service can be asked to keep a log on what it gets up to (this is in addition to the routine updating of the event log with errors that happens normally). The log contains details of all aspects of the service's operations rather than just error conditions.

A redacted sample of an error log follows :-

26/10/2010 11:17:34 0000016 6040 : E:\revsoft\OpenInsight\pbs_resources\zzPBSSvc\v1.3\zzPBSSvc.exe - v1.3.0.22
: ===========================================================================
:
26/10/2010 11:17:34 0000016 6040 : zzPBSSvc loaded log configuration settings:
: -> LogDirectory - E:\revsoft\OpenInsight\pbs_resources\zzPBSSvc\v1.3
: -> LogFile - zzPBSSvc.log
26/10/2010 11:17:34 0000016 6040 : zzPBSSvc loaded OEM configuration settings:
: -> APIPath - E:\revsoft\OpenInsight
: -> AppID - PBS
: -> UserID - PBS
: -> Password -
: -> HandlerProc - PBS_ZZPBSSVC
: -> InitProc - PBS_ZZPBSSVC_INIT
: -> FinalProc -
: -> PersistantEngines - 1
: -> MinEngines - 2
: -> MaxEngines - 3
: -> UseTCP - 0
: -> TCPStartPort - 54321
: -> PipePrefix - OEM
: -> ShowEngines - 1
: -> IdleTimeout - 10
: -> ExecTimeout - 10
: -> GCInterval - 5
: -> CallAttachCmd - 0
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc loaded SFTP configuration settings:
26/10/2010 11:17:34 0000047 6040 : -> SFTP Server - redact
: -> SFTP UserName - redact
: -> SFTP Password - redact
: -> SFTP Unbilled Path - redact/unbilled
26/10/2010 11:17:34 0000047 6040 : --------------------------
26/10/2010 11:17:34 0000047 6040 : -> SFTP Server - redact
: -> SFTP UserName - redact
: -> SFTP Password - redact/unbilled
26/10/2010 11:17:34 0000047 6040 : --------------------------
26/10/2010 11:17:34 0000047 6040 : -> SFTP Frequency - 5
26/10/2010 11:17:34 0000047 6040 : -> SFTP Download Path - E:\revsoft\OpenInsight\pbs\CDR Files\
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc loaded EMail configuration settings:
: -> Email To - support@sprezzatura.com
: -> Email From - SPBS_Service@redact
: -> SMTP Server -
: -> SMTP UserName - post.redact.net
: -> SMTP Password -
: -> SMTP Secure - 0
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc starting OE Manager...
26/10/2010 11:17:36 0001813 6040 : zzPBSSvc started OE Manager OK
26/10/2010 11:17:36 0001813 6040 : zzPBSSvc main thread entering processing loop...
26/10/2010 11:17:36 0001875 4124 : zzPBSSvc thread reports main PBS thread has finished initialization
: -> initError is 0
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc beginning Regular Payment check...
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc processing regular payments...
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc processed regular payments OK
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc ending Regular Payment check
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc beginning CDR check...
:
26/10/2010 11:17:36 0002016 6040 : ** EXCEPTION! **********************************
: ** Swapping to backup SFTP Server redact [Connection lost]
:
:
26/10/2010 11:17:36 0002282 6040 : ** EXCEPTION! **********************************
: ** Exhausted SFTP backup servers1 [Connection lost]
:
26/10/2010 11:22:36 0302391 6040 : zzPBSSvc beginning CDR check...
:
26/10/2010 11:22:37 0302485 6040 : ** EXCEPTION! **********************************
: ** Swapping to backup SFTP Server redact [Connection lost]
:
:
26/10/2010 11:22:37 0303375 6040 : ** EXCEPTION! **********************************
: ** Exhausted SFTP backup servers1 [Connection lost]
:
26/10/2010 11:25:16 0462344 4124 : zzPBSSvc received STOP request...
26/10/2010 11:25:16 0462344 4124 : zzPBSSvc handled STOP request OK
26/10/2010 11:25:16 0462344 6040 : zzPBSSvc main thread exiting processing loop...
26/10/2010 11:25:16 0462344 6040 : zzPBSSvc stopping OE Manager...
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc stopped OE Manager OK
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc engines statistics:
: -> TotalRequestCount - 1
: -> PeakRequestCount - 1
: -> PeakRequestsWaitingCount - 1
: -> PeakEngineCount - 2
: -> CreateEngineFailureCount - 0
: -> TerminatedEngineCount - 0
: -> RetiredEngineCount - 0
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc main thread is exiting ...
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread waiting for main PBS thread to exit
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread reports main PBS thread has finished
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread reports main PBS thread has been released
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread exiting OK

In conclusion
Engine farms aren't for everybody but when you want a robust, failsafe, self recovering engine farm service which can process pretty much anything and scale to meet your needs we have the technology that has been field proven over the past several years and we are confident it can meet your requirements. For further information or to request a quotation contact sales@sprezzatura.com.

Labels: engine farm, ftp, http, https, sftp