Sprezzatura :: Making Databases Happen

You want to do WHAT with Popups - Part 2

2011-12-25T18:55:00.001Z

Following on from our last post on colouring individual popup lines we move onto one of the holy grails of developers - the ability to add a button to a popup and launch our own program from it - interrogating the popup to display more information about the current row for example. Using the example we showed last time - note that we now have a button on the popup called "Display"

Clicking the button results in this

To accomplish this we need to be familiar with

The TIMER tecnique we discussed in the last blog posting
The fully qualified entity ids of the individual popup controls
The use of Window Common
The structure of the Window Common used for quick events

The TIMER technique we discussed in the last blog posting
If you haven't yet read the last blog entry then you might like to before continuing.

The fully qualified entity ids of the individual popup controls
To add a display buttton to the popup we could add a new control to the popup on the fly but that is a lot more complicated than taking over a control that already exists. So the trick that we are going to use is to add a button on to the popup that we're not going to use for its stated purpose and change its display text to "Display" and its action to something more suited to our use.

For this example I'm going to hijack the "Print" button so we need to add that to the popup definition.

For your purposes you might want to use one of the other controls so for your convenience here's a list of them all :-

POPUP.CB_OK

POPUP.CB_CANCEL

POPUP.CB_CLEAR

POPUP.CB_ALL

POPUP.CB_FIND

POPUP.CB_PRINT

Returning to our timer event - the first thing that we're going to do is to modify the TEXT of the print button to make it read "Display".

printButton = "POPUP.CB_PRINT"

objxArray := @Rm : printButton
propArray := @Rm : "TEXT" 
dataArray := @Rm : "Display"      

dataArray = set_Property( objxArray, propArray, dataArray ) 

With that done we need to go about modifying the action to take on the click.

The use of Window Common
To use Window Common you just have to add the following code to your subroutine.

   winId = "POPUP"
   $Insert oiwin_Comm_Init

If you're not familiar with Window Common then where've you been for the past 15 years?

The structure of the Window Common used for quick events
Window Common was effectively documented in detail in Technical Bulletin #2 - Compiled Window Structures, available on an old Works CD. We only need a very limited subset however so we'll deal with that.

Each control on the window has the events associated with it in the variable controlSemantics@. So we need to firstly find the control, then find the CLICK event associated with it and then remove the logic that is currently there and replace it with a call to OUR routine instead.

The quickevent structure that we need is as follows :-

quickEventInfo<0, 0, 0, 1 > = Code for type of event
quickEventInfo<0, 0, 0, 2 > = The action to take
quickEventInfo<0, 0, 0, 3> := The routine to call
quickEventInfo<0, 0, 0, 4> = The parameters to pass

So when the popup is originally launched the labelled common looks like this

We want to modify it to look like this

The code required to do this is as follows :-

Locate printButton In controlMap@ using @fm setting gotIt then

   /*
      OK let's change the quickevent to call OUR program passing in the usual
   */
   quickEventInfo  = "R" : @Tm   
   quickEventInfo := "EXECUTE" : @Tm   
   quickEventInfo := @appId<1> : "*STPROCEXE**" : atSelf : @Tm   
   quickEventInfo := "@SELF" : @stm : "@EVENT" : @stm : "@PARAM1" : |
                     @stm : "@PARAM2" : @stm : "@PARAM3" : @stm : |
                     "@PARAM4" : @stm : "@PARAM5" : @stm : "@PARAM6" : @tm   
   quickEventInfo := @tm            

   Locate "CLICK" In controlSemantics@< gotIt, 8 > using @svm setting gotClick then
      controlSemantics@< gotIt, 9, gotClick > = quickEventInfo               
   end

end

So now all we need to do is to take whatever action we want in our commuter program when the CLICK event is triggered. Here's the code from earlier in this blog post.

onClick:

   begin case
      case object = "POPUP.CB_PRINT"
         objxArray = "POPUP.ET_POPUP"
         propArray = "LIST"
 
         objxArray := @Rm : "POPUP.ET_POPUP"
         propArray := @Rm : "SELPOS"
 
         dataArray = get_Property( objxArray, propArray )
 
         list = dataArray[1, @Rm]
         selPos = dataArray[col2()+1, @Rm]
 
         call Msg(@window, "Your cursor is on " : list< selPos<2>, 1>)
   end case

return

Hopefully this will give you ideas of your own to extend the power and flexibility of popups.

You want to do WHAT with a Popup? Oh - OK then...

2011-12-14T16:22:00.001Z

A recent Works forum posting posed the question "Is it possible to set the color for individual rows in a popup rather than odd/even row colors?". Received wisdom is that "you cannot change colors on specific rows using COLOR_BY_POS due to the modality of the popup". This set the folks in Sprezz Towers to thinking about how this could actually be accomplished. This blog posting reflects the fruits of the combined thoughts and efforts of both the UK and Philadelphia Sprezz offices.

Take as a simple example a gender popup :-

It might be nice to render this as follows :-

To achieve this we need to be familiar with

User Defined Properties
The TIMER event
The TIMER property
The HANDLE property
The COLOR_BY_POS send_Message message.
The Window structure of a popup

User Defined Properties
The developer can add additional properties to any control within an OI app simply by setting a property name prefixed with an @. So call set_Property(@Window, "@SPREZZ", "FIDDLESTICKS") would set the @SPREZZ property of the current window to the value "FIDDLESTICKS". Any other process could retrieve this property using get_Property.

The TIMER event
The TIMER event is fired by OpenInsight as specified by the TIMER property regardless of whether the current dialog/popup/whatever is currently modal.

The TIMER property
The TIMER property tells the system when it should fire the TIMER event and at what intervals.

The HANDLE property
The HANDLE property provides a convenient shortcut to establish whether or not a control exists. If it exists it will have a handle.

The COLOR_BY_POS message
The COLOR_BY_POS message allows individual rows, columns or cells of an edit table to be set to a specific colour.

The Window structure of a popup
At its simplest a popup consists of an edit table (where the results are shown) and buttons for selection or cancelling. The edit table is always called POPUP.ET_POPUP.

Implementation
It might be easier before actually showing the mechanics of how to achieve the effect required, to describe in simple terms how it is achieved. The key is in the seemingly innocous comment above that "The TIMER event is fired by OpenInsight as specified by the TIMER property regardless of whether the current dialog/popup/whatever is currently modal".

Within our program that calls the popup we are going to do the following :-

Within the OPTIONS event for the prompt set two user defined properties

One listing the rows to have their colour set
One listing the colours to set the rows to

Set a TIMER event for the Window BEFORE calling the popup
Call the popup
Within the TIMER event, grab the user defined properties, set the colours using COLOR_BY_POS and then disable the TIMER event.

Whilst this may seem like a hack it works remarkably well with simple popups. Complex popups may require more error trapping.

The OPTIONS event would look like this :-

onOptions:

   objxArray = @Window
   propArray = "@ROWS"
   dataArray = 1 : @Fm : 2

   objxArray := @Rm : @Window
   propArray := @Rm : "@COLOURS"
   dataArray := @Rm : BLUE$ : @Fm : PINK$

   objxArray := @Rm : @Window
   propArray := @Rm : "TIMER"
   dataArray := @Rm : 50

   dataArray = set_Property( objxArray, propArray, dataArray )

   retVal = popup( @Window, "", "GENDER")

   objxArray = object
   propArray = "DEFPROP"
   dataArray = retval

   dataArray = set_Property( objxArray, propArray, dataArray )

return

So we firstly indicate that we would like to change the colours of rows 1 and 2 of the popup. We then explain that we'd like them set to blue and pink (using some previously declared EQUATES). We then tell the TIMER event to kick off 50 milliseconds later and then we call the popup. On a successful return from the popup we accept the value and put it into the control that has just triggered this OPTIONS event.

So the popup will display and within 50 milliseconds the TIMER event will kick in. This has the following code :-

onTimer:

    objxArray = @Window
    propArray = "@ROWS"

    objxArray := @Rm : @Window
     propArray := @Rm : "@COLOURS"

     objxArray := @Rm : "POPUP"
     propArray := @Rm : "HANDLE"

     dataArray = get_Property( objxArray, propArray)

     rows    = dataArray[ 1, @rm]
     colours = dataArray[ Col2() + 1, @Rm]
     handle  = dataArray[ Col2() + 1, @Rm]

     if Len( handle ) then
        ptr1 = 1
        ptr2 = 1
  
        loop
           nextRow = rows[ ptr1, @Fm ]
           ptr1 = col2() + 1
       while len( nextRow )
           nextColour = colours[ ptr2, @Fm ]
           ptr2 = col2() + 1
           call send_Message("POPUP.ET_POPUP", "COLOR_BY_POS", 1, nextRow, nextColour)
       repeat

        objxArray = @Window
        propArray = "TIMER"
        dataArray = ""
 
        dataArray = set_Property( objxArray, propArray, dataArray )
     end

Return

In this code we retrieve the rows to colour and the colours to set them to. We also retrieve the HANDLE of the popup to ensure that it has been created successfully. If it has we use send_Message to colour the individual rows of the popup before setting the TIMER property to null to prevent it from being called again. In theory we ought to be able to set the TIMER property to 0 but there seems to be a bug in the 9.2.1 implementation which ignores the zero and continues calling the TIMER event.

Thus we are able simply to modify the popup after it has been launched.

In the next article in this series we will show how to add a button to the popup to call a user defined routine to display additional information about the current row in the popup.

Using the Table Browser from within a program

2011-11-25T15:40:00.001Z

A developer recently asked on the Works forum if it was possible to use the table browser from within a program. Well the simple answer is "Yes". The slightly more complex answer is that since so many of the OI tools are now written in OI itself it is frequently possible to use these tools from within programs.The first thing we need to know is what the table browser window is called. It may come as a surprise to those more familiar with obscure naming conventions to find out that it is called "TABLE_BROWSER". So to call it programmatically we'd simply

atWindow = Start_Window("TABLE_BROWSER", @Window)

Of course launching it might not be enough - we might also want to populate it with the contents of a table. To do this we need to know the control names. To save you the investigative work we've included a table below :-

TABLE_BROWSER	The window
TABLE_BROWSER.BTN_CANCEL	The cancel button
TABLE_BROWSER.BTN_LOAD	The load button
TABLE_BROWSER.BTN_OK	The Ok button
TABLE_BROWSER.BTN_OPTIONS	The options button
TABLE_BROWSER.DATA_TABLE	The results edit table
TABLE_BROWSER.NUM_RECORDS	The number of records to display
TABLE_BROWSER.SELECTION_BTN	The selection button
TABLE_BROWSER.TABLENAME	The table name edit field
TABLE_BROWSER.TEXT_1	The Tablename static
TABLE_BROWSER.TEXT_2	The Number of Records static

So to launch the browser and load the AVERY_LABELS table we could

 Declare Function set_Property, start_Window      

 atWindow = Start_Window("TABLE_BROWSER", @Window)
 objxArray =  atWindow : ".TABLENAME" 
 propArray = "DEFPROP"
 dataArray = "AVERY_LABELS"    

 dataArray = set_Property( objxArray, propArray, dataArray )    

 Call send_Event(atWindow : ".BTN_LOAD", "CLICK")

Working with Multiple Cursors

2011-11-04T12:14:00.001Z

Finishing up our series on extended selects and readnexts, we will now turn our attention to managing the cursors.

When working with the extended select and readnext statements, the developer is required to maintain the status of the select and readnext cursor. Additionally, as we hinted in our previous entries on Latent and Resolved Selects and Selects and File Resizing, the extended select and readnext statements allow the developer to work with multiple cursors. Multiple cursors means that it's possible to have up to 9 active select statements working at any particular time.

We'll start with cursor status, since that needs to be maintained no matter how many selects may be active at any given time.

We previously mentioned that the extended select syntax is:

select table by columns using cursor then...

While that's true, it's a little simplistic as there are two other keywords, SETTING and ASSIGNING, that you may use as well. The following table describes each keyword in a little more detail:

Keyword	Description
Using	Assigns new criteria to the cursor. If a new table is specified, then the existing criteria is lost.
Setting	Assigns the criteria to the next available cursor.
Assigning	Allows you to assign the results of the cursor to a different table. Any existing results are assigned to the new table.

There are a total of 9 cursors available to the developer (0 through 8) and they can all be active at any given time. By allocating specifically nominated cursors, you can set the system so that cursor 1 is processing your CUSTOMERS table while cursor 2 is processing your INVOICES table. This would be useful if you wanted a report on your invoices sorted in a very specific customer order. As you readnext each customer using cursor 1, you can then issue a new series of selects against cursor 2 finding the invoices for each customer.

Note: While cursor 0 is not reserved exclusively for system use, it is the cursor the system uses. When working with multiple cursors, it's best to only use cursors 1 through 8.

We do realize that you can simply call btree.extract() to return results from the INVOICE table. However, using that you will find that the rows will be returned in key sorted order. If you want any additional sorting then you will need to process the sort, which will normally require an additional file pass. Additionally, since you are managing the cursors, you do not have to worry as much about the system using the cursor from under you, or issuing select statements in subroutines. The SETTING keyword ensures you have a new cursor to manage your selection.

USING keyword
The USING keyword is the most analogous to the basic system SELECT statement. By issuing repeated USING calls to a specific cursor, you can refine the results of your select. Normally, this is mostly used when making a two pass selection. The first pass would resolve based on indexed fields, while the second pass can trim down the results working on a subset of the data. Unlike a standard system select, you cannot take the results of one table and apply the results to a different table. If you do, the original results will be cleared and only the new criteria will be applied to the new table. The assigning keyword allows this functionality.

SETTING keyword
The SETTING keyword finds the next unused cursor and applies the criteria to that cursor. It's the recommended method of starting off a new set of criteria. Which cursor you use doesn't matter, and if you nominate specific cursors, you might accidentally step on a cursor you are using. So, for a two pass report, you would set up the initial select using SETTING, then finish it off with USING.

ASSIGNING keyword
The ASSIGNING keyword is only valid on an active cursor. It is used when you wish to assign existing criteria from one table to a different table.

MODE keyword
Sometimes you may not know at design time which keyword you require. MODE allows you to specify the criteria using a variable. The variable must contain an integer value which identifies the keywork type.

Keyword	Mode Value
Setting	1
Using	2
Assigning	3

For example, if you do not know if you are working with an existing cursor, because you are allowing the user to refine the result list, you can allocate "setting" or "using" through a simple check.

* // refineFlag is defined elsewhere and indicates if this
* // is an existing selection being refined
if (refineFlag = TRUE$ ) then
   cursorMode = 1  ; * // using
end else
   cursorMode = 2  ; * // setting
end

select hTable by columns mode cursorMode else
 ....
end

READNEXT line
Finally, once you have the criteria resolved as you wish, you need to issue the READNEXT command. It's important to remember to include the "USING cursorVariable" portion of the READNEXT when working with multiple cursors. Without this section, the system will assume you are using cursor 0.

READNEXT also supports an optional "by" clause. It's used to determine the direction keys are extracted from a cursor:

Integer	Literal	Description
0	AT	Ascending Terminating.
1	AN	Ascending Non-Terminating.
2	DT	Descending Terminating.
3	DN	Descending Non-Terminating.

For example:

select "MYFILE" by "SORT_FIELD" setting cursorVar else
......
readnext thisKey using cursorVar by DT else

will return the keys in the reverse order returned in the cursor, giving the same results as:

select "MYFILE" by "#SORT_FIELD" setting cursorVar else
......
readnext thisKey using cursorVar by AT else

Note: When using the literals (AT, AN, DT & DN) you must not enclose them in quotes.

CLEARSELECT
CLEARSELECT allows you to clear a cursor by referencing the cursor number you wish to clear.

clearSelect cursorVar

Additional Commands
A little known feature of the Basic+ language is the ability to use a cursor instead of a file handle in file I/O statements. These can be used in READ, WRITE, DELETE, LOCK, UNLOCK, MATREAD and MATWRITE. The basic syntax is the same for each command, just substitute "CURSOR cursorNumber" for the file handle.
For example

read atRecord from cursor cursorVar, atID else

As you've seen, working with multiple cursors gives a developer much more flexibility in working with the system. The small increase in code length and complexity is a small price to pay for the productivity gains you can achieve. As anecdotal proof of this, I once modified a sales report that used a PERFORM SELECT and a series of BTREE.EXTRACTS which was taking over 6 hours to run into a two cursor selection routine and dropped the execution time to just over an hour.

Run_Report, Walk_Report, Crawl_Report

2011-10-26T17:17:00.000+01:00

We'd recently installed a new report using RUN_REPORT at the client and the client complained that as the report progressed it became slower and slower and slower until it was taking minutes to print a line. When the report DID finally conclude it crashed with a SYS1000 error message.

We tested ourselves and sure enough the report was painful. We started it running against a couple of thousand records before going home for the night and by the time we came back in the morning it wasn't even half way through. When it did eventually finish it crashed! Still this last bit was something we were familiar with having already documented it in this blog posting. The slowdown on the otherhand was something else. So we did what most developers do in these circumstances and stripped the report back to bare bones and continued adding features in until it started misbehaving. The report had several BY clauses and several BREAK-ONs but it was when we came to the BREAK-ON modifier "'BP'" that problems started. Without this reports were produced at normal speed - with it they progressively got slower and slower.

Now much as we'd love to just throw the issue at Revelation, we know that the more information that we can provide, the more likely they are to be able to find a fix.Thus began several days of sleuthing. The first resort - as is our wont - was to install a profile file. This quickly grew to humongous proportions and we were faced with analysing a 3.6 GB profile log. Fortunately we have tools for this as regular readers will know, and pretty soon we were able to establish that the culprint seemed to be based around calls to get various printer object related properties.

Taking our investigations one step further we found that with every new line printed in the report an additional 700 GET/SET_PROPERTY calls were made.

To produce the first line of the report 5,324 calls were made
To produce the second line of the report 6,024 calls were made
To produce the third line of the report 6,724 calls were made

and by the time the 60th line of the report was reached GET/SET_PROPERTY was being called 40,000 times to print that one line! Using some additional tools we were able to establish what properties were being requested over and over again.

Armed with this information the stalwarts at Revelation were able to go off and discover a total variable not being cleared when the 'B' option was used. A patch was issued and once again we had RUN_Report. We did then run into another issue but we'll save that for another day!

Down on the farm

2011-08-25T02:56:00.010+01:00

With conference coming up soon in October (see you there we hope?) we had to carefully select our presentations to ensure that they remained educational and weren't just product showcases. One of the subjects that we therefore decided not to cover was creating scaleable distributed applications using our proprietary engine farm. Whilst the techniques involved are interesting we'd ultimately be promoting our proprietary technology and if you wanted a sales presentation you'd have asked for one, right? So we decided to create a blog article about it instead! The results are what you're now reading.

What is an engine farm?
An engine farm is a program that is designed to spin up and spin down a variable number of OpenEngines on a server to deal with incoming and outgoing transactions. As load volumes increase, more engines are brought into play and as demand decreases engines are retired. The farm also has to deal with engines crashing - be it through programmer error or operating system intervention, and has the ability to kill hung engines so that license count usage does not creep up until the system becomes disabled. The Sprezz engine farm was designed to be pluggable as we deal with a number of different clients (see later for a brief description of these) and as such it is currently dealing wih SFTP, FTP, HTTP, HTTPS and obviously SOAP for XML. It’s a native Windows Service controlled (normally) by a control panel app.

What is it used for?
Well the answer is, ultimately, whatever you want to use it for. In this section we'll look at a few of the uses that Sprezzatura's clients have put the engine farm to. The clients will have to remain anonymous for reasons of both commercial confidentiality and national security.

In one application the farm is used for maintaining a phone billing system and front end billing web site. Whilst one engine is downloading call data records using SFTP and then parsing them to create billing records another set of engines might be servicing web site requests to top up individuals phone accounts using HTTPS to do the credit card transactions.

In another application, localised engine farms at branch offices collect transactions from the local network and securely transmit these to another server farm which maintains a centralised database. This database is in turn published to the web from a secure site that is in turn run by a server farm. Data entered on the local systems is live on the national database and available for query within a second or two of entry.

Finally in another application an engine farm sitting on a local server consumes XML requests generated by an external entity and in turn publishes the requested XML back to the requester. As volumes increase more engines ensure that the system remains responsive.

Why not just use the OESocketServer?
Well naturally everyone is going to feel that their solution is the best solution but there's more to our answer than that. The major bullet points that we believe distinguish the Sprezz engine farm are as follows :-

When the Sprezz engine farm was developed the socket server was not available
The Sprezz engine farm runs as a native Windows service so configuration is easier as there is no reliance on the Java runtime being installed nor on using a 3rd party service host to run as a service.
The Sprezz engine farm talks directly to engines via revCAPI not via a Java Native Interface so there is no performance loss from string marshalling
As clients are written per installation it is possible to create highly secure connections without needing to configure a secure tunnel
The control panel application makes configuration more understandable

What does it look like?
Well that's the million dollar question! What do you WANT it to look like? In all cases as the engine farm is a service it doesn't look like anything. The only thing that you'll see is the configuration window - be this an application or a control panel component. For the purposes of this blog post we'll use the phone billing application as the example.

Whilst the core engine farm remains the same throughout, the front end is modified per application. In this instance the configuration window for the engine farm is an executable, in other instances it is a control panel app.

When it's running as said previously it's a native Windows Service - so the only sign that it is running is an entry in the services list :-

The remote engines are designed to periodically run specified stored procedures. These programs are periodically polled by the engines to perform the required tasks. For this application, we need two tasks running. These are the "CDR Handler Proc" and "FP Handler Proc" functions.

In the Phone Billing app the configuration window looks like this :-

In the first subtab we specify how to log into the engine, and the procedure name utilised by the farm to pull down and price call records (SFTP) and the procedure name used by the farm to do the credit card billing (HTTPS).

In the second subtab we actually configure information about the engine farm itself. The prompts are relatively self explanatory but to deal with them in order :-

Engine Folder - where the engines should be launched from
Show Engines - whether the individual engines should be visible
Persistent Engines - whether the engines should "stay alive" after they have been launched. If not checked the engine will be closed as soon as it has completed the requested task.
Idle Timeout - if the engines are persistent how long must they be idle before they are closed?
Min Engines - the minimum number of engines to have alive at any one point
Max Engines - the maximum number of engines to have alive at any one point. This prevents the engine farm from taking ALL of the engines if there are local network users too.
RCL Mode - as with the Universal Driver - how should communication with the engines take place?
TCPIP Start Port - for when TCPIP is selected over Named Pipe
Pipe Prefix - if Names Pipes is used, what prefix should be used for the pipe name?

The final tab contains less used information. Again the prompts are relatively self explanatory but :-

Init Proc - the routine to run when an engine is first started
Final Proc - the routine to run before an engine closes
Execution Timeout - if an engine has not responded after this many minutes, kill it as it is assumed to have died
GC Interval - time between internal housekeeping events
Bind Engines - this tells the engine manager to generate a unique ID for an engine so they can be identified easily.

In this particular application we need to log onto an SFTP server and if the logon fails walk through a list of alternate servers. This table is used to configure this.

In the event of errors the service will mail the error details to a nominated email address. This is configured here.

For trouble shooting the service can be asked to keep a log on what it gets up to (this is in addition to the routine updating of the event log with errors that happens normally). The log contains details of all aspects of the service's operations rather than just error conditions.

A redacted sample of an error log follows :-

26/10/2010 11:17:34 0000016 6040 : E:\revsoft\OpenInsight\pbs_resources\zzPBSSvc\v1.3\zzPBSSvc.exe - v1.3.0.22
: ===========================================================================
:
26/10/2010 11:17:34 0000016 6040 : zzPBSSvc loaded log configuration settings:
: -> LogDirectory - E:\revsoft\OpenInsight\pbs_resources\zzPBSSvc\v1.3
: -> LogFile - zzPBSSvc.log
26/10/2010 11:17:34 0000016 6040 : zzPBSSvc loaded OEM configuration settings:
: -> APIPath - E:\revsoft\OpenInsight
: -> AppID - PBS
: -> UserID - PBS
: -> Password -
: -> HandlerProc - PBS_ZZPBSSVC
: -> InitProc - PBS_ZZPBSSVC_INIT
: -> FinalProc -
: -> PersistantEngines - 1
: -> MinEngines - 2
: -> MaxEngines - 3
: -> UseTCP - 0
: -> TCPStartPort - 54321
: -> PipePrefix - OEM
: -> ShowEngines - 1
: -> IdleTimeout - 10
: -> ExecTimeout - 10
: -> GCInterval - 5
: -> CallAttachCmd - 0
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc loaded SFTP configuration settings:
26/10/2010 11:17:34 0000047 6040 : -> SFTP Server - redact
: -> SFTP UserName - redact
: -> SFTP Password - redact
: -> SFTP Unbilled Path - redact/unbilled
26/10/2010 11:17:34 0000047 6040 : --------------------------
26/10/2010 11:17:34 0000047 6040 : -> SFTP Server - redact
: -> SFTP UserName - redact
: -> SFTP Password - redact/unbilled
26/10/2010 11:17:34 0000047 6040 : --------------------------
26/10/2010 11:17:34 0000047 6040 : -> SFTP Frequency - 5
26/10/2010 11:17:34 0000047 6040 : -> SFTP Download Path - E:\revsoft\OpenInsight\pbs\CDR Files\
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc loaded EMail configuration settings:
: -> Email To - support@sprezzatura.com
: -> Email From - SPBS_Service@redact
: -> SMTP Server -
: -> SMTP UserName - post.redact.net
: -> SMTP Password -
: -> SMTP Secure - 0
26/10/2010 11:17:34 0000047 6040 : zzPBSSvc starting OE Manager...
26/10/2010 11:17:36 0001813 6040 : zzPBSSvc started OE Manager OK
26/10/2010 11:17:36 0001813 6040 : zzPBSSvc main thread entering processing loop...
26/10/2010 11:17:36 0001875 4124 : zzPBSSvc thread reports main PBS thread has finished initialization
: -> initError is 0
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc beginning Regular Payment check...
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc processing regular payments...
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc processed regular payments OK
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc ending Regular Payment check
26/10/2010 11:17:36 0001922 6040 : zzPBSSvc beginning CDR check...
:
26/10/2010 11:17:36 0002016 6040 : ** EXCEPTION! **********************************
: ** Swapping to backup SFTP Server redact [Connection lost]
:
:
26/10/2010 11:17:36 0002282 6040 : ** EXCEPTION! **********************************
: ** Exhausted SFTP backup servers1 [Connection lost]
:
26/10/2010 11:22:36 0302391 6040 : zzPBSSvc beginning CDR check...
:
26/10/2010 11:22:37 0302485 6040 : ** EXCEPTION! **********************************
: ** Swapping to backup SFTP Server redact [Connection lost]
:
:
26/10/2010 11:22:37 0303375 6040 : ** EXCEPTION! **********************************
: ** Exhausted SFTP backup servers1 [Connection lost]
:
26/10/2010 11:25:16 0462344 4124 : zzPBSSvc received STOP request...
26/10/2010 11:25:16 0462344 4124 : zzPBSSvc handled STOP request OK
26/10/2010 11:25:16 0462344 6040 : zzPBSSvc main thread exiting processing loop...
26/10/2010 11:25:16 0462344 6040 : zzPBSSvc stopping OE Manager...
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc stopped OE Manager OK
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc engines statistics:
: -> TotalRequestCount - 1
: -> PeakRequestCount - 1
: -> PeakRequestsWaitingCount - 1
: -> PeakEngineCount - 2
: -> CreateEngineFailureCount - 0
: -> TerminatedEngineCount - 0
: -> RetiredEngineCount - 0
26/10/2010 11:25:16 0462360 6040 : zzPBSSvc main thread is exiting ...
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread waiting for main PBS thread to exit
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread reports main PBS thread has finished
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread reports main PBS thread has been released
26/10/2010 11:25:17 0462454 4124 : zzPBSSvc thread exiting OK

In conclusion
Engine farms aren't for everybody but when you want a robust, failsafe, self recovering engine farm service which can process pretty much anything and scale to meet your needs we have the technology that has been field proven over the past several years and we are confident it can meet your requirements. For further information or to request a quotation contact sales@sprezzatura.com.

Selects and File Resizing

2011-08-23T12:51:00.001+01:00

In a previous blog on Latent and Resolved Selects, we talked about how processing through the Basic+ reduce function and select statements were quicker than using the RLIST command. As hinted at in the prior article, the speed increase comes with additional maintenance required by the developer. In this part of our series on Selection, we'll talk about this maintenance, and the steps you need to take.

There are three areas that the developer needs to be aware of. The first is file resizing. The second is readnext termination. The third is cursor status, which will be covered in the next in this series of postings.

File Resizing

As most of you know, Revelation's Linear Hash Filing System (LH) is more accurately known as "Linear hashing with two partial expansions, separate chaining, distributed control and variable length records". It's the "distributed control" portion we're interested in here, since that's the part that manages the file resizing.

Without going into a detailed technical discussion about how and why the system chooses to resize the file, we know it happens without requiring any specific action by the user. The system checks for a resize on certain file access. When a file resizes, it generally interacts with only one or two groups. The records in these groups can be split and moved anywhere across the entire file. This means, when you read a record from a table, you can potentially trigger a resize.

Since there is no way of knowing where the records in the group will move to, it's possible that the records will move to a group with a lower group number. This means it will be positioned earlier in the file than the current position.

This is important, because when the system is processing a non-indexed select without an active key list, it starts at group 0 and moves through each group, examining each record. If the file resizes while this is happening, a record can be moved to a group that has already been processed, meaning the record will be skipped. Of course, the opposite can happen as well, in that a record that has already been processed can be moved forward to a group that hasn't been processed yet. This record can then potentially show up twice.

To avoid this problem, the RLIST function increases the sizelock by two when issuing a select.

A sizelock of 0 means the file will resize normally, growing or shrinking as required.
A sizelock of 1 means the file will only grow when required. It will never shrink.
A sizelock of 2 or more means the file will never resize.

Increasing the sizelock by two ensures that the file will not resize out from under you while the select is being processed.

Since you are managing the select and readnext yourself, and not RLIST, you need to manually set the sizelock. This can easily be accomplished using the FIX_LH routine.

$insert DB_MGMT_EQUATES

equ INCREMENT_SIZELOCK$ to 1

call fix_lh( tableName, UPDATE_SIZELOCK$, INCREMENT_SIZELOCK$ )

When you have finished processing the list, you must decrement the sizelock back.

$insert DB_MGMT_EQUATES

equ DECREMENT_SIZELOCK$ to 0

call fix_lh( tableName, UPDATE_SIZELOCK$, DECREMENT_SIZELOCK$ )

It's important to remember that you should not decrement the sizelock until after you have completely finish processing the file. That means when you have finished processing the readNext loop, not just after you have issued the select statement.

ReadNext Termination

The second item to be aware of is readnext termination. When working with a latent list, the system continues to process the select in a non-terminating loop. To avoid this, you must use the AT (ascending terminating) or DT (descending terminating) options when issuing the readnext statement.

The extended ReadNext syntax is

readNext atId using cursorVar by AT else...

where the "by" option can be the following literals or numbers.

Literal	Number	Direction
AT	0	Ascending Terminating
AN	1	Ascending Non-Terminating
DT	2	Descending Terminating
DT	3	Descending Non-Terminating

Note: Do not put the "by" option in quotes. It's a statement, not a passed string.

If you know the sort mode at compile time, you can use either the literal or the number.
If you will only know the sort mode at runtime, you must use the number, which you can assign to a variable.

if option = ASCEND$ then
   sortMode = 0 ;  * // AT - ascending
end else
   sortMode = 2 ;  * // DT - descending
end
readNext atId using cursorVar by sortMode else...

Cursor Status

As mentioned above, cursor status and working with multiple cursors will be covered in the next post in the Select Series.

DATAFILE= In OpenInsight

2011-08-19T18:39:00.000+01:00

One of the nice features that AREV has that is missing from OpenInsight is the ability to change the datafile of a databound window on the fly by adding the modifier DATAFILE=NewDataFileName onto the sentence being used to launch the window. This was frequently used when multiple files shared the exact same structure - for example a "current" file and the associated archive files.

As we've mentioned we're working on a large AREV->OI conversion and we really missed this facility so in true Sprezz style we decided that we'd better implement it.

This proved to be easier than even we had anticipated due to the similarity between OI Window Common and AREV Window Common. The workaround we implemented is only good for windows which only update one file, though it wouldn't take much to make it suitable for multi-file windows. The following code shows how to implement it in a create event although of course if you have a promoted event framework you might choose to do it there.

   winId = @Window
   $Insert OIWin_Comm_Init
   $Insert OIWin_Equates


create:
   changeRequired = index( param1, "DATAFILE=", 1 )
   if changeRequired then
      newFile = param1[ changeRequired + 9, @Fm][1, " "]
      JoinMap@><1,JMAP_TABLENAME$>  = newFile
   end
return

EditTable Cookbook - Trapping the ENTER key

2011-08-18T11:24:00.003+01:00

A recent question on the Revelation Forum asked how to detect an ENTER keypress when using an EditTable as this is not exposed by the standard OpenInsight CHAR event (ENTER is one of the keys that the EditTable normally considers to be "reserved" like TAB and the Arrow keys).

Despite this limitation it is actually possible to detect an ENTER keypress with WM_KEYDOWN window message and the WINMSG event but ONLY if the EditTable is set to "Protected", i.e.

To trap the ENTER key we first need to tell OpenInsight that we want to be notified of any WM_KEYDOWN window messages that are sent to the EditTable by using the OpenInsight QUALIFY_EVENT message. This is normally done in the form's CREATE event:

0001     $insert winAPI_WindowMessage_Equates
0002     $insert logical
0003     
0004     call send_Message( @window : ".TABLE_1", "QUALIFY_EVENT", |
0005                        WM_KEYDOWN$, TRUE$ )
0006     

Once we've done this we can write a WINMSG event handler for the EditTable that responds to the VK_RETURN keycode:

0001  /*
0002     Example WINMSG event handler to trap the Enter key (VK_RETURN$)
0003     while processing the WM_KEYDOWN$ window message.
0004        
0005        hwnd    -> handle of the EditTable
0006        message -> Message number - should be WM_KEYDOWN$
0007        wParam  -> Virtual Keycode - we are looking for VK_RETURN$
0008        lParam  -> Keydown flags - see MSDN for more info!
0009        
0010  */
0011     
0012     $insert winAPI_WindowMessage_Equates
0013     $insert winAPI_VirtualKey_Equates
0014     $insert logical
0015     
0016     begin case
0017        case ( message = WM_KEYDOWN$ )
0018        
0019           begin case
0020              case ( wParam = VK_RETURN$ )
0021                 // Write your code to handle the event here
0022                 call msg( @window, "Enter key pressed!" )
0023                 
0024           end case
0025           
0026     end case
0027     
0028  return TRUE$

(Note in these examples we are using some insert records from our WinAPI Library that you can find here)

An Open and Shut Case

2011-06-21T13:30:00.002+01:00

We've been working quite a bit with OIPI at Sprezz Towers recently, and having not had a problem creating PDFs locally we were flummoxed when the same PDFs would not create on the client's workstations - rather, every time we called Set_Printer("TERM") we'd get the following...

Being a little overawed by the apparent complexity of the INIT call we checked and rechecked our code but could find nothing wrong with it. Finally we gave in and contacted Revelation Support in the US who asked simply "Do you have a SYSTEMP table?". Well as it happens, locally we do - but on client sites we tend to only install directories we need so - as we weren't using AREV32 - we hadn't installed the AREV_DIR subdirectory and a file called SYSTEMP that lives there.

So we created a SYSTEMP table and lo and behold our formerly recalcitrant program began creating PDFs like they were going out of fashion.

But this still begs the question - why a SYS1000 loading error? To understand this we have to understand how file IO works in OpenInsight - especially in relation to file handles. In OpenInsight a file handle is effectively the name of the various Modified Filing Systems (MFSs) attached to the file followed by the program responsible for implementing the various IO opcodes (the Base Filing System - BFS), followed by the name of the DOS file that the operation should be performed upon. So taking a simple example of an indexed file, the file handle might look like this :-

SI.MFS²RTP57²REV10101.LK

When the system encounters say a read opcode, it would take the first program in the list (SI.MFS) and call it passing in RTP57²REV10101.LK. SI.MFS would then do it's thing (which in this case is to call RTP57 passing it REV10101.LK, and on return update index transactions) and then exit.

How does this help us?

Well look at the call stack - we've gone from our program to SET_PRINTER1, to RTP36, to RTP27. To explain this simply, our program has called SET_PRINTER1 with a TERM request. SET_PRINTER1 has tried to place a lock on a table (RTP36), but RTP27 - the program loader - has complained that it has been asked to call a null program. Why would RTP36 try to load a null program? Referring back to our logic, IO opcodes strip the first program from the file handle and call it. They don't check to see if there is anything IN the file handle as why would a programmer ever ask the system to use a null file handle?

From this we're able to conclude that the file handle for SYSTEMP (fingered as the culprit by RevSoft earlier) must have been null. From this we can extrapolate that somewhere in the code for SET_PRINTER1 there is an

Open "SYSTEMP" to vSystemp Else vSystemp = ""

Rev assure us that this has been fixed in 9.3 but it's worth bearing in mind that when faced with a SYS1000 error you might be looking at a blank file handle.

Making Progress With Selects

2011-05-24T17:16:00.003+01:00

One of our current mega-projects is converting a very large AREV system over to OpenInsight and of course like most large systems it relies heavily on batch processes that perform selects and refine these selects before presenting the results. AREV had the useful status line update facility to let you know how processing was going but regretfully OpenInsight omitted this - presumably because lacking a dedicated status line there was nowhere to put it. With hindsight a SEND_INFO every 1% would at least have allowed the developer to intercept and update the information from a select.

We tried the usual approach of looking the other way but when some of the queries were taking upwards of five minutes to return results to the user we decided that enough was enough.

The problem as we saw it was split into two areas - selects against entire non-indexed files and selects against indexes. This article deals with the former. The next article in this series will deal with the latter.

To start let's be frank. This blog isn't about giving away the family silver. It's about telling you what is possible using Rev products and giving you enough information to do it yourself if you want to. To this end we won't be publishing reams of code - we'll be publishing snippets and explanations of how to achieve the end result!

The steps involved
At its simplest the easiest way of providing the user with a progress indicator when reading through a file or a resolved active select list is as follows :- include within the select statement an additional column which makes a callback to a user defined routine to provide feedback to the user about where we are in the process. Like most consultancy houses, Sprezz has a generic progress window that we use to keep the user informed of progress through a process. This is called in one of four ways :-

To start the progress window
To update the progress window
To update the log portion of the progress window
To end the progress window

Thinking about this, this means that for our select statement to report back to us, we need to do the following :-

Initialise our counter variables and any labelled common
Start our progress window
Call our rlist select command
From within the rlist select command call back to a routine to update the progress window
On termination close down the progress window

Now that's all very well, but it's a faff to have to go in and change all of our existing calls to RLIST to do this set of actions, so it makes sense to create our own select program that encapsulates all of the above steps.

The Result
Having done this we can then just issue a call to our replacement routine passing it the select statement required, and Bob as they say, is the brother of your mother. So in our simple example we issue the command :-

Call gen_Select_With_Progress( "SELECT CLAIMS" )

and the result we see is :-

(Please excuse the background colour of the MDI Frame. Like most development shops we use three systems, development, testing and live and we colour the background vividly to remind the person working on the system just where they currently are!)

The Breakdown
So looking at the important parts, let's see how they work - firstly let's look at a simplified version of our gen_select_with_progress. Our full version has more flexibility built in but we've stripped this down to the bare bones to make it easier to understand.

The shell program

Function PList( object, method, param1, param2, param3, param4)
/*
   Author      AMcA

   Purpose     To do a select with a progress bar

*/

   equ version$   To "1.0.0"
   $uses CMG*APPROW**GEN_CONFIGURATION:GEN_RESOURCES

   Declare Function get.reccount, gen_progress, retStack, zzx_res2Str
   $insert gen_dict_callback_equates
   $insert gen_module_equates

   retVal = TRUE$
   atSelf = retStack()<1>

   if method = "" Or method = "SELECT" then
      call set_status( 0 )
       /*
         assume that object contains the select statement To execute
      */
      object = trim( object )
      table = field( object, " ", 2)
      open table To vTable then
         if @rec.count then
            rowCount = @rec.count
         end Else
            rowCount = get.recCount( vTable, "", "")
         end
         loopCounter@ = 0 ; * loop counter
         * start the progress window
         retVal = gen_progress( "STARTWIN", rowCount, caption)
         * log what is being done
         retVal = gen_Progress("LOG", "", object)
         * add our callback into the select - it will now be called once 
         * per row evaluation
         addOn = " WITH ZZX_CALLBACK = 1"
         if index( object , " WITH ", 1) then
            addOn = " AND " : addOn
         end
         call rlist( object : addOn, 5, "", "", "" ) ; * do the actual select 
         call gen_progress( "END" )
      end else
         call fsMsg()
      end
   end else

      locate method In "UPDATE" using "," setting pos then
         on pos goSub update
      end
   end

return retVal

That copes with list items 1, 2, 3 and 5 so how does the CALLBACK dictionary item work?

CALLBACK Dictionary Item
If you place a compiled dictionary item into the SYSDICT table it can be used in all of your SELECT statements. All that this has to do is to call your update program. Again a simplified version of our SYSDICT item...

   declare function gen_select_with_progress, isEventContext
   $insert gen_dict_callback_equates
   @ANS = ''

   if isEventContext() then

      dict   = @DICT
      id     = @ID
      mv     = @MV
      record = @RECORD
      rnc    = @RN.COUNTER
 
      @ans = gen_select_with_progress( "", "UPDATE")
 
      @DICT   = dict
      @ID     = id
      @MV     = mv
      @RECORD = record
      @RN.COUNTER = rnc

   end

return @ans

As an aside, for efficiency we'd be better off using the transfer statement to move @RECORD and @DICT in the above code as all strings over 20 characters in length are held on the heap rather than in a descriptor, so a copy statement wastes resource.
The Update Routine
Finally the part of gen_select_with_progress that updates the progress meter.

update:

   loopCounter@ += 1
   if mod(20, loopCounter@) = 0 then
      call gen_select_with_progress("UPDATE",loopCounter@, |
                                    "{COUNT} of {MAX} - estimated time remaining|
                                    {ETR}")
   end

return 

The only thing to mention is that loopCounter@ is a labelled common variable. Because the update program and the calling dictionary item are on the program stack there is minimal overhead to this approach. It also allows for the user to press a cancel button and exit a protracted select. In our testing we've seen this add anywhere between 5 and 20% overhead. This might seem like a lot but when a user is staring at a blank screen waiting for a result time drags a lot more slowly. In addition the estimated end time allows the user to plan their time more effectively :).

In the next article we'll look at indexed selects.

Latent and Resolved Selects

2011-05-23T13:16:00.002+01:00

Recently at Sprezz Towers, we had an interesting discussion on resolved selects, latent (or unresolved) selects, sizelocks and manipulating records while in a select list. Surprisingly, there was some confusion, even between ourselves. Once we hashed it all out, we thought it would be a good idea to post on the topic.

Types of Selects

As most of you know, there are two types of selects in the Revelation environment, latent and resolved. In resolved selects, the results are processed and stored in a temporary SYSLISTS (or LISTS) record. In latent selects, the record IDs are not returned until the READNEXT (or READNEXT BY) statement is executed. Resolved selects are initiated through the RLIST function or PERFORM statement. Latent selects are initiated through the Basic+ reduce subroutine and select statement.

What many people do not know is that inside each resolved select is a latent select waiting to get out. In order to generate the resolved select, the system must process the select through a latent list.

So, if you were to issue the following as a resolved select:

  SELECT SYSPROCS WITH TYPEID = 'STPROC' BY ENTITYID BY APPID

the system will basically execute code similar to this:

0001  * // Parse the sentence
0002  * // Assign some vars
0003  * // Open handles
0004  * // Other initialization based stuff 
0005  abort = FALSE$
0006  script = "WITH {TYPEID} EQ 'STPROC'"
0007  sortList = "APPID"
0008  mode = NEW.REDUCE$
0009  tableName = "SYSREPOS"
0010  cursor = 0
0011  flag = ""
0012  
0013  call fix_lh( tableName, UPDATE_SIZELOCK$, INCREMENT_SIZELOCK$ )
0014  
0015  call reduce( script, sortList, mode, tableName, cursor, successFlag )
0016  if successflag else
0017      * // set error
0018      abort = TRUE$
0019  end
0020  
0021  if abort else
0022      select tableName by sortList using cursor else
0023          * // set error
0024          abort = TRUE$
0025      end
0026  end
0027  
0028  if abort else
0029      listData = ""
0030      reccount = 0
0031      done = FALSE$
0032      loop
0033          readnext id using cursor by AT else
0034              done = TRUE$
0035          end
0036      until done
0037          listData := id : @FM
0038          reccount += 1
0039      repeat
0040      
0041      listData[ -1, 1 ] = ""
0042      call dostime( dateTime )
0043      listKey = "W*" : @STATION : "*" : @APPID : "*" : dateTime
0044      listHeader = @VM : dateTime : @VM : @APPID : @VM : reccount : @VM
0045      listRec = listHeader : @FM : listData
0046      write listRec on hLists, listKey else
0047          * // set error
0048      end
0049  end
0050  
0051  if abort else
0052      call make.list( cursor, listData, hDataHandle, hDictHandle )
0053  end
0054  
0055  call fix_lh( tableName, UPDATE_SIZELOCK$, DECREMENT_SIZELOCK$ )

More or less...it's not perfect, but it's a pretty rough idea of what happens.

The system parses your sentence, generates a reduce, performs a select/readnext, stores all the returned keys, writes them off to a LISTS record (and updates the save select list queue) and then does a make.list so they are active and available.

In short, resolved selects are a giant shell around reduce/select. When you work with a resolved select, you're processing the list twice; once to resolve it, and once to process it.

It's always quicker to skip the shell and work with reduce/select yourself. However, as with many shelled functions, the system also handles some maintenance work for you.

In the next article, we'll talk about the maintenance work required to handle all this correctly.

OI Log Files - a practical demonstation of sleuthing

2011-04-07T15:18:00.000+01:00

Those of you who attended our talk at the last User Conference in Vegas will know that we're a big fan of using system logs to help track down aberrant behaviour. Recently we were asked by RevSoft UK to help solve a recalcitrant problem that was being encountered by one of their clients.

Put simply, the client had upgraded to 9.2.1 from 9.2 and was finding that whilst they could use the System Editor++ to open rows, any attempts to open Stored Procedures would result in a system hang. The engine would report that it was "processing" but was unresponsive and could only be closed from the task manager.

The first thing that we did was confirm that the SE++ worked as expected in SYSPROG. It did. We then created a new application and repeated the test there. It still worked. We then created an application inheriting from the client application and tested there. It failed. So we knew that the issue was localised to the client application. The question was WHERE was it actually crashing? We created a profile log and ran the application to point of crash and we were able to determine that the system was hanging inside a UTILITY call from DIALOG_BOX.

Moving on to the next logical step we wrote a small program to look for programs in SYSPROG that also existed in the client application. This revealed only a few potential mismatches. Of particular interest was a localised version of CENTER_WINDOW that took only 1 parameter instead of 3 and a localised version of OBJ_CALL_EVENT that took 0 parameters instead of 12.

Looking at the SYSREPOSEVENTEXEs for the Open Source Code dialog it was apparent that it made heavy use of OBJ_CALL_EVENT but that the Open Row dialog didn't use it at all.

So problem solved... DELETE_ROW "SYSOBJ","$OBJ_CALL_EVENT*CLIENTAPP" log off, log back in and all would be good.

Except it wasn't. We still had the same problem.

Time for the next phase - we installed a linear hash log and what we saw was as follows :-

RD 48 SYSPROG*OIWINEXE**RTI_EDITOR_ENTITY_OPEN OK
RD 54 SYSPROG**RTI_EDITOR_ENTITY_OPEN OK
RD 67 SYSPROG**RTI_EDITOR_ENTITY_OPEN ERROR:100 FILE:0 GROUP:0
RD 48 CLIENTAPP*STPROCEXE**OBJ_CALL_EVENT OK
RD 43 $OBJ_CALL_EVENT*CLIENTAPP ERROR:100 FILE:0 GROUP:0
RD 13 SET_DIALOG_INFO ERROR:100 FILE:0 GROUP:0
RD 43 $SET_DIALOG_INFO OK

So interpreting this for you... firstly the code RD simply means "Read". The number following is a filehandle. The file corresponding to the file handle can be found earlier in the log.

RD 48 SYSPROG*OIWINEXE**RTI_EDITOR_ENTITY_OPEN OK
The system reads the SYSREPOS entry for RTI_EDITOR_ENTITY_OPEN and determines that it must read the RTI_EDITOR_ENTITY_OPEN from SYSREPOSWINEXES. As stated we can tell that this is reading from SYSREPOS as described above OR by deducing it from the key structure.

RD 54 SYSPROG**RTI_EDITOR_ENTITY_OPEN OK
The compiled window is read

RD 67 SYSPROG**RTI_EDITOR_ENTITY_OPEN ERROR:100 FILE:0 GROUP:0
The SYSREPOSLANGUAGES table is checked to see if any alternative language defintions have been set up for this window. There haven't been so we get an FS100 - REC_DNE_EXIST$. This is benign.

RD 48 CLIENTAPP*STPROCEXE**OBJ_CALL_EVENT OK
It then goes back to SYSREPOS because the CREATE quick event of the window tells it to run a procedure calleed OBJ_CALL_EVENT. So it looks for the SYSREPOS entry called CLIENTAPP*STPROCEXE**OBJ_CALL_EVENT. It find this entry and the next line

RD 43 $OBJ_CALL_EVENT*CLIENTAPP ERROR:100 FILE:0 GROUP:0
shows that it has been told that it will find this object code in SYSOBJ with a row id of $OBJ_CALL_EVENT*CLIENTAPP. This read fails (because earlier we deleted the code) - this is not quite so benign as now the CREATE event will fail.

The system then carries on with the Dialog Logic :-

RD 13 SET_DIALOG_INFO ERROR:100 FILE:0 GROUP:0
RD 43 $SET_DIALOG_INFO OK

Now note that it DOESN'T try and read $OBJ_CALL_EVENT*SYSPROG as an RTP27 call would. RTP27 would automatically traverse the inheritance chain looking for the object code. Rather it assumes that it has got the correct definition and continues. This is because OBJ_CALL_EVENT is a Quick Event on the CREATE of the Source Code Open Dialog so it is called via a REPOSITORY EXECUTE. When the REPOSITORY EXECUTE is unable to load the code, rather than error out it just ignores the failure. Consequently the CREATE event never runs and the Dialog (which starts life invisible) is never made visible. So it is there on the desktop awaiting input but can never be closed as it can not be seen. Earlier in the log we could see GET_REPOS_ENTITIES being called and this was a clue to the fact that the Dialog was operating.

So we went into the Application Manager and deleted the SYSREPOS entry for OBJ_CALL_EVENT, logged out, back in and all was well with the world again. And the moral of this story (apart from the one involving routines named the same way as system routines)? When dealing with items that are tracked in the Repository - use Repository methods to manipulate them!

Revelation Conference 2011

2011-02-09T23:17:00.001Z

As you'll all doubtless be aware Revelation have announced the dates of the next conference (Tuesday, October 4th - Friday, October 7th, 2011 if you weren't) and whilst we've got some great ideas for presentations we're always on the lookout for more.

So we're appealing to you dear reader. Is there an aspect of the product you'd like to see taken apart in typical Sprezzatura fashion? What nooks and crannies would you want pushed and prodded? What esoteric diversion would you see pursued?

Answers on a postcard... or an email... or a blog comment. If you don't come back you'll just have to take what we give you :)

Thanks in advance!

Indexing Enigma

2011-01-12T16:39:00.002Z

Recently a Sprezz client's client experienced server issues and commissioned a new server to host the client's application. Shortly thereafter an index failure occurred and the client tried to rebuild the indexes in question. They were thwarted in their efforts however by the fact that no tables would show in the DM_INDEXES_ADD window. They tried everything, removing the indexes and adding back programmatically, rebuilding system indexes, recreating DBTs but nothing would work.

A Sprezz technician remoted into the system and performed some initial checks. This rapidly became an object lesson in the pursuit of red herrings. An LH3 log was generated and briefly checked for errors - but none were found. An engine profile log was generated and briefly checked for errors - but none were found. The files were detached and reattached but still they wouldn't show up.

The files were attached from Sysprog and the indexes were there for all to see. In addition a strange anomaly was found. At one of the locations being attached there was a table called "0" - yes, that's ZERO.

The technician was delighted - this must be the cause. Obviously this table showed up first in any list and the system must be using boolean checks to assemble a list of tables to display, you know the sort of thing :-

0001     filePtr = 1
0002     loop
0003        nextFile = fileList[ filePtr, @Fm ]
0004        filePtr = col2() + 1
0005     while nextFile
0006        goSub process
0007     repeat

instead of the safer :-

0001     filePtr = 1
0002     loop
0003        nextFile = fileList[ filePtr, @Fm ]
0004        filePtr = col2() + 1
0005     while len( nextFile )
0006        goSub process
0007     repeat

So the file was renamed (well the media map entries were copied around and deleted) and the location was reattached and the system indexes rebuilt. But still the issue persisted.

A new application was created and the files were attached and again indexes were present. So now the speculation was that a promoted event was interfering with the (now OI based) database manager window. A more thorough analysis of the engine profile log revealed that the only promoted event in use during the execution of the DM_INDEXES_ADD routine was a CLOSE event. So back to the drawing board again.

Next up the LH3 log file was examined in greater detail and the individual index files could be seen being opened and the *INDEX records being read. So there wasn't an issue with the files themselves - they were plainly there.

Finally the technician stopped and reasoning that the problem had to be associated with the move to the new server, he undertook a more detailed analysis of the processes associated with the DM_INDEXES_ADD dialog. With this new focus the technician observed that the DM_INDEXES_ADD called V119 and a light went off in his brain. Moving back to the Database Manager he called up the environment settings and noted that the sort path referenced there had not been created on the new server. Create the directory, log out and back in and all was "sorted".

Sometimes the easiest solution is just that - the easiest solution.

The finer details.

2010-12-29T02:17:00.001Z

Sprezzatura's attention to the smallest details are legendary.

On Thursday I finished up the code for a window we're converting from ARev to OI 9.2. I was just about to start on the softkeys when I see it's 4:00 so I figure this is a good stopping point and I head off to pick up the kids.

I leave the office, and Ellen (the office manager at Sprezz Towers West) teases me about leaving before her on Christmas break.

I head home, and cycling down Spruce Street I think there's loads of extra delivery trucks, but very little traffic. I chalk this up to being just before Christmas.

As I walk to the daycare center, I talk to my dad, whose a little confused why I'm picking them up now.

I get Jacob and his class is on the way to the gym, and I thought that's a bit odd at this time of day, but I've only been picking up the kids since Monday, so I'm not clear on their schedule.

As we walk home, Jacob and I are talking about how the days are getting longer and how it's still light going home, and I think to myself, it is really light out considering how dark the walk home had been the past few days, the change isn't that drastic so quickly.

Then he wants to go to the park, and I tell him it's far too late and we need to make dinner. He argues that it's not, and we have time, so I take out my watch to show him the time, and it's not 5:20 at all. It's 4:20. I think my clock's wrong, so I call my dad to ask him the time, and sure enough its only 4:20.

I was looking the clock on the client's box (the work is done remotely) and they're an hour ahead, on Atlantic time. So, despite looking at the clock by the front desk when I left the office, and despite looking at the clock at the house when I dropped of the bicycle, and despite my Dad wondering why I was picking the kids up now, and despite looking at the clock at daycare when I arrived, I never noticed it was an hour earlier than I thought.

Another week, another month's work

2010-12-28T16:50:00.002Z

Just before Christmas we were invited by one of our clients to visit their site and to perform a number of OI related tasks in a very finite time frame. Specifically to upgrade a 16 bit OI 3.7.5 production system and an OI 8.0 web based system to OI 9.2+ whilst installing the UD Heavy. We were allocated 5 days to do this so were confident that it was achievable.

Of course our confidence was rooted in real world experience - both of of our combined skill sets and of typical UK weather...

So early Friday afternoon came and exercising caution the Sprezz lead consultant ("the Sprezzie") on this project began the 200 mile drive to the client - this was to be a Sat/Sun/Mon/Tue/Wed job to minimise impact on the factory production system and to fit in with some tight scheduling requirements for Sprezz. Unbeknownst to us this was to mark the beginning of some of the worst weather in recent UK history. In a normal world this 200 mile journey would take 3.5 hours.

Two hours into the journey the Sat Nav decided to change the route due to traffic conditions ahead and rerouted the car to within 20 miles of the original starting point then started again... snow was blizzarding and road conditions were becoming treacherous. Average speeds dropped and past experience of snow driving became essential. The next 5 hours or so were not fun but eventually the Sprezzie arrived at their target hostelry - a snowed in Bed and Breakfast in the middle of nowhere - just in time to discover that after a 7.5 hour journey, all the places serving food had shut. And so to bed.

The next morning dawned and the 12 mile drive to the client had never looked so difficult. The roads were snowed in and the gritting lorries had yet to visit. Thanking providence for front wheel drive the Sprezzie eventually got to the client and started work. The initial UD Heavy installation was a cinch, it installed and it did what it said on the tin. The 4.6 UDH is one we at Sprezz Towers have a lot of experience with and we have no hesitation in recommending it to our clients. It addresses a lot of issues found in previous releases of the UDH and the documentation is an order of magnitude better!

The 9.2 upgrade was a bit more of an issue - the install was fine but combining the 3.7.5 and the 8.0 system proved to be a bit more of a challenge. Eventually two APPBACKUPs were taken and a custom program was created in conjunction with the client to combine "the best of both worlds" and this custom program installed all elements into a virgin 9.2 system without the need for an APPRESTORE.

Everything came together BUT the system wasn't running for some reason. So thus began the multi-sourcing quest. First off we downloaded Sprezz's "Recompile All" utility. This recompiled all events, stored procedures and windows. Things started to run again but the windows looked ugly. So then we downloaded Rev's "Font changer" utility and changed all the fonts to Tahoma in Windows and Popups. Finally we downloaded and ran SRP's form fixer to reset the errant style bits and we were back in the running.

Next up the client agreed that hiding the LK and OV files was a worthy goal - no more users accidentally trashing systems. So we followed the incredibly simple instructions in the UDH installation guide and attempted to hide the files. It should be said that the installation was on a 64 bit server. The Sprezzie in charge tried and tried but could not get the files to hide - the registry settings would not take. So throwing it open to all Sprezz crew he mailed the internal distribution list and was cajoled by colleagues into triple checking his registry entries. To much embarrassment it became apparent that Rev create two key sets in the WOW3264 section - one as Revelation Software for the client - and one as RevSoft for the UDH. It behoves the developer to use the RevSoft key if s/he wants Shares to work...

So onto the next thing on Santa's wish list - converting their OECGI installation to OECGI3 on an IIS 7 web server. Quick start read, product up and running in less than an hour. Way to go RevSoft! These quick start guides REALLY help.

Starting to wrap up and wind down the client mentions that they'd like to convert their existing mail server (all based around Rev's MAPI technology) to a more future proof technology. No problem - Sprezz's free SMTP mail client is downloaded and installed and the conversion of the mail server continues apace - until.... what? You want to CONSUME mail as well as PRODUCE it? Uh Oh....

It suddenly became apparent that there is a gap in the RevSoft offerings...

GETMAIL as shipped with OI is a handy socket based POP3 mail client which pulls down the mail from the pop server and delivers it as text files to the client - the only downside is that it

a) is underdocumented and
b) it delivers effectively binary streams unlike the MAPI equivalent that delivers nicely documented structures.

Fortunately for us (and the client) the client only needed to consume several very specific mail formats so again mining the collective Sprezz gestalt we were reminded of the existence of BASE64DECODE in OI and were able to put together an OI based program to decode multi part MIME emails in a manner consistent with the existing MAPI calls. This allowed the existing Mail Server technology to be migrated to SMTP/POP3 with minimal change.

All in all a fun filled and technically challenging week made possible by the flexiblity of the OI product, the contributions of the Rev community and the whole being greater than the parts that is the Sprezz consultancy team.

Building a better Delay() - Part 1

2010-11-04T12:26:00.000Z

In this post we’re going to turn our attention to an old Arev subroutine called Delay() that can still be used in OpenInsight. It’s a simple function and is designed to halt a Basic+ program for a specified number of seconds before continuing again (full details can be found in the OpenInsight on-line help).

However, while Delay() made the transition to the Windows world, it still behaves in a fashion befitting its DOS/Arev heritage - i.e. it expects to run in a single, non-multitasking environment. Internally it simply sits in a tight until loop until the nominated interval has passed after which it returns. This is all well and good but on a modern OS this steals virtually all the CPU time, which is silly when you consider that the actual intent is to simply “do nothing”!

So, assuming that we actually want to introduce a delay of some sort into a program, what we really need to do is only put the actual OpenEngine thread (i.e. the thread that executes your Basic+ code) into a state whereby it uses no CPU processing time until the required interval has passed, after which processing can resume.

The Windows API has several functions for putting threads to “sleep”. i.e. putting them into a state whereby they use no CPU resources until they are “woken up” based one or more wake criteria, and we’ll begin by looking at the most simple of these called, appropriately enough, Sleep().

The Sleep() function

This is a very simple Windows API call that suspends the calling thread for at least the specified number of milliseconds. It is documented by Microsoft like so:

VOID WINAPI Sleep( DWORD dwMilliseconds );

Parameters

dwMilliseconds

The time interval for which execution is to be suspended, in milliseconds.

A value of zero causes the thread to relinquish the remainder of its time slice to any other thread of equal priority that is ready to run. If there are no other threads of equal priority ready to run, the function returns immediately, and the thread continues execution.

Return Value

This function does not return a value.

(Full details on Sleep can be found at: http://msdn.microsoft.com/en-us/library/ms686298(VS.85).aspx, though we make no promises on the future validity of that URL as MSDN seems to get a new facelift far too often :)

Using Sleep()

As Sleep() is a Windows API function it must be prototyped before you can use it from Basic+. You can create your own prototype if you wish, but it is already included in OpenInsight, aliased with the curious name of "Sleepery" in the DLL_KERNEL32 prototype record so you may use that version if you prefer (Quite why Sleep() was aliased to Sleepery() is beyond your humble authors understanding, but I digress...)

(It is also available in our Windows API Library package, aliased as "WinAPI_Sleep", for those of you who have installed it into your systems. However, for this example we'll use the version installed with OpenInsight)

So the Sleep() function, on the surface, sounds ideal and we can write our new version of Delay() something like this:

compile subroutine DelayEx( delayTime )

   call sleepery( delayTime * 1000 ) ; * // Sleep works with milliseconds
   
return

If you test-run this function you’ll see that your CPU usage is kept at 0% during the Sleep() interval and everything appears fine... or does it?

Every thread in Windows has a message queue, and messages can be posted to this queue to interact with the thread (Forms and controls that are created by a thread also share its message queue, so any messages posted to them will end up here too). Unfortunately, during a Sleep() interval, the thread won’t check it’s message queue (or do anything else for that matter) which means that other clients relying on the result of the message may even appear hung, and this can happen readily in the case of programs which broadcast messages to every top level window in the system and wait for a response before continuing. You may well think that this won’t apply to OpenEngine, as it’s only running Basic+ code right? Wrong. Don’t forget OpenEngine has its own window (you know, the actual OpenEngine window with the list of executed commands and the ‘Debug’ and ‘Log’ buttons) and that window still exists even when you launch OpenInsight and specify that you don’t want a visible engine. It hasn’t gone anywhere – you just can’t see it!

So, what we need is a function that can put a thread to sleep for a defined period of time BUT has the capability to be awakened should a message need to be processed. The Windows API provides just such a function in the form of MsgWaitForMultipleObjects(), and this is something we’ll look at in the next post.

OI and the Windows 7 System Font

2010-10-21T20:38:00.001+01:00

For a while now I've been suffering with an annoying problem when running OI on my Windows 7 laptop: any window or control that used the "default" font was rendered at the wrong size - it was being rendered as though I had a high DPI (120) set, which of course I didn't - the laptop was set to run at 96 DPI.

Here's a couple of examples to show you what I mean:

None of my other PCs running OI had this problem - just my laptop. The only difference was that the laptop OS was installed by Dell, whereas all my other machines were installed by me.

After some searching I came across the following blog post which gave the solution:

Windows 7, bitmap fonts and Microsoft Dynamics GP

Basically the DPI was set to 120 during the installation process which means that the standard OS bitmapped fonts were installed as 120 DPI rather than their 96 DPI versions. Unfortunately in Windows 7 these fonts are not changed by a user's individual DPI settings - they are system wide, and so affect all users regardless of their DPI setup.

Making the registry changes described in the MSDN blog article above solved the problem and OI now looks normal on my laptop again.

The Mysterious Case of the Dialog Box checkbox

2010-09-17T09:21:00.000+01:00

To my mind one of the longest-standing mysteries in OpenInsight has been the purpose of the "Dialog box" checkbox in the Window Properties dialog as you can see here:

For years the function of this checkbox has eluded me (and my erstwhile colleagues), so much so that I believed it was simply something that the original Rev programmers intended to use but never implemented.

Well, after some recent work with the Revelation crew I came across the answer, and it's really simple:

It's used by the Repository( "EXECUTE" ) method for launching a window - if it's checked then the Repository function calls the Dialog_Box() function to execute the window instead of the normal Start_Window() function.

Mystery solved!

The Case of the Absent MENU Event

2010-09-15T09:30:00.002+01:00

Here at Sprezz Towers we're hard at work on the Big Project: a large Arev to OI conversion using our shiny new promoted event framework (well it ain't that new but it gets more polished each time we use it). Part of this week's programming fun has been implementing Arev Softkeys functionality as a menu that dynamically configures itself based on the active MDI child - we have a promoted event that simply alters a static "placeholder" menu on the fly using a combination of properties such as VISIBLE, ENABLED, TEXT and so on.

Visually all this was working really well, but as I waded into event handling I came across an unexpected problem: No events for the softkey menu items would fire! Hmm...

First of all I decided to check if the MDI frame was actually being told by Windows about the menu click: The trusty Microsoft Spy++ tool soon revealed that the frame was indeed receiving the menu item's WM_COMMAND message, but still nothing was getting to OpenEngine. This was really strange as normally ALL menu items call the RUN_EVENT function to see if there's a MENU event handler to execute - this time there was no action whatsoever.

Next I started turning off the Set_Property calls one by one to see if I could determine if any particular property was responsible for the strange behaviour and I came across an interesting result: As soon as I set the ENABLED property to FALSE$ the menu item lost the ability to execute a MENU event, even if I re-enabled it by setting ENABLED back to TRUE$.

This had me stumped at first until I tested some more and removed the code setting the VISIBLE property: If I didn't use the VISIBLE property then ENABLED worked as advertised. Progress of a sort I guess...

More testing revealed this: If I removed a menu item by setting it's VISIBLE and ENABLED properties to FALSE$, in that order, and then I subsequently set the VISIBLE and ENABLED properties back to TRUE$, again in that order, then the menu item's event handling stopped working. So, the next question was ... Why?

The evidence suggested that the ENABLED property did something different based on whether or not a menu item was hidden. A little more work with Spy++ showed that a hidden disabled menu item still raised a WM_COMMAND message when fired (via an accelerator key combination), but RUN_EVENT wasn't called. However, if the menu item was visible and disabled no WM_COMMAND message was raised. Ergo the menu ENABLED property must trigger a switch inside OpenInsight that turned off hidden menu item event handling.

Thinking about this some more led me to the following conclusions:

When a menu item is visible OpenInsight can use the Windows API EnableMenuItem function to disable it - the menu item is "grayed out" and clicking on it produces no WM_COMMAND message.

When a menu item is hidden it is actually deleted from its parent menu - Windows knows nothing about it and therefore EnableMenuItem can't be used. However, OpenInsight still keeps the item details in memory so that hidden items can be executed via accelerator keys, so there must be a mechanism to disable this as needed.

Indeed there is, and it's even there on a checkbox I totally ignore when designing menus in the Form Designer:

You can actually turn off menu item event processing by design and it's this flag that the ENABLED property is setting when you use it with a hidden item. The actual OpenInsight bug is that the flag is never set or removed if you use the ENABLED property on a visible item!

In other words, if you disable a hidden item you must re-enable it before you make it visible again if you want it to fire events.

So, armed with this new wisdom it was a simple matter to ensure that the ENABLED and VISIBLE properties were set in a better order, and now event handling works like a charm - another case closed!

Bespoke? Well yes but...

2010-09-04T12:59:00.002+01:00

The Rev community probably won't be aware of this but this month we're running an advert for Sprezzatura in a "mostly small business" magazine here in the UK. We figure that small business owners are sufficiently independently minded that they'll be able to see the benefit of custom software over commercial off the shelf software that still needs heavy customisation but costs ten times as much.

The target market place is sophisticated so we don't want to hit them with reams of copy - rather we want to appeal to their innate good sense. You don't as a rule get to grow a small business by being stupid. So the advert we came up with is reproduced here :-

We thought (and fought) long and hard about it internally and were pleased with the end result. The advert was due to go to press on September 1st. Today (September 3rd) we received an email addressed to info@sprezzatura.com from a gentleman who wanted us to fit him in for an appointment to tailor a suit for him.

We were mortified. Adverts aren't cheap and to think that we might have so misjudged it that people thought we tailored bespoke suits?

A flurry of investigation followed and one of our smarter members noticed that the person asking for the tailoring might have a Canadian presence as their company was present there. A couple of clicks in a browser and the answer was apparent. There exists in Canada a bespoke tailors with a website of www.sprezzatura.ca. It has existed for quite some time BUT the first time in our existence we get a tailoring query is two days after our "bespoke" advert went to press....

Of course with hindsight if it went to press two days ago it is unlikely to be in the wild yet so the chances of the mail being in response to our advert is 0% but hindsight is always 20/20!

Popups - OI Downs

2010-08-27T18:17:00.002+01:00

We recently came across a situation where launching a popup would display a white screen where the popup should be and then GPF OpenInsight. This wasn't the effect we were actually looking for so we turned to our old friend WinDbg to track down the issue. Unsurprisingly it came down to the data table dll. (We say unsurprisingly because at the end of the day a popup is just an edittable with some extra buttons).

Unfortunately this didn't help so an anguished cry rent the air as your weary reporter decried the vexatious piece of software. Another Sprezzie hearing the cry enquired as to its cause. Upon being told he asked "would you have a long column heading on your dictionary item?". This was rapidly checked and sure enough the dictionary column heading was three lines deep. This was modified to just be the necessary length and lo and behold the popup now worked.

It hasn't been blogged previously by the other Sprezzie as it hadn't been reproducible - it was just a suspicion. Suspicions confirmed we're pleased to blog it lest anyone else be equally as baffled!

Semiversal Driver

2010-08-19T16:04:00.004+01:00

It has come to our attention recently about a misconception regarding the lore of the Universal Driver. It has been commonly accepted by all (including us here at Sprezz Towers) that on small networks (2-4 users), using one of the Universal Drivers without an NT Service is the preferred network configuration.

However, after a lengthy debugging session involving OpenInsight 9.1.1 and the UD 4.6, Revelation US has verified that without a Linear Hash Server product, you must use the All Networks Driver.

To quote Revelation:

Yes, out of the box the UD clients are included but that is to ease the installation. You still need a proper server setup for shared access.

You are right, but the “Universal Driver” refers to the combination of the Linear Hash Service and the Client Driver. Together the service and client drivers are the “Universal Driver” package. Using only the client driver without the service is only half the setup.

...

No, the “All Networks” driver was a unique code base and worked on a completely different principle than the Universal Driver. To think of the UD as an all networks driver is not accurate.

OpenInsight 9.x includes the Universal Driver (LinearHash service and client driver) for , as I believe, the reason of making the setup of a proper stable multi-user setup easier. This is especially true for small 5 user networks that couldn’t justify the cost of the LinearHash service.

With the Universal Driver NUL (Network User License) free for all OpenInsight 9.x customers, we urge all pure OpenInsight shops not using a Linear Hash service to install the Universal Driver 4.6 NUL on your servers.

Those customer that choose not to upgrade to 9.x and are not using a Linear Hash Service product should switch drivers to the All Networks as soon as possible. Remember, switching back to the All Networks driver could cause you problems if any of your tables have been created with the new UD style header.

To help with this, Sprezzatura has enhanced our Revert LH Utility. This takes a linear hash table with the new UD 3 style header and converts it to standard LH format, readable by all network drivers.

Since the last release a few years back, we've fixed a few small bugs involving keys traversing frame boundaries. We've also added a new user interface window to help you with reverting your files back to the original header structure.

Our new user interface allows you to see the Revelation file names of the files you wish to revert. It also allows you to pick and choose files, if, for some reason, you'd like to keep the new header on some files.

This is a free of charge utility, and can be obtained by contacting us here.

Please note: The free version of the UD 4.6 does not work with any version of OpenInsight prior to version 9. It also does not work with Advanced Revelation.

(This post brought to you by the letter A(aron))

Random numbers in OpenInsight

2010-07-29T19:52:00.000+01:00

One issue that recently reared it's head again at Sprezz Towers Support Central was the problem of using the Basic+ Rnd() function to generate a random number sequence - apparently the same series of numbers was being produced every time.

On first investigation it appeared that the customer was not using the InitRnd() function to seed the Rnd() generator before asking for the random numbers. This is a documented requirement - Calling InitRnd() is an absolute necessity before using the Rnd() function.

Even so, using InitRnd() as specified in the online help failed to produce sufficiently random results so we decided to take a closer look at the process to see what the problem was and how we could improve it.

Under the hood the Basic+ InitRnd() and Rnd() functions are little more than thin wrappers around the srand() and rand() functions in the Windows C-runtime library. The rand() function actually produces a pseudorandom sequence of numbers, so it is not truly random at all - instead the srand() function is used to seed the starting point for the pseudorandom sequence so it begins in a different place after each initialisation. Also, if srand() is called again with the same seed the same sequence of numbers is generated by rand()!

Going back to the problem, the OpenInsight online help gives an example of using InitRnd() with a seed based on the current time and date (which is what our customer tried next):

0001  /*
0002     The present time and date set the random number 
0003     generator with a unique value.
0004  */
0005  
0006     InitRnd Time(): Date()

Unfortunately for this example using the date results in a seed value that doesn't show much variance in it's low order value (it only increments by one per day), and this appears to have the effect of generating a identical Rnd() sequence between each initialisation (especially if called multiple times on the same day which is something that should be avoided). We then thought about swapping the date and time around so that the low order value would hopefully be more varied, but here we find ourselves in danger of generating predictable sequences if we started our application at the same time every day (like at 9:15am for example).

What we needed was some way to fulfil the following criteria:

Ensure that InitRnd() is only called once per session to avoid the same seed being used more than once.
Provide InitRnd with a varied seed value that is less predictable and has a greater "spread" than a simple date/time value.

We quickly put together the following function to do this that gave much better results:

0001  subroutine zz_InitRnd( bReset )
0002  /*
0003     Author  : Mr C, Sprezzatura Ltd
0004     Date    : 29 July 2010
0005     Purpose : Simple function to call initRnd with with a more varied 
0006               seed value, and to also ensure we only call it once per 
0007               session.
0008               
0009     Parameters
0010     ==========
0011     
0012     bReset  : If TRUE$ then call initRnd regardless of whether or
0013               not it's been called before.
0014               
0015  */
0016     declare function getTickCount, getEngineWindow
0017     $insert logical   
0018     
0019     if assigned( bReset ) else bReset = ""
0020     
0021     common /%%_ZZ_INITRND_%%/ zzInitRndSeed@
0022     
0023     * // Check to see if we've already been called this session.
0024     if zzInitRndSeed@ then
0025        * // We've called it already - are we asking for a reset?
0026        if bReset else
0027           * // Nope!
0028           return
0029        end
0030     end
0031     
0032     * // Start off with the tickcount (ms) and the
0033     * // OE hwnd ...
0034     seed = getTickCount() + getEngineWindow()
0035     
0036     * // Now adjust this slightly to help avoid the time
0037     * // factor
0038     if mod( seed, 2 ) then
0039        b = seed[-1,1]
0040        if b then
0041           seed = int( seed / b )
0042         end else
0043           seed = int( seed / 3 )
0044        end
0045     end
0046     
0047     * // Right - run it through some more mods to keep it
0048     * // within range of a short int if it's not already...
0049     loop
0050     while ( seed > 0x7FFF )
0051        if mod( seed, 2 ) then
0052           seed -= 0x7FFF
0053        end else
0054           b = seed[-1,1]
0055           if b then
0056              seed = int( seed / b )
0057           end else
0058              seed = int( seed / 3 )
0059           end
0060        end  
0061     repeat
0062     
0063     initRnd seed
0064        
0065     * // Flag that we've already called this...
0066     zzInitRndSeed@ = seed
0067           
0068  return

(Note that we have a global variable in place to ensure that initRnd is only called once per session, unless we explicitly override this)

You can download a text version of zz_InitRnd here.

Disclaimer

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

EditTable Cookbook - Row selection without focus

2010-07-12T11:16:00.001+01:00

Following on from our recent post on how to remove the row-selection from an EditTable we came across a related issue last week that we think you might like to know about.

Basically we designed a form with several EditTables, all row-select enabled, but during testing it became difficult to know where the actual input focus was as so many controls had a selection highlighted. We could have used the technique in the aforementioned blog post to clear the row-selection during the LOSTFOCUS event, but in this case we still needed to see which row was selected even though the focus was on another control.

The solution we used was to change the highlight color of the EditTable during the LOSTFOCUS event, toning it down to a lighter shade than normal. We then reset it during the GOTFOCUS event. This is actually quite easy to achieve with the COLOR_BY_POS message, the only real challenge is to devise an algorithm to fade the highlight color.

As usual we've saved you the trouble - here's small function that you can use as a starting point for your own application should you wish to use a similar technique:

0001  subroutine edt_FadeSelection( edtID, bFade )
0002  /*
0003     Author   : Darth C, Sprezzatura Actual
0004     Date     : 12 Jul 2010
0005     Purpose  : Function to adjust the row selection color of an edit table
0006     
0007     Parameters
0008     ==========
0009     
0010       edtID    -> Fully qualified name of the edit table 
0011       
0012       bFade    -> If TRUE then fade the selection color, otherwise
0013                   reset it to normal 
0014                   
0015     Requirements
0016     ============                 
0017       
0018  */
0019     declare function rgb
0020     $insert winAPI_EditTable_Equates
0021     $insert winAPI_SysColor_Equates
0022     $insert logical
0023     
0024     if assigned( edtID ) else edtID = ""
0025     if assigned( bFade ) else bFade = FALSE$
0026     
0027     if len( edtID ) then
0028        if ( bFade ) then
0029           goSub fadeSelectionColor
0030        end else
0031           goSub resetSelectionColor
0032        end
0033     end
0034     
0035  return
0036  
0037  ///////////////////////////////////////////////////////////////////////////////
0038  ///////////////////////////////////////////////////////////////////////////////
0039  
0040  fadeSelectionColor:
0041  
0042     origColor =        winAPI_GetSysColor( SYSCOLOR_HIGHLIGHT$ )
0043        
0044     origColor = fmt( oconv( origColor, "MB" ), "R(0)#32" )
0045     bleach    = 185 ; * // this is how much white we want to add...(0..255)
0046             
0047     selColor =        iconv( origColor[25,8], "MB" )
0048     selColor := @fm : iconv( origColor[17,8], "MB" )
0049     selColor := @fm : iconv( origColor[9,8], "MB" )
0050        
0051     * // Now add the bleach to each component
0052     for x = 1 to 3
0053        pct = ( 1 - ( selColor<x>/255 ) )
0054        selColor<x> = selColor<x> + int( pct * bleach )
0055     next
0056        
0057     selColor = rgb( selColor<1>, selColor<2>, selColor<3> )
0058     txtColor = winAPI_GetSysColor( SYSCOLOR_WINDOWTEXT$ )
0059     if txtColor else
0060        * // 0 (BLACK) means "default color" in COLOR_BY_POS processing!!!
0061        txtColor += 1
0062     end
0063        
0064     dtcs =       DT_DEFAULTCOLOR$                                           |
0065          : @fm : DT_DEFAULTCOLOR$                                           |
0066          : @fm : selColor                                                   |
0067          : @fm : txtColor
0068             
0069     call send_Message( edtID, "COLOR_BY_POS", 0, 0, dtcs )
0070  
0071  return
0072  
0073  ///////////////////////////////////////////////////////////////////////////////
0074  
0075  resetSelectionColor:
0076  
0077     dtcs =       DT_DEFAULTCOLOR$                                           |
0078          : @fm : DT_DEFAULTCOLOR$                                           | 
0079          : @fm : DT_DEFAULTCOLOR$                                           |
0080          : @fm : DT_DEFAULTCOLOR$
0081             
0082     call send_Message( edtID, "COLOR_BY_POS", 0, 0, dtcs )
0083  
0084  return
0085  
0086  ///////////////////////////////////////////////////////////////////////////////
0087  ///////////////////////////////////////////////////////////////////////////////

In your EditTable LOSTFOCUS event you call this:

    call edt_FadeSelection( @window : ".TABLE_1", TRUE$ )

And to reset the colors in your EditTable GOTFOCUS event you call this:

    call edt_FadeSelection( @window : ".TABLE_1", FALSE$ )

(You'll notice that the function uses two "WinAPI" $insert records, both of which can be found the WinAPI Library that we recently posted.)

You can download a text version of edt_FadeSelection here.

Disclaimer

EditTable Cookbook - Selecting and deselecting rows

2010-07-12T11:14:00.000+01:00

There's a quick way to select or deselect all rows in a multi-select edit table, and that's via the DTM_SELALLROWS message. It takes a single wParam argument which is TRUE$ to select all rows, or FALSE$ to deselect them.

Here's an example:

   $insert logical
   equ DTM_SELALLROWS$ to 1085 ; * // ( WM_USER + 61 )
   
   hwndEdt = get_Property( @window : ".TABLE_1", "HANDLE" )
   
   * // Select all rows....
   call sendMessage( hwndEdt, DTM_SELALLROWS$, TRUE$, 0 )
   
   * // Deselect all rows....
   call sendMessage( hwndEdt, DTM_SELALLROWS$, FALSE$, 0 )

From the department of shooting oneself in the foot...

2010-06-25T14:58:00.000+01:00

Here at Sprezz Towers we're big fans of trying to write clean, maintainable code that lends itself to easy modification. As part of this, we like to keep our resource strings in a separate configuration row and pull them in at runtime using a call to one of our function libraries.

Today this technique rose up and bit us on the proverbial... consider the following code :-

begin case
  case memberRelatedCall= ''
    //* CTC012: Member call question must be answered|
    errorText := zzx_Utility( "RESSTR", RESKEY2$, "CTC012", "", 1)

You'll see that as a friendly reminder to following programmers we include the text of the error message as a comment before the call to retrieve the error text.

We were unit testing a module and ensuring that the appropriate errors messages were displayed when error conditions were found. But try as we might we couldn't get the errors to display and clearly incorrect data was somehow escaping validation.

This frustrated us no end. We single stepped through code and whilst the error condition was clearly there we just couldn't get it to set the error text to flag the exception. Even more mysteriously in the debugger we couldn't get the cursor to stop on the assignment of the error text.

Finally light dawned. As we were building up a potentially long error message we were including pipes in the literal as above. And whilst a pipe in a message means new line a pipe on the end of a program statement means the opposite - continue line. So our error text assignment was being seen as a comment and was not acted upon.

Of course if we'd thought to use #PRAGMA OUTPUT Table Row our mistake would have become immediately apparent.... hindsight eh?

As Homer so accurately says... D'oh!

Sprezzatura Windows API Library

2010-06-11T13:30:00.000+01:00

The Windows API provides a rich set of features for your application to utilise, but it can be a daunting task unless you’ve spent many hours programming in C/C++. Although the Windows API is well-documented on-line these days it can be a arduous task trying to map OI to Windows data types when prototyping functions and defining structures, or even searching C/C++ header files for constant definitions.

Here at Sprezz Towers we've done a lot of Windows API programming over the years and we've managed to amass quite a collection of DLL function definitions, structures and constants, and you can find them all in an RDK at the link below.

We've used some naming conventions with these so hopefully they don't collide with your own definitions:

DLL prototype records are prefixed "DLL_WINAPI_"
DLL functions are prefixed "WINAPI_"
Structures are prefixed "WIN32_"
Inserts are prefixed "WINAPI_"

A list of all the library components can be found in the RDK README.TXT file.

Sprezzatura Windows API Library Download

Enjoy!

Disclaimer

EditTable Cookbook - Clearing single row selection

2010-06-11T11:48:00.002+01:00

One of the more annoying visual traits we find with the EditTable is with single row selection: There seems to be no way to unselect a row and remove the highlight once it's there, and this can lead to a confusing state when it appears that more than one control has the focus.

We've often been asked how to rectify this as simply setting the SELPOS property doesn't suffice, so here's the solution - we use the Windows API SendMessage function to clear the selection like so...

   $insert logical

   equ DTM_SELROW$    to 1083 ; * // ( WM_USER + 59 )
   equ DTPOS_INVALID$ to -3

   hwndEdt = get_Property( @window : ".TABLE_1", "HANDLE" )
   call sendMessage( hwndEdt, DTM_SELROW$, FALSE$, DTPOS_INVALID$ )

Thou shalt encrypt

2010-05-06T17:14:00.001+01:00

One of the more scary things to come out of the recent Revelation conference was the frequent discussions about the new legislation coming through in the US mandating the encryption of data like SSNs. This is something that is obviously of concern to end-users and software developers alike. As some commentators mentioned:

If you have a database that contains 1,000 names of Massachusetts residents and lose it without the data being encrypted, that's $5,000,000.

Yikes could your business, or that of your clients, afford to take such a risk with unencrypted data?

A product that we’ve been using ourselves for several years provides secure commercial grade encryption capability for OpenInsight applications (OpenInsight’s ENCRYPT_FORMAT uses an algorithm which is widely viewed as not being suitable for new applications). S/Encrypt provides multiple encryption algorithms (including AES256) to satisfy the most demanding of clients (in our case security aware financial firms and “sensitive” Government Agencies).

This is a topic that every software developer, working with sensitive data, needs to be taking seriously these days. If you find yourself in this camp, then you’ll be interested in S/Encrypt as your fast, secure and economical solution to this legal requirement. To learn more, please contact Martyn (Martyn at Sprezzatura dot Com) for a copy of the latest S/Encrypt documentation.

RevCon 2010 - the dust settles...

2010-05-04T15:16:00.001+01:00

Well we're back from another successful Revelation Conference and we'd just like to say a big "thank you" to Revelation for organising it. Thank you also to all of the attendees who dropped by our stand and opted to "take Sprezzatura's Speakers home with them". Whilst we don't know if it's true that "Sprezzatura always have the best giveaways" as several people commented, it is nice to know that our efforts to provide giveaways that combine a promotional message with something that you can actually use are appreciated.

Highlights and lowlights of the conference will differ for everybody, but for us some of the highlights included the comments of our fellow developers.

The fact that Aaron Kaplan is back with Sprezz after a two year hiatus met with universal approval - especially as he will be moving back to Philly and thus be more on time zone to help with our US clients there. He won't actually be there until summer - there are too many logistics to sort out, but we know many of you look forward to seeing him again - if not before then at next year's conference.

Carl's talk on embedding the web browser control was admired by many for the way in which he took a potentially highly technically complex subject and reduced it to a series of easily comprehensible slides and presentations. As one delegate said afterwards "you know - I consider myself to be a technically competent developer, but when I see the things that Carl puts together I realise that I'm just not in the same league". We don't know about that, but we appreciated the sentiment.

Andrew's talk on troubleshooting OI systems using system logs was well received with one delegate expressing surprise that such powerful tools had been in the product for so long, but that he'd never heard of them. Perhaps our favourite delegate quote was from one well respected developer who commented that this presentation had taught him more long-term useful information than any other session he'd attended at this conference. Even Sean (Revelation’s documentation maven) left with some information he said he could add to the documentation.

The low point of the conference for us had to be Andrew's concussion (caused by a wind borne heavy cushion – hence a con-cushion) at the roof party on the Tuesday evening which led to a lost evening for him. Andrew lost the next day as the concussion continued to move him in and out of consciousness before his hospital visit followed by a sometimes painful and frequently scary next few days as the concussion played havoc with his short term memory and anxiety levels. He is very grateful to all delegates who helped him avoid trouble in the aftermath of the post party concussion and those who volunteered to submit witness statements to the hotel management concerning the event. Four days after the event, the symptoms continue but at least they seem to be lessening.

Perhaps the highlight of the conference for most was seeing just how mature the O4W product has become. It really seems as though this might be the next generation of development tools from the Rev stable to help our community easily deliver Web 2.0 based applications. In the medium term, this could even be the tool to help us to reach out to a wider community. Equally as exciting was the announcement of the new M class licensing for people who wish to use another back end database, but use the OI front end tools for development.

Finally we have to give a big shout out to the newer Revelation Staff - Jared Bratu and Erik Smith - who both provided incredibly polished maiden talks, made all the more impressive by their lack of previous form. It's great to see such obvious enthusiasm from the next generation of Revelation programmers.

All in all the conference had something for everyone and so we look forward to seeing you all at the Rio again next year for RevCon 2011!

UDH 4.6 - Cracking the code

2010-04-18T02:15:00.000+01:00

We've got a couple of clients with UDH installations and this weekend we upgraded one of them to 4.6 - the latest and greatest. There's so much to like about this release - not least the improved documentation and improved replay of journalling. But sometimes the unforeseen happens and you hit a "Well nobody would ever do that would they?" moment - the moment when just as you ask yourself the question you see several possible scenarios unfold where not only might they do that, they'd have damn good reasons for doing it.

So after the weekend was over we returned to base and expected a tranquil week. Until Monday morning that is. We seemed to have two unrelated problems :-

Users were getting kicked out with "Maximum number of variables" in RTP32
The Event Viewer on the (Windows 2008 r2) UDH Server was full of thousands of similar error messages.

Well fixing the RTP32 problem was going to be easy - We just consulted our on-line REVMEDIA for what RTP32 actually does. OK FIELDSTORE. So we'd just look for where that was going into an infinite loop. For details of how we'd actually achieve this why not come to our talk on using logs to troubleshoot Revelation problems at RevCon 2010?

Regretfully it didn't seem to be as easy as that as we weren't actually making much use of FIELDSTORE. So we'd come back to that issue and concentrate on the Event Viewer issue. As we said the Event Viewer on the (Windows 2008 r2) UDH Server was full of thousands of similar error messages. All of which made no sense whatsoever to us. To make matters worse, scattered amongst these errors were more meaningful ones.

Thankfully as ever the tech team at RevSoft swang into action and decided that we was seeing two problems. One they thought might be related to messages (we'll come back to that) and one that might indicate that we did in fact have data errors in some of our tables.

Tracking down the data errors was made difficult by the fact that with nearly 10,000 errors to choose from it's difficult to search for a specific message - in this case the text "REV" to find the actual DOS files affected. Initial attempts to export from the Event Viewer failed as selecting Tab delimited text resulted in a small subset of the log being exported. After several failed attempts we dropped back to CSV and lo and behold, a 70K file became a 12 MB file. With this we were able to open in notepad and identify the files in question. One was FUBAR, a 10K LK and a 4 GB OV - on a dictionary file. That was soon rectified. The second was a null key which caused the UDH to report a LHReadNextKeyIdGroup error with an unfeasibly large error code.

Now we had the physical errors sorted it was time to address the logical. Revelation suggested that what we might be seeing could be a problem with overly long messages.

We were slightly at a loss as to why this might be until they pointed out that in AREV when you call MSG it firstly assumes that the first passed parameter is a row id and tries first to read it from MESSAGES and then SYSMESSAGES before deciding that it was a literal and attempting to display it. The reason this becomes a problem is that in 4.6 maximum row id lengths are enforced and generate errors if exceeded.

We set about creating proof of this hypothesis and wrote a program to display an increasingly long message.

and sure enough we dropped to the debugger

in RTP32! Hurrah! And what was the magic number?

553 - one more than the maximum allowed key length of 552.

So now we had the problem the solution became obvious (well talking it over with colleagues helped - "by the power of Sprezz"....) - we'd just write a shell program to intercept message calls and if they were longer than the longest message key we'd pass over a message map that told MSG that what was being passed was a literal so it didn't need to try reading it. So $MSG was copied to $MSG_RTI and our replacement was created. And as you can see it worked...

We implemented and all seemed to be going swimmingly but just when we thought it was safe to go back into the water we started getting a new batch of errors which seemed to be coming from input validation not direct message calls. Fortunately now that we had our own message shell it was easier to check the program stack and find out the culprit...

It seems that when the user was entering an invalid key there were a number of possible patterns the key could match. So IN.PATTERN was called to see which, if any it passed. On determining that the pattern match had failed PAT_CHECK was called to construct an "English type" failure message. This message was passed to MSG as a msgRec with msg being left blank. So once again MSG_RTI looked at the script and tried to read it as a message - failing as before. So once again all we had to do was check any passed script and if it was longer than our maximum row id length set the literal flag. Once these modifications were made all that was left were legitimate day to day errors.

We present below the code we used which you are free to use at your own risk - Sprezzatura makes no warranties etc etc etc. If you want warranties we'll happily come and help you for our usual daily rate!

Subroutine Msg( msg, msgRec, resp, args )
/*
  Author    AMcA
  Date      14th April 2010
  Purpose   To provide a shell to normal system MSG as with the new UDH
            there are issues if it does not know the first parameter is a
            literal. Given this system will not be changing we have
            simply found the longest message in SYSMESSAGES and MESSAGES
            (29 characters) and told MSG_RTI that if it is longer than
            35 it is a literal. If your system is still in active development
            then you might want to set the MAXIDLEN$ equate to something
            larger to avoid shooting yourself in the foot.
*/
  If Assigned( msg )    Else msg = ''
  If Assigned( msgRec ) Else msgRec = ""
  If Assigned( resp )   Else resp = ""
  If Assigned( args )   Else args = ""

  Equ MAXIDLEN$         To 35
  saveMsgRec = msgRec

  If Len( msg ) GE MAXIDLEN$ Or Len( msgRec< 11 >) GE MAXIDLEN$ Then
    msgRec< 13 > = 1 ;  * Literal - do no reads
  End Else
    msgRec< 13 > = "" ; * Normal - check MESSAGES and SYSMESSAGES
  End

  Call Msg_RTI( msg, msgRec, resp, args )

  msgRec = saveMsgRec

Return

Danger Will Robinson!

2010-04-09T19:43:00.000+01:00

Working with 9.1.1 today I had occasion to use the global search and replace utility built into the Editor++. In this instance I was replacing one resource string with references to another. The search found six programs so I selected them all and told the editor to go ahead and do the replace.

Imagine my surprise to find that what actually happened was that the six programs were completely replaced with blank rows other than the line to be changed. This was definitely not the result I had wanted - especially as it also compiled the programs (at my request) thereby wiping out the existing object code.

Revelation are aware of the issue and hopefully it will be fixed in 9.2. In the meantime exercise extreme caution if using this utility.

BFSs Redux - being an examination of the concept of Base Filing Systems

2010-03-23T21:30:00.000Z

Those of you with long memories may well remember the heyday of Advanced Revelation when it seemed that every week somebody would come out with a new BFS – or “Bond” as we then called it. Starting with Icicle Software and their Lotus 123 bond followed by AS400 bonds, dBase bonds, SQL Server Bonds, Universe Bonds – the list grew and grew. These days though we seem not to hear quite so much about them so we thought that with the forthcoming re-introduction of the SQL Bond in 9.2 it might be time to revisit this subject.

So what is a BFS?

The clue is – as ever – in the title. Base Filing System. A Base Filing System is the lowest level of file system available in a Revelation environment. Ultimately it is an abstraction of all of the operations required to manipulate a particular file format encapsulated into one conceptual black box program. We say “conceptual” because there is nothing stopping a BFS being made up of more than one program – a classic example of this is the Linear Hash BFS, which uses both RTP57 and RTP57A to implement Linear Hash.

Consider the problem faced by Revelation back in the day. They wanted to create a tool set that could work against ANY back end without requiring reprogramming. Such a tool set would be a boon for developers and users alike. So they came up with the concept of a BFS. They created a list of primitive codes that a filing system would need to support and the codified this in the BFS standard. Basically they said “if you respond in the following manner to the following requests we really don’t care what it is you’re talking to on the back end”.

So how does it work?

In the first instance we need to provide OpenInsight with some information about the data we're trying to access. If you think about it, all that OI needs to be told is just the name of a data row that defines both the location we're interested in and the name of the BFS to use. And this is exactly what a volume pointer is. Now admittedly it might need some additional information such as where to keep shadow dictionaries but location and the BFS are the main constituents.

Attaching a volume could be mapped as follows :-

So let's describe a simple application of this by reference to an imaginary BFS called dBASE.BFS which is a program that has been created to read and write dBASE files. We can obviously store dBASE files anywhere on the disk so we need to tell OI their exact location. Let's assume that we're going to keep them in DATAVOL - yes we can mix and match different file types at the same location because we're using different BFSs. So to "dry run" the above diagram we firstly create a volume pointer called "DBASE_INFO" that tells OI that the BFS is called DBASE.BFS and that the location is called DATAVOL...

Attach a volume - we ask OI to attach DBASE_INFO

Read Volume Pointer - OI reads the volume pointer and gets back DBASE.BFS : @Fm : DATAVOL

Determine BFS - OI takes field one

Determine location - OI takes field two

Call BFS passing location - a series of calls are placed to DBASE.BFS asking it to open the media map for the location, select the rows in the map and return these rows. In actuality the DBASE.BFS would implement this by doing an InitDir(Location : "\*.DBF") and returning the file list

Update SysTables - SYSTABLES would have the names of the DBF files added into the array along with the fact that they use the DBASE BFS

So now if you looked at SYSTABLES you would find rows for each DBF table detaiing its location and the fact that it uses the DBASE.BFS. For the sake of argument let us assume that one such row was called MEMBERS.DBF.

Now when we come to want to open the file for any reason RTP9 (the routine responsible for opening files) would be called with the parameter MEMBERS.DBF. It would read the row from SYSTABLES and establish that the BFS used for that file was DBASE.BFS. It would then call DBASE.BFS passing it the location and the name of the file to open - so DATAVOL and MEMBERS.DBF. DBASE.BFS would simply validate that the file existed and if it did it would return a file handle which could have any structure that the DBASE.BFS wanted as long as it followed the syntax BFS_NAME : @Vm : HandleIdentifyingFile. In its most simple form this could be DBASE.BFS : @Vm : MEMBERS.DBF. This resulting handle would be put into the variable that the file was opened to and would become our file handle.

So now we have a file handle. When we come to read or write to the file handle OI is completely agnostic as to what the BFS actually is. It doesn't care if we are reading or writing Linear Hash, DBASE or SQL. The engine in its role as opcode interpreter simply comes across the opcode for, say a read and says "Oooh a read - so the next opcode will identify the file handle, the next opcode will identify the row id and the next opcode will identify the variable to place the resultant read in". It then looks at the file handle and effectively says

FirstBitOfFileHandle = FileHandle<0, 1>
Call @FirstBitOfFileHandle(Read, FileHandle, RowId, Record, Unused, Flag)

which you'll recognise if you've ever written an MFS. It is then down to DBASE.BFS to implement the read as it sees fit - as long as it sets the Flag variable to indicate a successful or unsuccessful operation (and in the event of it being unsuccessful ideally sets @File.Error to indicate the nature of the failure) OI will be perfectly happy.

So in conclusion - BFSs abstract the logical layer from the physical layer and OI tools work with the logical layer rather than the physical layer. If the BFS is well written then all of OI's data manipulation tools (Window, Popups, Rlist, Basic+ etc) will just assume that the underlying filing system is effectively linear hash.

UTF8 - The BRemove statement

2010-03-20T17:59:00.002Z

Recent changes to the "[]" operators in OpenInsight 9.2 have resulted in substantial performance improvements to UTF8 mode string handling. This post highlights another such enhancement introduced in 9.2 to help bring UTF8 mode applications up to the standard of their ANSI counterparts.

Consider the Loop/Remove construct below:

0001  /*
0002     Example showing standard loop/remove construct used
0003     to parse dynamic arrays at high speed
0004  */
0005  
0006     mark = 1
0007     pos  = 1 ; * // This is the CHARACTER position
0008     Loop
0009        Remove nextVal From dynArray At pos Setting mark
0010        
0011        // Process nextVal...
0012        
0013     While mark
0014     Repeat 

This is a common way to efficiently parse dynamic arrays in Basic+, but just like the normal "[]" operators it suffers from a severe performance degradation in UTF8 mode due to the need to find the byte offset of a character when given the position.

To alleviate this Revelation have introduced the BRemove statement - this operates in exactly the same fashion as the normal Remove statement, but the index variable used in BRemove refers to a byte offset rather than a character position.

Here is the same example rewritten to use BRemove:

0001  /*
0002     Example showing UTF8-friendly loop/remove construct used
0003     to parse dynamic arrays at high speed
0004  */
0005  
0006     mark = 1
0007     pos  = 1 ; * // This is the BYTE offset
0008     Loop
0009        BRemove nextVal From dynArray At pos Setting mark
0010        
0011        // Process nextVal...
0012        
0013     While mark
0014     Repeat 

As you can see it's a simple change and one worth making - using BRemove in your UTF8 applications will ensure that your dynamic array parsing remains fast and efficient.

Indexing in OpenInsight Part 3 - How transactions get updated

2010-03-13T10:43:00.003Z

So now we’ve covered how the index transactions get put into the bang table all that is left is to discuss how they move from the bang table as transactions into the bang table as balanced index nodes.

In the first article we explained that transactions were introduced to allow slow hardware to distribute the transaction processing. In addition to this the engineers at Cosmos had to come up with a way of allowing individual workstations to use spare processing power (when the PC was left unused) to move the transactions from the bang file into the index itself in a way that was easily interruptible if the user wanted to take control of their PC again. This being the case they opted not to move transactions straight from the bang file into the indexes as this could be an intensive operation.

Before pressing on with an explanation of this let’s briefly review what the transactions actually contain. At this stage we’re not going to explain the precise structure of the index transaction rows, just the concepts behind them. As part II explained transaction records are made up of the changes to the indexed columns, specifically the row id of the row that has changed, the column that has changed and the old and new values of the indexed column. If time were no constraint each transaction row could be picked up and all of the indexes referenced therein be updated before returning control to the user. However it is unlikely that a user would be prepared to wait this long so the process has been subdivided into tasks.

There are essentially two tasks – move the transactions from generic transactions to indexed column specific transactions and finally move the index specific transactions into the index, removing the old value if appropriate and inserting the new. This is achieved using three routines :-

    REV_BGND_UPDATE
    F.DISTRIBUTOR
    F.INDEXER

REV_BGND_UPDATE
This is the routine that runs when the system is idle. It works through the indexed files in the system – seemingly using the system variable @INDEX.TIME which has three fields – field one contains an @Vm delimited list of indexed tables, field two contains the table number to start on and field three contains the pointer to the indexed column to work on. It is responsible for calling F.DISTRIBUTOR and F.INDEXER as required.

F.DISTRIBUTOR
This is the routine that moves the generic transactions (0, 1, 2, et al) into column specific transactions (e.g. NAME, NAME*1, NAME*2 et al).

F.INDEXER
This is the routine that takes the columns specific transactions (be they BTree, Relational or Computational) and updates the appropriate index row.

UTF8 and the extended string operators

2010-03-12T13:21:00.007Z

As was pointed out in a recent post the performance of the "[]" string operators in UTF8 mode is pretty poor. In fact it's downright painful - If you've not seen the effects before go and create yourself a UTF8 application and then try compiling a program. The speed drop you see is due to the system pre-compiler (REV_COMPILER_EXPAND) making heavy use of the "[]" operators during the compilation process in a manner similar to this:

0001  /*
0002     This is the usual way of implementing fast string parsing in Basic+.
0003     We scan along the string looking for a delimiter, and remember
0004     where we found it via the Col2() function. For the next iteration
0005     we increment that position and scan from that point.
0006  */   
0007     src = xlate( "SYSPROCS", "MSG", "", "X" )
0008  
0009     pos = 1
0010     eof = len( src )
0011     
0012     loop
0013        token = src[pos," "]
0014        pos   = col2() + 1
0015        
0016        * // Do stuff...
0017        
0018     while ( pos < eof )
0019     repeat

The problem is caused by the need to find the correct starting character position before any string processing can take place. Because UTF8 is a multi-byte encoding scheme it is necessary to start looking from the beginning of the string to find the byte offset of the specified character, as it's possible for a character to be encoded in more than one byte. As you can appreciate, parsing a large string over many iterations wastes a lot of time looking for the character at the specified position - we could get much better performance if we had some way to access the actual byte offset and pass that to the "[]" operators instead.

Well, the good news is that with the upcoming release of OpenInsight 9.2 Revelation have addressed this problem by extending the "[]" operators and adding two new functions to allow access to the byte offset: BCol1() and BCol2().

BCol1() and BCol2()

The usual way to access the position of delimiters found with the "[]" operators or the Field() function is to use the Col1() and Col2() functions, which return the character position. The new BCol1() and BCol2() functions work in a similar fashion but return the byte offset instead, so you know how many bytes from the beginning of the string that a particular character was found.

The extended "[]" operators

Although BCol1() and BCol2() allow access to the byte offsets they can't be used with a normal "[]" operator because it expects the character index as the first argument not the byte offset. The extended "[]" operators take an extra argument (a simple "1" as a flag) to indicate that the first argument is a byte offset, and can be used like so:

0001  /*
0002     This example shows a UTF8-friendly way of parsing a string using
0003     byte offsets with the extended "[]" operators. 
0004  */
0005     src = xlate( "SYSPROCS", "MSG", "", "X" )
0006     
0007     pos       = 1
0008     delim     = " "
0009     delimSize = GetByteSize( delim )
0010     eof       = GetByteSize( src )
0011     
0012     loop
0013        token = src[pos,delim,1]    ; * // Extended - note the last "1" argument
0014        pos   = BCol2() + delimSize ; * // Get the byte offset and increment by
0015                                    ; * // the delimiter _byte_ size
0016        
0017        * // Do stuff...
0018        
0019     while ( pos < eof )
0020     repeat

(Note also that we check the byte size of the delimiter we are using - although we *know* that a space is 1 byte in both ANSI and UTF8 modes, it's good practice to check this at runtime in case you ever end up using a delimiter that is multi-byte encoded instead)

Both Field() and the normal "[]" operators update the BCol1 and BCol2 offsets, as well as the normal Col1 and Col2 positions. The extended "[]" operators only update the BCol1 and BCol2 offsets for obvious reasons.

[EDIT: 20 March 2010] To maintain naming consistency with other UTF8-related enhancements the Col1B and Col2B functions have been renamed to BCol1 and BCol2 - this has been changed in the post above.

UTF8 - To Malloc or not to Malloc - that is another question

2010-03-10T10:03:00.000Z

In our recent post on using memory pre-allocation when building large strings commenter M@ pointed out quite correctly that using the normal [] operators while in UTF8 mode results in a severe performance hit due to the necessity of calculating the character position of the insertion point during each iteration.

A workaround that was suggested was to temporarily switch to ANSI mode for the [] operation and then switch back afterwards. This is a valid solution and one we've used ourselves before, but it does create a possible failure point: If your system hits a fatal debug condition before you switch back you might unknowingly be stuck in ANSI mode which could result in subsequent data corruption.

A safer alternative to this is to use the PutBinaryValue function that we documented here - this ignores any string-encoding and does a straightforward binary copy to the specified offset.

Here's the Preallocation sample program from the previous post updated with the binary functions:

Subroutine ZZ_SpeedTest( Void )

   Declare Function TimeGetTime

   startTime    = TimeGetTime()
   stringLength = GetByteSize( @Upper.Case : @Fm )
   totalLength  = stringLength * 99999
   newArray     = Space(totalLength)
   arrayPtr     = 1

   For loopPtr = 1 To 99999
      PutBinaryValue( newArray, arrayPtr, CHAR, @Upper.Case : @Fm )
      arrayPtr += stringLength
   Next

   endTime   = TimeGetTime()
   totalTime = endTime - startTime

   Call Msg(@Window, "Total time was " : totalTime)

Return

This option took 95 milliseconds in UTF8 mode in our testing. Pretty much on a par with the [] operator in ANSI mode (As a aside the [] operator in UTF8 mode took....... well we don't know actually - we gave up after 10 minutes of waiting for it to finish!)

We also tested the concatenation (:=) option in UTF8 mode - this slowed down the program by half - better than the [] operators but still not great.

To Malloc or not to Malloc - that is the question

2010-03-09T23:21:00.001Z

As part of one of our conference presentations we're showing log file analysis in OpenInsight. If you've ever tried any of this you'll know that the files can get pretty big pretty quickly and trawling through them extracting substrings and building new arrays can be a time consuming process.

For one particular application we had to build a field mark delimited array from the data we were retrieving from file and - conscious of the penalty incurred by using <-1> operators - we were merrily using := String : @Fm syntax. Despite this attempt at efficiency the operation was still woefully slow - not so much as put the kettle on slow as cook gourmet dinner from scratch, eat and wash up slow.

So we had to find a way to speed up the operation. Fortunately as we've posted on the Revelation forum before there is an easy way to do this - preallocating memory. Let's compare the rival techniques and see how we get on.

The <-1> Operator

Subroutine ZZ_SpeedTest( Void )

   Declare Function TimeGetTime

   startTime = TimeGetTime()
   newArray  = ""

   For loopPtr = 1 To 99999
      newArray<-1> = @Upper.Case
   Next

   endTime   = TimeGetTime()
   totalTime = endTime - startTime

   Call Msg(@Window, "Total time was " : totalTime)

Return

This option took 4207 milliseconds

The := Operator

Subroutine ZZ_SpeedTest( Void )

   Declare Function TimeGetTime

   startTime = TimeGetTime()
   newArray  = ""

   For loopPtr = 1 To 99999
      newArray := @Upper.Case : @Fm
   Next

   newArray[-1,1] = ""
   endTime   = TimeGetTime()
   totalTime = endTime - startTime

   Call Msg(@Window, "Total time was " : totalTime)

Return

This option took 4164 milliseconds

The Preallocation Option

Subroutine ZZ_SpeedTest( Void )

   Declare Function TimeGetTime

   startTime    = TimeGetTime()
   stringLength = Len(@Upper.Case : @Fm)
   totalLength  = stringLength * 99999
   newArray     = Space(totalLength)
   arrayPtr     = 1

   For loopPtr = 1 To 99999
      newArray[arrayPtr, stringLength] = @Upper.Case : @Fm
      arrayPtr += stringLength
   Next

   endTime   = TimeGetTime()
   totalTime = endTime - startTime

   Call Msg(@Window, "Total time was " : totalTime)

Return

This option took 81 milliseconds - so 50 times quicker - and the bigger the string you are creating, the more impressive the improvement in speed.

The reason for this is simple - when we concatenate to a string the engine has to grab more memory using a process called a malloc - memory allocation. These operations are resource intensive as they need to juggle memory around to make room for the new string. Resource intensive operations are, by their very nature, slow. By preallocating the space needed we do all of our mallocing in one fell swoop and can concentrate on the task at hand.

EditTable Cookbook - Handling Large Arrays

2010-03-08T07:20:00.000Z

Way back in the days of OpenInsight 7.2.1 the EditTable was modified so that the speed of setting large amounts of data via the LIST or ARRAY properties was significantly increased.

What was not changed was the speed of getting data via LIST or ARRAY and this can have a impact on the setting speed if you're not careful, because each Set_Property operation performs an implicit Get_Property regardless of whether or not you actually want the original data.

For example, if you already have 10000 lines of data in your EditTable and you want to use the LIST or ARRAY property to set new data you may still see a speed drop as the system uses the slow Get_Property to retrieve the original data before the update.

To overcome this problem you can tell the EditTable to clear the existing data before you use Set_Property, that way the implicit Get_Property has nothing to process. This can be done with SendMessage() and the DTM_RESETDATA message like so:

0001     equ DTM_RESETDATA$ to 1025
0002
0003     hwndEdt = get_Property( @window : ".TABLE_1", "HANDLE" )
0004     call sendMessage( hwndEdt, DTM_RESETDATA$, 0, 0 )
0005
0006     call set_Property( @window : ".TABLE_1", "LIST", lotsOfStuff )

UTF8 and Binary Manipulation

2010-03-03T09:45:00.009Z

One of the most important points to bear in mind when using the Basic+ string handling functions is that all normal string operations are character-based - not byte-based. This has major implications if you wish to manipulate your data in a byte-oriented fashion when in UTF8 mode, because UTF8 is a multi-byte encoding scheme; i.e. it doesn't always follow that one byte represents one character as is the case in ANSI mode.

To overcome this issue Revelation introduced several new Basic+ functions way back in OpenInsight 7.0 that explicitly allows binary manipulation regardless of the string-handling mode you are currently in (Note that these functions are intrinsic to the Basic+ language and do not need to be declared before use).

These functions are:

GetByteSize
GetBinaryValue
PutBinaryValue
CreateBinaryData

The intention of this blog post is to document these functions and to make you aware of them so that you can develop your applications correctly should you wish to work in UTF8 mode.

(Also note that most of these functions expect you to specify a variable type when using them. This type should be chosen from one of the standard "C" types understood by the Basic+ compiler and listed at the end of this post)

GetByteSize

Returns the number of bytes occupied by the specified variable. This is in contrast to the Len() function which returns the number of characters.

sizeInBytes = GetByteSize( varData )

Argument	Description
varData	Variable to query.

E.g.

rec = Xlate( "SYSOBJ", "$WRITE_ROW", "", "X" )
recSize = GetByteSize( rec )

GetBinaryValue

This function extracts a binary value from a variable at a specified offset. You must specify the type of data to extract, and if you are extracting a type with a variable length, such as a string of bytes, you must also pass the number of bytes you wish to copy.

binVal = GetBinaryValue( varData, byteOffset, varType, [,noOfBytes] )

Argument	Description
varData	Variable to extract the binary value from.
byteOffset	1-based offset into varData to extract the binary value from.
varType	Type of data to extract. This must be one of the Basic+ "C" types as listed below.
noOfBytes	Number of bytes to extract. This argument is only required if varType is CHAR or BINARY.

E.g.

   rec = Xlate( "SYSOBJ", "$WRITE_ROW", "", "X" )

   // Get the first byte of the record as a number
   firstByte = GetBinaryValue( rec, 1, BYTE )

   // Get the next 10 bytes as a binary string
   someBytes = GetBinaryValue( rec, 2, BINARY, 10 )

PutBinaryValue

This subroutine modifies a variable by replacing binary data at a specifed byte offset. You must specify the type of data you wish to insert as well as the data itself.

PutBinaryValue( binData, byteOffset, varType, varData )

Argument	Description
binData	Variable containing binary data to modify.
byteOffset	1-based starting starting position to begin the modification from.
varType	Type of data to copy into binData. This must be one of the Basic+ "C" types as listed below.
varData	Data to copy into binData. OpenEngine converts this to the binary format specified by the varType argument before copying.

E.g.

   * // Example showing how to access and update
   * // a Windows API structure using
   * // the binary operators.
   * //
   * // A RECT structure consists of four LONG types
   * // (32-bit signed integer, each 4 bytes long)
   * //
   * // typedef tagRECT{
   * //   LONG left,
   * //   LONG top,
   * //   LONG right,
   * //   LONG bottom
   * // } RECT;

   * // We're going to use the GetWindowRect API function
   * // to get some RECT coordinates

   hwnd = Get_Property( @window, "HANDLE" )
   rect = blank_Struct( "RECT" )
   rect = GetWindowRect( hwnd, rect )

   * // Increment the top member by 10
   top  =  GetBinaryValue( rect, 5, LONG )
   top  += 10

   PutBinaryValue( rect, 5, LONG, top )

CreateBinaryData

This function creates and returns a "blank" binary variable of the specified type.

binVal = CreateBinaryData( varType, varData )

Argument	Description
varType	Type of variable to create. This must be one of the Basic+ "C" types as listed below.
varData	Initial value of the new variable.

E.g.

   * // Create a binary integer with an initial value of
   * // 100

   a    = "100"
   intA = CreateBinaryData( INT, a )

Basic+ "C" types

The following is a list of variable types that may be used with the Basic+ binary manipulation functions described above.

CHAR
BYTE
UBYTE
SHORT
USHORT
LONG
ULONG
FLOAT
LPVOID
LPCHAR

LPBYTE
LPUBYTE
LPSHORT
LPUSHORT
LPLONG
LPULONG
LPFLOAT
LPDOUBLE
DOUBLE
HANDLE

INT
UINT
LPINT
LPUINT
LPHANDLE
ACHAR
WCHAR
LPACHAR
LPWCHAR
LPSTR

LPASTR
LPWSTR
BINARY
LPBINARY

[EDIT: 05 March 2010]

Due to a recently discovered compiler bug (since fixed) the following "C" types will NOT work with the binary manipulation functions prior to OpenInsight 9.2.0:

ACHAR
WCHAR
LPACHAR
LPWCHAR
LPSTR
LPASTR
LPWSTR
BINARY
LPBINARY

Probably the biggest impact this will have is processing BINARY types, but you can work around this by using the CHAR type instead as they both perform exactly the same operation.

EditTable Cookbook - Trapping Mouse Messages

2010-03-01T12:05:00.000Z

Getting notification of mouse messages has always been something of a problem with OpenInsight EditTable controls. With other controls this process is quite easy - you simply qualify the WINMSG event with the relevent mouse message number and you can easily respond to it, but with an EditTable this process fails.

This is mainly due to the architecture of the EditTable itself - it's not really a single control, it's actually two: The visible control that you interact with (known as the "DataTbl" control) and a very thin parent wrapper around it (called the "Editable" control).

When you use an EditTable control in your application OpenInsight creates the wrapper control which in turn creates the visible "DataTbl" control. When you interact with an EditTable in Basic+ you are interacting directly with the wrapper - it simply passes on your request to the "DataTbl" as appropriate. If you qualify the WINMSG event on an EditTable you are qualifying against the wrapper - you are NOT qualifying against the visible "DataTbl" control!

When the user interacts with the EditTable control they communicate with the visible "DataTbl", so it is this part that receives the mouse messages. These messages are interpreted and the relevant notifications passed up to the wrapper and then onto OpenInsight - they are not passed directly, so you will never be able use a WINMSG event with the usual mouse messages.

However, since OpenInsight 9.1 it has been possible to trap mouse messages by another means. In this version, when the "DataTbl" receives a mouse message, it actually sends a notification message to the wrapper that you can pick up with a WINMSG event. The mouse message itself is simply offset by the value 3124 (WM_USER + 2100).

For example to detect a "Right Mouse Button Down" message you would qualify against WM_RBUTTONDOWN + 3124 like so:

   $insert logical

   * // From the Windows SDK headers:
   equ WM_USER$                   to 0x0400

   equ WM_LBUTTONDOWN$            to 0x0201
   equ WM_LBUTTONUP$              to 0x0202
   equ WM_LBUTTONDBLCLK$          to 0x0203
   equ WM_RBUTTONDOWN$            to 0x0204
   equ WM_RBUTTONUP$              to 0x0205
   equ WM_RBUTTONDBLCLK$          to 0x0206
   equ WM_MBUTTONDOWN$            to 0x0207
   equ WM_MBUTTONUP$              to 0x0208
   equ WM_MBUTTONDBLCLK$          to 0x0209

   * // Offset value
   equ ETM_MOUSEMSGOFFSET      to (WM_USER$ + 2100) ; * // 3124

   * // EditTable Mouse message notifications
   equ ETM_LBUTTONDOWN$         to (ETM_MOUSEMSGOFFSET$ + WM_LBUTTONDOWN$)
   equ ETM_LBUTTONUP$           to (ETM_MOUSEMSGOFFSET$ + WM_LBUTTONUP$)
   equ ETM_LBUTTONDBLCLK$       to (ETM_MOUSEMSGOFFSET$ + WM_LBUTTONDBLCLK$)
   equ ETM_MBUTTONDOWN$         to (ETM_MOUSEMSGOFFSET$ + WM_MBUTTONDOWN$)
   equ ETM_MBUTTONUP$           to (ETM_MOUSEMSGOFFSET$ + WM_MBUTTONUP$)
   equ ETM_MBUTTONDBLCLK$       to (ETM_MOUSEMSGOFFSET$ + WM_MBUTTONDBLCLK$)
   equ ETM_RBUTTONDOWN$         to (ETM_MOUSEMSGOFFSET$ + WM_RBUTTONDOWN$)
   equ ETM_RBUTTONUP$           to (ETM_MOUSEMSGOFFSET$ + WM_RBUTTONUP$)
   equ ETM_RBUTTONDBLCLK$       to (ETM_MOUSEMSGOFFSET$ + WM_RBUTTONDBLCLK$)

   * // To trap a right button down message:
   call Send_Message( @window : ".TABLE_1", "QUALIFY_EVENT", ETM_RBUTTONDOWN$, TRUE$ )

A week in the life of a Revelation Consultant

2010-02-20T11:22:00.000Z

I've just spent one of the most frustrating weeks of my life working on an AREV 2.12 to AREV 3.12 conversion prior to an OI Upgrade. The upgrade itself was relatively painless but the application itself then began to fail in mysterious ways - the worst of which was in a custom program designed to display grids of information. This program - checking in at nearly three thousand lines of code - just wouldn't display the current line in the grid correctly - everything was off by 4 bytes.

Well as the object code hadn't changed it had to be something that had changed between AREV 2.12 and 3.12 right?

So I started comparing @Variables between the two systems and found a few likely contenders - @CRTHIGH for example. But try as I might nothing worked. Finally as Friday evening approached a small light went off in my head. The program referenced @ATRBT and this doesn't actually compile in AREV 3.12...

So all we had to do was load a non-existent variable with its old value and then everything would work... suffice to say this proved challenging to say the least but eventually I had a routine that loaded what @ATRBT used to be and lo and behold the recalcitrant program now worked.

It's not very often that you have to know about the absolutely obscure aspects of system internals but when you do you're grateful for the knowledge.

Installation Procedures for an OpenInsight Thin Client System

2010-02-10T10:45:00.001Z

From the Sprezz archives we dig out this venerable document as it still seems to have relevance to our customers in the marketplace today!

The new Universal Driver 3.0 from Revelation Software gives customers new methods of formatting and structuring the location of data files.

One of these methods is the ability to run OpenInsight applications in a thin client model. For the purposes of this document, “thin client” means storing the repository files (or REVBOOT) on a local workstation while storing the datafiles on a remote server. This relies on the hiding files concept of the Universal Driver, and requires some initial configuration of the local system and the registry on the server the UD is installed on.

This document will describe how to move a system from your local development system to a client’s site. During development, we will have everything stored in one path. For example, OpenInsight sits in the following directory path:

S:\APPS\TABLET\OINSIGHT\RELEASE

This is the directory that contains the OINSIGHT.EXE, which we’ll refer
to as REVBOOT. Underneath, we have all of our data volumes, stored in
the DATA volume.

S:\APPS\TABLET\OINSIGHT\RELEASE\DATA\CUSTS
S:\APPS\TABLET\OINSIGHT\RELEASE\DATA\PRODS
S:\APPS\TABLET\OINSIGHT\RELEASE\DATA\INVS

When the application gets installed at the client site, the client will install
this in two sections, on the client’s server, which we’ll call SVR1PHL, the entire application will be installed on the D:\WINAPPS\3RDPTY directory. (Installing the entire application allows for server side maintenance of the data, but is
optional.) This means that we’ll have paths like

D:\WINAPPS\3RDPTY
D:\WINAPPS\3RDPTY\DATA\CUSTS
D:\WINAPPS\3RDPTY\DATA\PRODS
D:\WINAPPS\3RDPTY\DATA\INVS

On each workstation, the user only needs a REVBOOT directory and single directory called DATA. Following standard Windows™ conventions for program installation, our client workstation would probably have installed this section as C:\Program Files\My Company\TabletApp

Everything up to this point gets installed and placed as would be expected. The data is on the server, the programs and windows are stored on the local workstation. What we need to do now is tell the workstation how to find the files.

There’s a new attribute of the REVPARAM file which allows the definition of a share name to store data. This attribute is called ShareName. This is used in conjunction with the ServerName attribute. What we need to do is create a DATA subdirectory underneath the REVBOOT of our workstation copy. This gives us

C:\Program Files\My Company\TabletApp\Data

In that subdirectory, we put in a single file, REVPARAM, with the following entries:

TcpIpPort = 777
ServerOnly = 1
ServerName = SVR1PHL
ShareName = DATA

Because of the path tree method of the UD driver, when we go to access any LH volume in this directory (or lower), the driver will find the REVPARAM and use that for instructions.

The UD driver finds the paired attributes ServerName and ShareName and determines that whenever the user wants anything from the DATA volume, it needs to go to the SVR1PHL server to get the files. This means if the user looks for DATA\PRODS, the UD driver will query the SVR1PHL server for the DATA\PRODS directory and use that as it’s file store.

We also need to tell the server how to find this data, and for this, we need to make some additional modifications on the server. The first step is to go into the server’s registry and create some additional keys. Go to the registry location for the Universal Driver settings, which is

HKEY_LOCAL_MACHINE\SOFTWARE\RevSoft\Revelation Universal Driver\3.0.

Underneath this location, we need to create a new subkey called “shares”. This gives us

HKEY_LOCAL_MACHINE\SOFTWARE\RevSoft\Revelation Universal Driver\3.0\Shares.

Inside this section, we need to create a new string value called DATA, with
a value of D:\WINAPPS\3RDPTY\DATA.

This tells the LH Service that when it gets a request for the volume named DATA to start at this location. Because this also cascades down the directory tree, if the LH service gets a request for the DATA\CUSTS directory, it will look for the files in D:\WINAPPS\3RDPTY\DATA\CUSTS.

There’s one more step, and that’s the creation of an additional REVPARAM file, this time in the D:\WINAPPS\3RDPTY\DATA directory. This REVPARAM file contains the following:

TcpIpPort = 777
ServerOnly = 1

No other REVPARAM file is needed on the server.

That’s all that’s needed to find and run OpenInsight in a thin client mode.

How did this all work? This is possible through the initial REVPARAM created on the workstation. As far as the local workstation is concerned, all the data files are stored in directories below REVBOOT. As this tree is traversed, the UD Driver comes upon a REVPARAM file. In this REVPARAM file, there are the ServerName and ShareName attributes. This tells the UD driver that there’s no point in trying to find the files at this location, but to send a message to ServerName and ask for the data in ShareName. When ServerName receives the request, it queries its registry for a ShareName matching ShareName, and finishes the directory traversal from that point.

By extension, using the volumes we defined above, we could have on the workstation, the following:

C:\Program Files\My Company\TabletApp\Data\CUSTS
C:\Program Files\My Company\TabletApp\Data\PRODS
C:\Program Files\My Company\TabletApp\Data\INVS

Each of these folders could have a revparam pointing to a share on different servers. This would allow the CUSTS volume to exist on SVR1PHL, the PRODS volume to exist on SVR2PHL and the INVS volume to exist on SVR1NHMP.

Sometimes it's good to be reminded...

2010-02-09T11:14:00.001Z

of just how productive you can be using Revelation's tool set.

Last week I spent a day on client's site as he had some issues of stability that he wished to resolve. We spent the morning going through these then, as afternoon approached, he asked "OK can you knock me up a copy protected installation routine for me to ship my product with?".

Here's where a Revelation/Open Source Community mashup really comes into its own.

Firstly download NSIS - we have the engine to create a full SETUP.EXE.

Next take the excellent NSIS installation script provided at one of the conferences by our friends at SRP and spend 5 minutes customising it - we have the working OI Installation routine.

Next install Sprezzatura's registry method handling routines registry_methods.zip - we have the wherewithal to copy protect the application by embedding encrypted registry keys.

Finally write a new front end screen for the application that compares the Windows computer name (using the GetComputerName declaration in DLL_KERNEL32) to an encrypted version thereof in the registry and if found launches the app, if not launches a newly written registration screen. Have the registration screen display an encrypted string to read to the client's tech support line to receive back an unlock code which permits the stamping of the registry. We have the solution.

Now we're not claiming this is bullet proof (bullet proof we can do but it'd take longer) but it's one heck of a lot of return for 4 hours of invested time!

Indexing in Opensight Part 2 - How index transactions get created

2009-12-18T12:23:00.002Z

In the last article we provided a generic overview of how indexing works. In this article we look at how the index transactions get into the bang table, and in the soon to be published followup article we look at how they get moved from being transaction rows to actually being meaningful data in the indexes.

Index transactions are created by a modified filing system called SI.MFS – the Secondary Indexing MFS. The following description assumes that this is the first time that an index has been added to a table.

When the developer adds an index to a column, SI.MFS is added to the table to ensure that the index transactions are created. The actual mechanism by which this happens is that a flag is set in the dictionary row for the column indicating what kind of index is being created along with its case sensitivity. The MFS responsible for looking after dictionaries, DICT.MFS, sees this flag and calls MAKE.INDEX to ensure that SI.MFS is added to the table. In addition it creates the ! table and populates the initial control information describing the index structures.

The workhorse in the creation of the index transactions is a program stored in the ! file with a row id of ! – generally referred to as the bang code in the bang file/table. This program code is structured as a symbolic dictionary item – so the object code is on attribute 8 of the row. The code itself looks at the row being written and compares it to the row on disk. If indexed columns have changed then it generates index transaction rows and puts them into the ! table.

We can find out a lot about how this ! code works by looking at the source code for it and fortunately for us Revelation have made it very easy for us to examine it. Under normal circumstances the ! code is generated and compiled and written away without preserving the source code BUT if we as developers create a blank row with an id of !! in the ! table, the system will see this as an instruction to preserve the source code in the !! row. So if you wanted to examine specific ! code you would just take the following simple steps

• Delete the ! code from the ! table
• Create a blank row with an id of !! in the ! table
• Reattach the ! table
• Edit the data table

These steps will leave the source code for the ! row in row !!.

We generated some sample code ourselves and used it to create the following flowchart which explains the process in detail :-

So explaining this flow chart in English...

Assuming the table is indexed, when the write occurs, SI.MFS is called with a WRITE opcode.

Now normally MFSs just do what they need to then call the next MFS in sequence. This carries on until the operation has been executed by the Base Filing System (BFS) and then the chain is reversed out of. SI.MFS doesn’t work this way. It doesn’t actually do the updating of the index transaction rows itself – rather is calls the ! code we discussed earlier. As it’s stored as a dictionary item it can’t call it directly – rather it invokes it using CALCULATEX – the system function for resolving a dictionary item.

The ! code looks to see what it has been asked to do and if it has been asked to do an update it compares the old values with the new values to see what has changed. Normally it’ll try and use a cached copy of the old record but if it isn’t there it’ll reread the row. If it determines that index updates are required, the code reads the 0 row in the ! table and appends the newly generated transactions to this row.

If this forces the length of row 0 over 2000 bytes then it creates a row called XTRANS and writes this out to the ! table. It then grabs the next free transaction row number from the row 0 header and writes out the new transaction row to that number. Finally it updates the row 0 with the new number blanking down the transactions that were there and it deletes the XTRANS row.

If the row 0 isn’t over 2000 characters the new transactions are simply written back to row 0
Finally the ! code attempts to update the relational indexes if any are present by invoking RELATER.

Note that on the flowchart above I've marked a few elements in red. These are areas where it would seem that the index code could be optimised for performance reasons.

When updating a row in which there had been index changes you would expect there to be a read of row 0, a write of row 0 and possibly the creation of a new index transaction row. In fact what seems to be happening is that the 2/3 i/os occur but in addition if Row 0 exceeds 2000 bytes in length then an XTRANS row is created, written then deleted, thereby potentially doubling the disk i/o required for the update. This could have significant performance implications on mass updates.

In addition even if the Relational Index updates correctly the index transactions are still left if the ! file to be reprocessed at a later date. This seems like a waste of resource.

Indexing in OpenInsight – Part 1 – What is indexing?

2009-11-05T12:04:00.000Z

A discussion on the Revelation Forum reminded us that we haven’t reviewed our public documentation of how Revelation indexing works in over 20 years. Whilst the RevMedia article on our website remains largely accurate we thought that this provided an excellent opportunity for us to revisit the subject with our improved understanding.

The first question to ask is “what is an index in OpenInsight”?

Well to answer this strictly we’d have to distinguish between Btree/Xref indexes and Relational indexes as they both are very different in structure. A Relational index by its very nature is just an inverted index (http://en.wikipedia.org/wiki/Inverted_index) and so it easy to maintain – a read of the inverted row, an insertion and a write of the inverted row. A Btree/Xref on the other hand (the two being identical in index structure, the difference being that an Xref is a Btree on sub strings of a column as opposed to a Btree on the entire column) requires a lot more maintenance. For details of how BTree indexes work see http://en.wikipedia.org/wiki/B-tree. The insertion of a new or revised entry into a Btree index could generate a large amount of disk i/o as the tree is rebalanced and the operation is a costly one in resource terms.

When Revelation’s indexing was first introduced, hardware was a lot slower than it is today. When saving a single row it used to be possible to see the disk latency. You’d hit the save key then wait whilst the disk was updated. This was made acceptable in a data entry environment by adding a keyboard type ahead buffer (one of the reasons that Revelation used to ship with a utility to extend the keyboard typeahead buffer from the default 16 to the humungous 256 bytes required by touch typers). However forcing the user to wait whilst the BTree indexes associated with the table were updated would have introduced a completely unacceptable lag.

So the architects of Revelation came up with the idea of not updating the indexes at the point of saving. Rather the fact that the indexes needed to be updated was recorded in a series of transactions rows in what came to be known as the “Bang table” (the separate table named after the source table but prefixed with an exclamation mark) and these were either flushed by idle workstations or by any operations which required the indexes to be up to date. This way the resource intensive balancing operation was pushed off onto non data entry personnel and throughput was maintained.

In our next article we'll look at the way that index transactions get to be in the bang table in the first place.

Blogging Basic+ - Version 1.7.0 update

2009-10-22T11:51:00.001+01:00

Here's another update for those of you using our Basic+ Source code publisher. This time I had a bit of a major rewrite to the HTML generator to handle the crazy 4K chunk HTML parsing in the Revelation Forum Domino server.

Originally I had wanted to keep the generated HTML for this blog and the forum pretty much the same, but alas that wasn't to be, so I had to drop back to "pre" tags again for the forum, and keep the "div" tags for here.

Thanks to Jared at Revelation for his help in tracking down the weirdness.

You can download version 1.7.0 from here in standard RDK format or here as an NSIS installer version.

Windows 7

2009-10-08T17:53:00.002+01:00

Well we've been running with Windows 7 on a couple of our laptops for a few weeks now and we like it! The ability to launch a program by typing just a few letters of its name is brilliant... the only downer was not being able to run AREV at all anymore - oh yes, and the fact that as of the time of writing OpenInsight 9.1 completely hangs - I mean COMPLETELY hangs Windows 7 64 bit.

Note I said "was" because today we installed Virtual PC for Windows 7 along with XP Mode and huzzah - we have an environment where we can actually work with AREV and OI! You can read all about it here.

The only thing to point out is that once you enabled virtualization in your BIOS you have to do a cold reboot - turn the machine completely off. Easily overlooked and potentially frustrating.

[Edit] - The hanging issue is fixed for OI 9.1.1