Home page Home page Home page Home page
Pixel
Pixel Header R1 C1 Pixel
Pixel Header R2 C1 Pixel
Pixel Header R3 C1 Pixel
Pixel
By Sprezz | Friday, 20 May 2016 16:00 | 0 Comments
In what has been one of the most exciting announcements since the trailers for OI 10, Revelation Software have just released UD 4.7.2 and 5.0.0.4. Whilst the small incremental release number might make this seem insignificant it actually marks one of the biggest changes in linear hash since the Universal Driver was released.

For some time now we have been aware of issues with "enterprise" sites where under certain ill-defined circumstances, the network would crawl to a halt and frustrated users would sit helpless until the system eventually recovered. Under certain extreme conditions the system would never recover and the only solution would be to reboot the server with possible ensuing data corruption. After such halts linear hash files would be seen to have "exploded". Growing in size from tens of megabytes to tens of gigabytes.

The suspicion has long been that some form of alpha corruption led to a massive resizing exercise and that this in turn led to the network slowdown.

With the help of a large UK client Revelation staff have finally been able to track this error down and to finally fix it! And while they were under the hood they've made a whole slew of other changes that make fine tuning an enterprise system so much easier!

VSS has been made more reliable.

Resizing has been made programmatically controllable. So a site that experiences peak transaction volumes at specific times can now tell the UD not to do any resizing until it is appropriate to do so. The underlying filing system naturally copes well with this and when the resizing is re-enabled  it can catch up with the necessary tasks.

Size-locking has been revisited and fixed. We have an unpublished blog article about how size-locking was broken in later releases of OI - which made it very difficult to control whether files expanded/contracted or not. This has now been fixed and even better enhanced so that it works with the hidden volume/share feature of the UD so the client no longer needs physical access to the file to set the size-lock.

The UD can now reliably be shut down even if clients are not responding. This fixes an issue where the UD would refuse to close and the server would have to be rebooted to force closure.

A new registry flag has been added called CreateFileFlags. The service expects that, if this registry entry exists, it has a DWORD value. If it’s not set, its default value is the value associated with:

FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS | FILE_FLAG_OVERLAPPED (which evaluates to 0x50000080)

So the developer can now ensure that writes are passed straight through to the disk and not cached. See the Microsoft documentation here.
 
Not forgetting the fact that the UD now supports encryption AND reestablishing dropped connections!

All of us at Sprezzatura salute Revelation for the mammoth efforts they invested into putting this right - along with grateful thanks to the UK client who made this possible.
By Sprezz | Tuesday, 3 May 2016 10:10 | 0 Comments
At the recent conference in Orlando I was given the opportunity to present a very early preview of the sample application that Sprezzatura will be providing for the final release of OpenInsight 10. To make the application simpler for new users to follow and understand how MV can be used to improve on SQL data structures the decision was made to convert an open source SQL bug tracking system (Bugzilla) to OpenInsight 10.


This article represents a distillation of the conference presentation without the rambling asides and with a little more depth given to some of the tools. If you were at the presentation then thank you for being there - but this isn't a PowerPoint of my talk! As you'll know the talk was just a live dem. (And we didn't hit the debugger once!).

To make the conversion easier pre-conference the decision was made to develop the application in 9.4 and then move it over. To keep this blog post focused I won't include the screen shots of the app that were demonstrated at conference but suffice to say that the data tables were populated with a small sample data set (500 clients, 8 products, 5000 bugs, 1-4 incidents per bug) and supporting windows were constructed to allow viewing of these.

An interesting thing to come out of the work undertaken during the conversion was to see just how tight OpenInsight security has been made. At one stage I found myself unable to log into my system to continue working as the system told me that my passwords were wrong. I was slightly taken aback by this as I was fairly sure that I'd remembered them correctly but rather than waste time asserting that I was in the right I decided to cheat. I installed a new copy of the pre-Alpha in a different subdirectory, logged in using the default passwords and set up my accounts and users again. Then I made a note of the system tables used to store the user details and copied those corresponding DOS LK and OV files over to my compromised system.

I was still unable to log in.

I queried this with Revelation and all was revealed. To stop this sort of backdoor entry from happening, passwords are encrypted using a salt value that is NOT stored with the password.This is a significant improvement over previous behaviour. I still remember being at a PC Trade Show with Release C of Revelation when an attendee came up and suggested to me that he could easily break into our password protected system. I scoffed. He TYPEd the ROS file containing all the system details and logged straight in. It's nice to see this sort of security hole well and truly closed!

Back to the product!

The first thing that strikes you about the product is how simplified and clean the interface appears.


As has been explained before - the above window is just an OpenInsight form designed using several of the new translucency features that make UI potentially so much more attractive in 10. Logging in takes you to the launch pad.



Gone are the myriad of tools and differing ways of opening an entity or creating a new one. Now if you want to create a new message, you simply File/New Entity/Message. Whereas if you want to create a new form you have to File/New Entity/Form. Popups on the other hand are created using File/New Entity/Popup... I think you get the idea! The only exceptions to this currently is for entities using the new improved system editor which have their own menu options.

Choosing the File/New Entity launches the Select Entity window from where you can choose the class and type of the entity you wish to create (or in the case of existing entities File/Open Entity). No more do we have to remember the correct tool to use for an entity. Choose the entity and the correct tool launches. (I know this was possible in the Application Manager but you had to know that - it wasn't intuitive). If you know that the entity you want doesn't use the new system editor then just clicking the New or Open folder image will launch the appropriate selection dialog.

The New Entity dialog contains all possible entities



Whereas the Open Entity dialog only shows entities you actually have - you can hardly open something you don't have now can you?!


Message Designer

The message designer has added a few new features which were the subject of a completely separate talk by David Hendershot, so rather than duplicate efforts I simply made reference to the ability of the designer to render simple HTML commands to improve message appearance...



Consistently throughout the product now when a designer is open, a property panel provides access to the properties of the selected control or entity. You'll see here that I've highlighted the icon property and the cursor property. You can change the cursor to be displayed at run time to a standard cursor or to custom ones of your own.

As stated above the HTML selection is limited in the message designer to those tags shown in the image below.


QuickHelp

When QuickHelp was introduced we all hoped that this would remove the burden of creating proper HTML help for users. In AREV we were used to delivering a new system to the end user and training their power user in the use of the system so that THEY could document the system in a way that their colleagues could understand. Unfortunately Windows users are used to better help than plain text so QuickHelp never really took off.

OI 10 addresses this issue and makes QuickHelp editable HTML. So if I define a QuickHelp as follows :-



When I run the QuickHelp it remains editable



but if I set it to read-only the edit controls disappear.



Note also that if you're a real code jockey you can edit the HTML directly by clicking the HTML button at the end of the toolbar.


Form Designer


Where to start with the revised form designer? Well immediately it has to be said that so much has changed that the Form Designer was the subject of several sessions at conference, so my presentation simply highlighted some of the areas that I was most impressed by.

One of the most fundamental  internal changes made to the product has been the way it has been redesigned to be much more "object oriented". Entities have been designed to be reflective so the tool set can ask an entity what properties and methods it supports. No more hard coded lists. You may have noticed that with newer releases of the product, newer properties were not reflected in the form designer - you had to set them at run time. This added to the coding burden in a way that was entirely avoidable if the form designer had allowed access to them at design time. The issue was that the Form Designer had one list of properties and the Presentation Server had another. Keeping them in sync was an issue that was neglected.

With OI 10 all of this has changed. The form designer now exposes EVERY property that is settable meaning that less code has to be written at run time. The properties are exposed in the property panel - so selecting a control allows access to the properties associated with that control :-


If you have multiple controls selected, only the properties common to those controls are displayed in the property panel. So consider EDITFIELD properties and BITMAP properties


The BITMAP obviously has more "Appearance" properties. But if you select an EDITFIELD and a BITMAP together the property panel changes to just "Common Properties" and these common properties can be set by a single adjustment.



Worth noting is the improvement of support for images. Selecting the BITMAP control allows access to the bitmap property dialog where properties of the image in this context can be set. Note that some properties are of the image itself so cannot be set via this dialog.



The property panel also provides access to the events associated with a control via the Events tab


The Events Editor dialog is a work in progress but it has a logical flow that is easily understood. Clicking on an event launches the Events Editor. So clicking on the Click event for the BITMAP control displays an "options button".
 


and clicking that leads to the Events Editor


At first I was confused by this dialog as nothing was showing. But then I noticed the check boxes from which I can select whether I want a quick event or an event script - or both.

Checking the event script check box displays the event editor


and praise be! The event script finally defaults to 1! The amount of grief that the event script defaulting to 0 caused can now finally be avoided!

Checking the quick event check box displays a slightly familiar set of choices


So - if on the double click of the bitmap we decided that we wanted to send an "Options" event to the Clock No edit line we'd do the following...

Firstly select what we want to happen



Then we identify the control we want it to happen to


and finally we identify the event to be sent


So double clicking the bitmap will send an "Options" event to the Clock Number edit line. As for the "Options" event on the edit line - firstly we tell it we wish to execute a repository method against a chosen popup entity


Then we tell it that on return it should populate the current editfield.


So far all well and good - familiar territory. But then a stroke of genius...

How many times have you had to write code to launch a popup and then send a read event to the screen? No more - there's a new "Finalize" section for quick events. What should happen after the quick event has fired? A wrap up quick event as it were.


So with one simple addition creating screens just got a whole lot easier.

There are so many features of the new form designer to get excited about. For now I'll just leave you with a screen shot of some of the  new control types available to us.


An Aside

When I first started using the product I kept looking to the File menu for the "Most Recently Used" menu so I could return rapidly to wherever I was last working. I couldn't find it so queried this with Rev. Suffice to say that sometimes it's easy to miss what's under your nose! The start screen has the MRU embedded within it!



Even better - the open dialog remembers the tool you were working with last and reorders the open dialog to present you with the tool you were lasting with in a style reminiscent of an MRU. See how this open dialog differs from the initial open dialog presented in the article as since then I've worked with messages, quickhelp and forms!


Another nice touch is the tabbed interface showing you all of the entities that you have open allowing quick return to them.


Finally


 Remember reflection?


Well now that tools can simply ask what properties a control exposes the debugger has been re-engineered to take advantage of this! So now ALL properties are available for inspection from the debugger. It now shows all nested controls, all their properties, including UDPs and it has improved fonts so you can work in High DPI.


OpenInsight 10 is a landmark release for us developers and personally I can't wait to get developing real world applications with it. I'd actively encourage you to play with the Alpha available for download for Works members from here.

I look forward to interacting with you all on the Beta forum in due course!
By apk | Tuesday, 26 April 2016 15:28 | 0 Comments
One of our largest customers decided to use the Good Friday holiday to handle some server maintenance.  While normally a 24/7 shop, this is one of the few days of the year that not only are they on a skeleton crew, it's a skeleton of a skeleton crew and they generally receive no calls on this day.

After performing and verifying the backups, upgrading the OS with all the relevant patches, the server was rebooted.  A quick check, and the LH service was not running, which seemed a bit odd mostly because it's never failed to start before.  Starting the service immediately displayed an error stating that the service did not start in a timely manner.  While none of us was completely sure what's considered "a timely manner", we were all in agreement that "a timely manner" was not "immediately".

At this point, we thought it's time to examine the Window's event log for some clues.  However, this proved to be a problem because the event viewer wouldn't load.  This did not seem like a good sign, so we rebooted the server, but there were no changes.  Looking deeper into the system, we found that the event viewer service wasn't starting either.  Now we think there's something horribly wrong with the server and something went wrong with the patch installation.  While some people were checking on what was installed and what side-effects there might have been, others looked into why the event viewer wouldn't work.

Eventually, we worked out that the event viewer wouldn't start because the subdirectory storing the log files was read-only.  Once we set those files to read-write, the event viewer service started.  Once the event viewer started, the LH Service also started.

As near as we can work out, the LH Service couldn't update the event log and immediately failed.  This started an interesting discussion with the client.  How would we have coded for this particular error.  Revelation is correct in that the LH Service should halt when an OS feature so basic as event logging is unavailable.  This points to a fundamental error on the server which should be handled immediately.  There isn't a very good way to log the error.  Normally the error would go into the event log files, but the error was that it couldn't write to the event logs.  The LH service managed to avoid the perpetual loop of writing to the event log, generating an error, which writes to the event log, generating another error.

The most interesting thing in all this was that Windows never bothered to inform us of the error, and all (most?) Windows services loaded.  We really have no idea what started and what failed.  Had the LH service not acted correctly, a major production server could have run for weeks in a potentially inoperable state.

By Sprezz | 15:25 | 3 Comments
This is going to be one of those blog posts that will either have you saying "Well d'uh everybody knows that" or "Wow what a neat trick". I know that I fell into the latter camp which is why I'm blogging it.

Aaron and I are working together on a new tool for network administrators and it has been a fun ride. Recently we had cause to have Aaron TeamViewer into my machine to help debug a specific routine. So I watched as he took over and waited for him to fire up the System Editor, open the program, insert a DEBUG, compile and run. This is how I've traditionally done it and of course you have to be really careful to remember to remove the DEBUG before shipping the code to anybody. (Or run your release through a release checker that warns of the DEBUG opcode in object code). Instead he performed the neat trick documented below. When I expressed admiration he was incredulous. "I've been doing it this way since OI 2.0" quoth he.

So allow me to present an illustrated guide to debugging without DEBUG statements.

Firstly open the engine window


and click on the Debugger button. Now give focus back to the System Editor or any active OI screen in a way that causes it to trigger an event. The debugger will be invoked.


Now from within the debugger, from the File Menu choose "Open Stored Procedure" and select the routine you wish to debug




Scroll through the code and select your break point in the usual way by double clicking on a line of code



then click on the Run button or press F5 to continue.

The system will return to "Normal". Now invoke the program you wish to debug (for simplicity I will just illustrate by running from the System Monitor)



et voilà - the program breaks at the nominated break point.


This is so cool - it's slightly faster than inserting a DEBUG and you don't run the risk of forgetting to remove the statement!

By Sprezz | 15:23 | 0 Comments
Of late at Sprezz Towers we've been all about taking apart Linear Hash frame headers to diagnose some, frankly bizarre, issues being experienced by one of our clients. Our utilities were working fine on "old style" Linear Hash files where the maximum frame size was 64K and the maximum file size was 4GB but on the newer UD3+ files (or "type 3" as they're referred to) we just didn't seem to be getting the consistent results we were expecting.

As an example consider this section of code for examining the "Alpha Space Usage" - the total bytes of data stored in the Primary frames :-



  lo_alpha  = file[alphaPtr$, 4]
  hi_alpha  = file[alphaPtrHi$, 4]

  If hiBitSet Then lo_alpha := hi_alpha

  value = lo_alpha
  Gosub Convert
  lo_alpha = value

  hlo_Alpha = Oconv( lo_alpha, "MX")
    

So this snippet takes the 8 bytes that comprise the Alpha value and converts them into a very large integer.

When the Alpha was a value of 3,349,830,462 the MX conversion returned C7AA5B3E which is indeed correct. The issue arose when the Alpha had increased to 7,644,806,759, then the MX conversion returned an erroneous value and got the hex conversion wrong.

Those of you with knowledge of typed languages can probably see where this is going! As OpenInsight is a 32 bit app, the largest value that can be represented in an unsigned integer is 4,294,967,295, which the 3,349,830,462 is less than - 7,644,806,759 however is over this limit so the MX conversion failed.

Fortunately for us the solution was simple (once we realised that something needed solving. The MX conversion works off an integer, but the HEX conversion works off an ASCII string. So all we had to do was modify our code to use this, and convert the actual string value not the number value as below :-



  lo_alpha  = file[alphaPtr$, 4]
  hi_alpha  = file[alphaPtrHi$, 4]

  If hiBitSet Then lo_alpha := hi_alpha
  hlo_Alpha = Oconv( lo_alpha, "HEX")

  value = lo_alpha
  Gosub Convert
  lo_alpha = value
    

and all was once again right with the world with the HEX conversion returning 677EAAC701000000.

Finally we were able to report on the corrupted file accurately.

Frame length 1,634,013,184 00 10 65 61
Modulo 2,342,159 0F BD 23 00
Alpha 7,644,806,759 67 7E AA C7 01 00 00 00
Threshold 80
Sizelock 0
Row Count 20,379,590 C6 F7 36 01
Alpha% .00

Header 32 0 0 0 0 0 0 0 0 15 189 35 0 0 16 103 126 170 199 204 0 0 198 247 54 1 101 97 1 0 0 0
Header 20 0 0 0 0 0 0 0 0 F BD 23 0 0 10 67 7E AA C7 CC 0 0 C6 F7 36 1 65 61 1 0 0 0

CAVEAT
It should be noted that the code snippet above is not UTF-8 safe. We knew that the data we were working on was purely ANSI and had customised one of our routines accordingly. The square bracket operators, CHAR and SEQ are generally not safe when processing binary data so unless you're 100% sure you're in ANSI mode avoid them. In this case we should have been using GetBinaryValue.
By Sprezz | Friday, 22 January 2016 16:30 | 0 Comments
One of our clients started having issues recently when an online payment system that had worked for well over a decade suddenly started to fail. The error returned from the third party payment DLL was less than useful.

In its entirety it read "".

We spoke to the company's tech support and expressed some surprise that the routine did not return discrete meaningful error messages for all failure points and they basically confirmed that from their perspective - embarrassed cough - this was a valid way of dealing with errors. Well at least they didn't modify the payload parameter to return an error string - small mercies and all that. So we set about debugging this ourselves.

Our first port of call was to instantiate an Msxml2.ServerXMLHTTP object and open the API URL for a POST payload. This worked well and oleStatus() indicated that all was well. But when it came time to deliver the payload using a SEND method things weren't quite so copacetic. Instead of a zero, oleStatus returned a somewhat large negative integer, "-2147012894" to be precise.

Googling this got us nowhere. Unsurprisingly because what is actually being returned is a 32 bit unsigned integer which is being interpreted by OI as a 32 bit SIGNED integer. What does that mean in real life?

Well take the binary string 10000000000000000000000000000001. If this is a signed integer then the bit at the beginning is used to indicate whether the integer is positive (0) or negative (1), leaving 31 bits to represent the integer. In this case the above number is -1. If this is an unsigned integer then the bit at the beginning is part of the integer itself so there are 32 bits to represent the integer. In this case the above number is 2,147,483,649.

The reason that Google searches failed to bring up any useful information on this error is straightforward. Windows errors (HRESULT values) are expressed as hexadecimal numbers. So we need to convert this value into an unsigned 32 bit integer and get the hex value. Fortunately the Windows 10 calculator makes this easy. Simply invoke the calculator and change to Programmer mode.


Now paste in your error value and click on QWORD to make it DWORD


Note the hex value and search for HRESULT 0xhexResult
 

Armed with this information we were in a much better position to track down the culprit - in this case a tightening up of firewall rules!
By Sprezz | Wednesday, 11 November 2015 13:25 | 6 Comments
Many people who initially made the move to AREV32 with the intention of slowly re-engineering their way to a purely OpenInsight based environment still find themselves with systems that are heavily AREV32 reliant. With this in mind we thought it might be an idea to spend a little time examining CFG_CTO to see what little goodies might be lurking in there. So with the help of Revelation's esteemed CTO guru, Bryan Shumsky we've put together this little guide.

CFG_CTO consists (at the time of writing) of 10 columns in a single row. Note that several of these columns are only relevant when working with the CTO rather than AREV32 but for the sake of completeness we will document all of the columns.

Bear in mind that when looking for a config row for the CTO OpenInsight will firstly look for CFG_CTO*<appid>*<username>, then  CFG_CTO**<username>, then CFG_CTO*<appid> before finally settling on CFG_CTO.

Column 1 - Server Details

URL

If a URL is entered here it will be assumed to be the path to the engine server responsible for dealing with engine requests

Path

If a path is entered here it will be assumed to be the path to the REVCAPI32.DLL to use for dealing with engine requests. Normally this is set to a period (full stop) to indicate the currently running engine.

If an exclamation mark is entered here the IP address and port number  will be taken from the CFG_ESERVER row in SYSENV.

 

Column 2 - Additional Details

This is a multi-valued column having up to four values.

Value 1 Port Number

The port number on which to communicate with the engine (server). By default this will be 8088 so if you are not changing the default this does not have to be entered.In OI 10 this value will default to 18088 to allow earlier versions and 10 to coexist on the same server.

 

Value 2 Engine Name

If you want to communicate with a specifically named engine server then enter the name here.

 

Value 3 Startup Flags

The flags to apply to the engine on startup. These can be as follows :-

   0 = Open Existing engine.
   1 = Create a New Engine
   2 = Open existing if available, otherwise create a new engine
   65 = Create a New Engine with no UI


The default value is 65

 

Value 4 Shutdown Flags

The flags to apply to the engine on shutdown. These can be as follows :-

   0 = Leave engine open
   1 = Close engine

The default value is 1

Column 3 - Foreground Color - CTO Only

This contains a text string, being chosen from a list of hard coded colours, namely

BLACK
DARK_BLUE
DARK_GREEN
TURQUOISE
DARK_RED
PURPLE
OLIVE
LIGHT_GREY
DARK_GREY
BLUE
GREEN
CYAN
RED
MAGENTA
YELLOW
WHITE

 

Column 4 - Background Color - CTO Only

This contains a text string, being chosen from a list of hard coded colours, namely

BLACK
DARK_BLUE
DARK_GREEN
TURQUOISE
DARK_RED
PURPLE
OLIVE
LIGHT_GREY
DARK_GREY
BLUE
GREEN
CYAN
RED
MAGENTA
YELLOW
WHITE


Column 5 - Window Title

The caption to appear in the AREV32 window.


 

Column 6 - Program

The name of a program to run within the AREV32/CTO environment. It will be run and then AREV32/ CTO will be terminated so this is NOT a startup program in the traditional sense. It does not run and then continue to AREV32/CTO. This permits the running of a single AREV32 routine without having to have the user log off.

 

Column 7 - Font Name

If you enter the name of a fixed pitch ANSI Windows font here it will be used in place of the default AREV32.



 

Column 8 - Macro Name

This does not refer to AREV32 Macros but rather the VBScript-alike Macros supported by the underlying AccuTerm OCX. This is specified as the name of a record in SYSENV, or as <table><space><recordname> if stored anywhere else.

 

Column 9 - History Lines

This column determines how many lines of "history" will be kept. You will likely have noticed that normally the AREV32 console has a vertical scroll bar. This is because by default a history is being kept of previous screens. This is a new feature and is not present in the shipping version with 9.4.

 

Column 10 - Character Mapping Function

The AREV32 console actually displays UTF8 characters not ANSI. So when working with alternate character sets it can sometimes be desirable to display different UTF8 characters for specific ANSI values. Column 10 permits the specification of the name of a function to be called that actually returns the mapping to be used.

When a mapping has been specified, any time the control receives one of the mapped ANSI characters (either via the keyboard, or sent from the host for display), the associated Unicode character will be displayed on the screen. The function should take no parameters, and should return a string that has the following format:

<char_code>:<Unicode_value>,<char_code>:<Unicode_value>{,…}/<char_code>:<Unicode_value>,<char_code>:<Unicode_value>{,…}

Where <char_code> represents the “input” ANSI character (as a decimal value), and <Unicode_value> represents the desired Unicode “output” (specified as a hex string). The <char_code> and <Unicode_value> are separated by a colon; multiple mappings are separated by commas. The first “/” delimited section of the string specifies the mappings (if any) that should be applied to the control when it is in “text” (CTO) mode; the second “/” delimited section of the string specifies the mappings (if any) that should be applied to the control when it is in “graphics” (AREV32) mode.

For example, the following string shows a very simple set of mappings:

65:010C/65:0171

This says that, when an ANSI 65 (“A”) is received by the control, it should display either Č (when running in CTO mode) or ű when running in AREV32 mode.

Note that the data that is actually RECEIVED or SAVED into the database will still be the associated ANSI(single byte) character under all circumstances.

This is a new feature and is not present in the shipping version with 9.4.
Pixel
Pixel Footer R1 C1 Pixel
Pixel
Pixel
Pixel