Home page Home page Home page Home page
Pixel
Pixel Header R1 C1 Pixel
Pixel Header R2 C1 Pixel
Pixel Header R3 C1 Pixel
Pixel
By Sprezz | Tuesday 30 June 2009 11:59 | 0 Comments
In May we blogged warning that OI 9.0 shipped with a blank SYSPROCNAMES.

Just this week a client contacted us with screen shots of some frankly bizarre Popup behaviour.

"You haven't by any chance recompiled POPUP_SUB have you?"

"Well yes..."

As longstanding OI developers will be aware, the source code for POPUP_SUB that is out there was provided during the "fire sale" of previous management who, having no 32 bit product to deliver tried instead to give away elements of the product source to assuage a malcontent user base. So it's very out of date. The blank SYSPROCNAMES allowed the client to recompile this old code and to trash their working popups.

If you're at 9.0 it might be an idea to follow the advice given in the original blog post and manually update your SYSPROCNAMES.

Labels: , ,

By Captain C | Friday 26 June 2009 10:23 | 0 Comments
In our last post about DLL Prototyping we looked at using Namespaces to avoid collisions with other programs in the system. Over the next few articles we're going to take a look at Windows API calls that take strings as arguments along with the considerations you need to take into account when you use them.

Unicode and ANSI functions

One thing that you'll find working with the Windows API is that nearly every function that accepts string arguments has two versions: one that takes ANSI strings (one byte per character) and another that takes Unicode strings (two bytes per character or UTF-16). By convention each of these functions is named slightly differently when exported from its parent DLL - The ANSI version is suffixed with an "A" and the Unicode version is suffixed with a "W" (for "wide char" which is a 2-byte character type).

For example, if you want to use the GetWindowText() function it's actually exported from User32.dll as two functions:

  • GetWindowTextW()

  • GetWindowTextA()

There is no exported function named GetWindowText!

The documentation is written to use the plain function name because a C/C++ compiler can automatically substitute the correct A or W version when it sees the plain version. When prototyping the function for use with OI this is something you have to take care of yourself!

To carry on with the GetWindowText example here's how Microsoft documents the function:


 
 int GetWindowText( HWND   hWnd,
                    LPTSTR lpString,
                    int    nMaxCount
                  );
   


However, if you were writing a C/C++ program here's what the compiler would actually use for a Unicode program:


 
 int GetWindowTextW( HWND   hWnd,
                     LPWSTR lpString,
                     int    nMaxCount
                   );
   


and here's what it would use for an ANSI program:


 
 int GetWindowTextA( HWND  hWnd,
                     LPSTR lpString,
                     int   nMaxCount
                   );
   


These last two are the definitions you would have to use when prototyping the function in OpenInsight - NOT the first definition as given in the documentation.

TEXT strings (LPTSTR and LPCTSTR)

As well as the difference in the function name notice how the type of the string argument "lpString" has changed as well. In the Unicode version the argument has been translated from LPTSTR to LPWSTR, while in the ANSI version it has been translated to LPSTR.

This type of string argument (LPTSTR) is called a "TEXT string" and functions that support Unicode and ANSI versions always want them as arguments. However, in reality there is no TEXT string type - what actually happens is that the C/C++ compiler resolves the TEXT string to either a Unicode string (LPWSTR) or an ANSI string (LPSTR) in the same way that it works out which version of the function ("W" or "A") to use at compile time.

So, when dealing with Windows API functions that support this dual string interface you must:

  1. Prototype the function with the "W" or "A" suffix
  2. Prototype the function arguments using the correct string type


This is all very well but how do you know when you have to watch out for the W/A suffix? Well, there are two easy ways to tell:

  1. Read the documentation! Microsoft documentation has been improving steadily over the years - these days they usually explicitly mention the A and W versions along with DLL they are exported from (usually at the bottom of the actual function documentation).

  2. Look at the arguments passed. If you see any arguments that look like this:

    • LPTSTR (pointer to a TEXT string)

    • LPCTSTR (pointer to a constant TEXT string)

    Then you can be pretty sure you're dealing with a function that has an ANSI and a Unicode version.

(Incidentally the "constant" version of the argument means that the function promises not to alter the contents of the string that you pass it - it actually makes no difference to the way you prototype in function in OI)

Unicode or ANSI?

Having a dual string interface harks back to the days of Windows 9x operating systems that were basically ANSI systems with a thin layer of Unicode bolted on the top. Modern NT-based systems (Win2K, XP, Vista/Win2008 and Windows 7 etc.) all use Unicode internally, and the "A" functions simply convert passed ANSI strings to Unicode and invoke the "W" function instead, thus incurring extra overhead (In fact you may notice that many of the newer API functions that were not present on Win9x systems do not have a dual interface, and are exclusively Unicode, so there's no suffix/TEXT string translation to worry about).

As OpenInsight no longer supports Win9x it makes sense to exclusively use the "W" versions where you can, thus avoiding the overhead of the internal "A" to "W" translation.

What's next?

That's the heavy theory lesson over now, so you should be aware of the "A" and "W" style Windows API functions and the fact that they take different types of strings. In Part 2 of this series, we'll be taking a look at actually passing strings to these functions and the different ways this can be done.

Labels: , , ,

By Sprezz | Sunday 21 June 2009 23:57 | 0 Comments
At one of our Canadian clients recently we were processing our way through a large file and without warning the AREV would crash and an NTDVM error would appear. This was a little annoying to say the least (this is what we British call understatement :)). As a rule of thumb, systems that don't crash the NTDVM are to be preferred.

We debugged this as much as we could in AREV but were getting nowhere. We could establish exactly where the crash occurred but it wasn't directly related to an i/o operation - as in it happened AFTER an i/o operation and not on the else clause

Eventually inspiration struck - we'd use OI to perform the same process and if it failed too we'd report a UD bug to Revelation, if it succeeded we'd report a terminal AREV bug to the client. With these two options in mind (A and B) we installed an evaluation copy of OI onto the server and reran the rest program using OI. Sure enough the program failed - but it failed cleanly with an FS error. An FS129 to be precise.

It was in this way that we discovered Option C. If at some point in the mists of time your client has gone directly to a network product that does not contain warnings about Group Calculation Error bugs and have thus never had the opportunity to fun FIXVOL.EXE or FIXGCE.EXE then they may still have tables out there containing this error. This would manifest itself by "losing" data and by occasionally abending the NTDVM. The solution is of course simple - run FIXVOL.EXE against the errant data.

To explain - when the 1.5 Network Service was introduced it came with an explanation that earlier versions of the Linear Hashing algorithm came with a bug in group calculation in tables having a modulo of greater than 64K. The documentation recommended using supplied programs to fix the problem tables.

We located a copy of said program on the Sprezz servers and were able to get back in business. It's at times like these that a long memory goes a long way to troubleshooting problems.

PS Revelation (as is comfortingly the case nowadays) when notified of this have moved to update the Service documentation AND provide download links for these utilities. Jolly good show!
By Captain C | Wednesday 17 June 2009 12:13 | 0 Comments
As you probably know DLL prototyping with OpenInsight is the process of defining functions exported from a Windows DLL so that they may be called from your Basic+ programs.

In a nutshell the process involves creating a "prototype record" in your SYSPROCS table containing a list of all the functions in the DLL you wish to use along with the type of arguments each function expects. After that you run a system process called Declare_Fcns() which uses the prototype record to create one or more "DLL stub programs" in your SYSOBJ table. These programs have a special format that, when executed, tells OpenEngine that it should load and call a function in a DLL rather than trying to execute it as a "normal" program.

(The entire process is described in detail in Chapter 7 of the Programmers Reference Manual on-line help)

When creating the prototype record there are several conventions that the product documentation follows that can lead to problems over time, and in this post we'll have a look at how to avoid them using the concept of "namespaces" - i.e. making sure that any naming conventions you use during the prototyping process are unique to your application so you avoid collisions with other objects in the system.


Naming the prototype record

The first issue is with the actual name of the DLL prototype record itself. The documentation recommends that you give it a name of

   "DLL_" : <dllname>

e.g. If you want to use a function from User32.DLL you'd prototype those functions in the DLL_USER32 record in the SYSPROCS table.

This is all well and good, and Revelation supply prototype records for several of the core Windows DLLs because many internal OI components use functions exported from them... and herein lies the problem:

If you make changes to the Revelation-supplied prototype records there is a good chance that they will be overwritten on a subsequent OpenInsight upgrade if Revelation decide to release an updated version.

To avoid this you should create a unique name for your prototype record - for example here at Sprezz HQ we tend to use "ZZ" or "ZZX" a lot, so if we're prototyping functions from User32.DLL we'd be updating a record called "DLL_ZZX_USER32".

The actual name of the prototype record has no bearing in the name of the DLL loaded at runtime - it's only used when you run Declare_Fcns() to create the stub programs (the actual runtime DLL name is contained on line 1 of the prototype record). You could have called your prototype record DLL_BOZOSLIVEHERE and you'd still be able to call the functions from User32.DLL


Naming the DLL functions

The second issue is the name of the actual prototyped functions themselves. Depending on the results you wish to achieve it is possible that a DLL function may be able to be prototyped in more than one way.

e.g. If you have a function that expects a pointer to a string you could prototype it as:

   INT STDCALL MyFunc( LPSTR )

or

   INT STDCALL MyFunc( LPVOID )

The first version expects OpenEngine to get the pointer to a Basic+ string variable and pass it to the function for you, while the second version expects you to provide your own pointer to a string variable via the GetPointer() function (There are valid reasons for being able have more than one way of defining the function argument types, but that's a story for another post...). Both definitions would create a DLL stub program in SYSOBJ called $MYFUNC so you could use MyFunc() in a Basic+ program.

You can probably see what's coming here. If MyFunc is defined in more than one prototype record and the definitions within are different then you have the potential for another naming collision. A case in point is the FindWindow() API function exported from User32.DLL - Many years ago I created my own prototype record to hold my own versions of some User32 functions. My definition of FindWindow was different from the one Revelation supplied. When I created my FindWindow the entire OI system stopped working as some internal OI functions were relying on the Revelation version!

The solution here is to alias the DLL function using the "as" keyword in your prototype so that your stub program has a unique name too.

e.g.

   INT STDCALL MyFunc( LPSTR ) as zzx_MyFunc

This creates a stub program in SYSOBJ as $ZZX_MYFUNC - You'd use zzx_MyFunc() in your Basic+ programs but it would still use the real MyFunc() function in the DLL at runtime.

(Revelation documentation on using the AS keyword can be found on their website here

Labels: , , ,

By Sprezz | Monday 15 June 2009 10:44 | 1 Comments
So I'm on client site in NYC working on an old AREV system we've web and OI Tablet application front ended for them and we've got the old Linear Hash Service 2.1 non-unique station Ids on the same workstation coming up.(If you're not familiar with this, running multiple instances of AREV on the same workstation returns non-unique station ids). This is causing us no end of problems and the obvious fix (upgrade to the Universal Driver) isn't possible just now. So I had to develop a routine to change the current Station Id to make it unique.

I developed the routine (using date time and a random number) and it seemed to be working well but here's where I discovered an interesting fact. If you PERFORM or EXECUTE an AREV command with field marks in, it will "stack" the commands and issue them one at a time. This isn't really that useful but it is a constant delight to discover new information about a subject you thought there was no more to learn about!

Labels: , , , ,

By Captain C | Wednesday 3 June 2009 12:00 | 0 Comments
With the advent of OI 9.1.0 the number of controls allowed per form in the Form Designer has been raised from 920 to 2048. This is all well and good but we're often asked why there's a limit at all?

Well, there are a few reasons, but currently most of them are to do with the internal design of the Form Designer executable itself (allocating static arrays and suchlike). However, as you've seen with OI 9.0.0 work is under way to move the Form Designer into OpenInsight proper, so the limitations imposed by the design of the standalone Form Designer will be moot.

Unfortunately, before you get too excited at the prospect of the new Form Designer having no limits at all (and we don't know if that's going to be the case anyway), there's still another factor to take into account: each process in Windows is limited to 10000 User objects (or window handles as they are more commonly known).

Consider the following:
  • Say you can create a form with 3000+ controls (not an impossible requirement according to the wishes of some clients) and you test-run it.
  • You're now looking at 6000+ (your design form and the test form) out of the 10000 handle limit taken up, and this doesn't take into account any other OI forms running like the Form Designer itself, the main OI IDE, the System Editor, your other application forms, and so on.
  • You can see it won't take long to hit that 10000 limit!

Even Microsoft themselves had to deal with this problem for Internet Explorer. It's theoretically possible to create far more than 10000 controls with an HTML form, so all the form controls in the browser are emulated - they are not actual User objects.

So, the bottom line is if you're designing forms with a very high number of controls just remember that Windows itself imposes limits on how many you can create - it might be wise to consider a UI redesign?

Labels: , ,

By Sprezz | Tuesday 2 June 2009 00:45 | 4 Comments
Following on from our post about extra long row ids causing trouble with pre 4.6 versions of the UDH and LHVerify it became apparent that we'd have to write a utility to scan every single table on the system for aberrant ids. So this is all very straightforward right? Open SYSTABLES, build a table list, loop through opening and select the table checking id length. A job for a junior programmer right?

Wrong!

Whenever you open a table, the handle for the table is put into field 5 of that table's row in SYSTABLES. Regretfully although we refer to a "row" in a "table", SYSTABLES is actually an in memory variable manipulated by the RTP50 BFS. So as it is an in memory variable in AREV this has a maximum length of 64K. So if this gets to the limit you're going to see an RTP50 crash! On BIG systems the strong likelihood is that you've never had all of your tables open before - and guess what? When you do.... !RTP50 Variable Exceeds Maximum Length'...

When I first started selling Revelation C one of the joys was explaining to programmers of other systems that "You don't need to CLOSE files after opening them, the system does all that for you". How I have come to rue those words... (and yes, we did say files then, not tables. Catch me on an off day and I'd probably do the same now).

What we needed to prevent this RTP50 problem was to be able to close the table when we'd finished processing it (thereby keeping the SYSTABLES table small and manageable) but the BFS doesn't expose such a call - which is slightly strange given that BFSs work with any kind of filing architecture.

Originally we developed our own routine to achieve close functionality but a comment to this blog post from M@ drew our attention to an undocumented OI routine called CLOSE_TABLE which was actually introduced in AREV 3.11 with a syntax of

CLOSE_TABLE(lpTableName, bCloseAssociatedTablesAsWell, bSuccessFlag))

Usage is straightforward. To close the table, simply pass the table name to this function and the handle will be removed from attribute 5 of the SYSTABLES row for the table. To close the table AND the dictionary and index table, add a second parameter of True$ to the call. To see if the call was successful examine the third parameter.

Now the only problem you'll face is working out which tables are safe to detach without shooting yourself in the foot.

Labels: , ,

By Sprezz | Monday 1 June 2009 17:29 | 0 Comments
Returning from yet another West Coast trip (albeit this one was for pleasure - I actually ran in Bay to Breakers for the first time) I was struck by how stupidly low priced international travel has become.

Here at Sprezz towers we're blessed with a huge choice of airlines with which to fly to the US although we tend to favour Virgin Atlantic or British Airways and these two seem incredibly well synchronised when it comes to pricing deals. The major problem with flying out of the UK is the stupidly high taxes levied on departure. Recently we've seen return transatlantic fares fall below £300 ($480) of which over £275 ($440) was tax meaning that the airline itself was making less than £25 ($40) on the flight - or £12.50 ($20) each way. This is economic madness and speaks strongly of the recessionary pressures on international travel.

Naturally there's a plus side to this or I wouldn't be blogging about it! It now costs less to fly in a Sprezzatura consultant to the US than it does to take most domestic flights - especially if they're a couple of thousand miles! So if you've got a pet project you'd like to have peer reviewed or an application that you know would benefit from the application of Sprezz's in depth knowledge of OpenInsight and AREV's internals then there's never been a better time to involve us.

Alternatively we already have plans for Sprezz people to be in the US in June, July and August so if you'd like us to schedule a trip to see you around one of these existing paid for flights then let us know.

Contact us via the contact page and let's talk!
By Sprezz | 16:00 | 0 Comments
We've had some good successes with the Universal Driver Heavy, including my own personal favourite - installing it on an AREV 2.12 site despite the warnings on the tin! It took a bit of tweaking but we got there.

Anyways on two separate sites recently we needed to replay the journal files to bring the secondary server back up to synchronicity with the primary server and on both of these sites the process would fail, resulting in us having to copy the secondary from the primary - thus rather removing the point of having the UDH in the first place.

We'd start the UDH and launch the manager to instruct the service to go into mirroring and it would happily start chugging along doing something - perfmon showed a lot of I/O and CPU activity associated with the LH31SRVC.EXE so we were confident that it was trying to do something, but after a short while the LH Manager would start to display REV_LOADREC errors and crash. Now we well knew that this meant that the LH Manager could not contact the service and sure enough the service was no longer in memory but it had exited so cleanly that there were no reports of errors in the UD Log OR the Event Log.

Given the complete lack of available diagnostic information we were initially stumped until the idea of running the service in debug mode was mooted. So opening a command prompt we changed to the UDH directory and executed LH31SRVC.EXE -debug.

The service started reading through the journal files and optmising them. The numbers crept up, 10, 20, 30, 40 and then at 44 the UDH just abended. Well, exited is probably more accurate as there were no errors to speak of. It would seem that something in the 44th file was causing problems for the UDH.

Revelation are understandably keen to ensure that the UDH is as stable as possible so the files were mailed to Revelation who tested them on the latest 4.6 build of the UDH (after writing a utility to make the journal files the correct format) and discovered the same error. At this point they lept into sleuth mode and within hours were able to declare that the UDH replay had some fairly fatal problems when row id length exceeded 512 bytes!

Now given that the maxiumum row id size has been documented as 50 characters we were surprised by this strangely excessive row id length, but the fix was confirmed and in future releases of the LH driver a maximum row id length of 512 bytes will be enforced - attempts to write with a longer id will cause the ELSE branch of the write to trigger.

Mind you these investigations did trigger an interesting discovery. Given that the maximum row id size was 50 bytes, the system verify routines would report a GFE if any such row ids were encountered. So if you've ever had a mysterious GFE which didn't seem to be there then check the size of your row ids! We had to develop a routine to do such a thing but that's a subject for another blog post!

Labels: ,

Pixel
Pixel Footer R1 C1 Pixel
Pixel
Pixel
Pixel