Home page Home page Home page Home page
Pixel
Pixel Header R1 C1 Pixel
Pixel Header R2 C1 Pixel
Pixel Header R3 C1 Pixel
Pixel
By Sprezz | Monday, 8 October 2018 14:18 | 2 Comments
We've increasingly been working with people who have a requirement to encode their data in OI using UTF-8 and with this increase we've seen many concomitant problems. This article sets out to correct some common misconceptions and to attempt to provide guidance for the successful implementation of a UTF-8 OI system.

To ensure that we're all singing from the same hymn sheet let's firstly consider what UTF-8 actually is. Historically we've been used to having up to 256 characters available to us with the first 127 being defined by the ASCII standard and the subsequent 127 being sort of defined by the ANSI standard. When recording unaccented data this meant that pretty much all of our characters fell into the ASCII range. However when we start to require accented characters we run into a problem. We only have 127 characters to play with and there are THOUSANDS of accented characters in use around the world. In addition there are alphabets using characters which we don't even recognise as letters - Cyrillic, Kanji, Arabic etc.

For this reason the concept of "code pages" was implemented. This effectively told the computer what glyph to display when a character over ASCII 127 is encountered. So if the code page was set to 437 (the original PC code page) an ASCII 142 would be displayed as Ä but if the code page was set to 1250 say (Western European) the same ASCII value would be displayed as the character Ž.

Code pages were normally adequate when you were dealing with a single language, but as applications became more international clashes started to occur. For example using the code page 437 you wouldn't be able to store a record which contained both Ž and Ä because the former character doesn't exist in that code page.

UTF-8 provided a solution to this issue as it allowed the encoding of pretty much any character by allowing a character to be represented by more than one byte - a multi-byte string. Most letters are encoded in 2 bytes, but extended diacritical marks and other complex characters can take 3 or even 4 bytes to encode. As ASCII existed as a standard, UTF-8 will see any character <= 127 as an ASCII character requiring only 1 byte. However the moment you go above 127 you are telling UTF-8 "this is a multi-byte character string!". How does it know how many bytes are in the string? It doesn't just guess or look for some kind of termination character, rather it looks at the significant bits of the first byte.

Taking you back to binary 101, an eight bit byte starts with the most significant bits and continues to the least significant bit. so the first bit represents 128, the next 64, the next 32 and so on down to the last bit representing 1. If a byte is 00000000 then this represents 0, it it is 11111111 then this represents 256 (128 + 64 + 32 + 16 + 8 + 4 + 2 + 1). So if the most significant bit is a 0, UTF-8 knows that this is a single byte character. Logically that leaves only 7 bits to represent the character, so we can have up to 127 values (128 including 0).

If the first three bits are 110 then UTF-8 knows that this is a 2 byte character and expects the 2nd byte to begin with the bits 10. If the first four bits are 1110 then UTF-8 knows that this is a 3 byte character and it expects the 2nd and 3rd bytes to begin with the bits 10. If the first five bits are 11110 then UTF-8 knows that this is a 4 byte character and expects the 2nd, 3rd and 4th bytes to begin with the bits 10.

If UTF8 hits an initial character that indicates this is a multi-byte string (ie anything > char 127) and the following character or characters DON'T match the rules above relating to the first two bits then UTF-8 won't know what to do and will just process the required number of characters and display a "?".

Now this has MAJOR repercussions when moving from ANSI to UTF8. Let's firstly consider those multi-byte identifiers.

In ANSI which characters start with 110? Well 5 bits can represent 0-31 so with a byte beginning 110 we can have 32 characters - being ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß. Which characters start with 1110? Well 4 bits can represent 0-15, so 16 characters being àáâãäåæçèéêëìíîï. Finally, which characters start with 11110? Well 3 bits can represent 0-7, so 8 characters.ðñòóôõö÷.

Now following on from this, you'll remember that in multi-byte characters the trailing bytes must begin with the bits 10 to be considered valid. Which characters start 10? Well there are 6 bits so 64 possible characters and those characters are€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿.

(From hereon in I am going to concentrate my explanations around a single sample character - the lowercase e-acute (é). This is purely to simplify the explanation. I could have chosen any valid UTF-8 character.)

Putting all of this together, let's look at a possible scenario. Imagine that in your ANSI OpenInsight data you stored the name "Beyoncé Giselle Knowles-Carter". Then one day you clicked the magic UTF-8 check box without first running an ANSI_UTF8 conversion.. The UTF-8 processor might get to the é (11101001) and say "Oooh look, it starts 1110 - this must be a 3 byte character - but hang on the next two characters are a space and a G - they don't start 10, so that isn't a valid multi-byte character. I'll just insert a question mark". And suddenly QueenB might become "Beyonc?iselle Knowles-Carter"

The chances of the following characters starting 10 are slim, so all of your accented characters might just get swapped out for question marks with some of the following bytes being eaten.

That's the theory, so let's see what actually happens...

Firstly with OI in ANSI mode, we'll create a row containing the string "Beyoncé Giselle Knowles-Carter" and we'll write a little program which does nothing but read the row so that we can look at it in the debugger. The debugger is great for looking at what is actually in the variable at a byte level rather than an a character level. The hex representation shows the bytes NOT the characters.

So firstly, in ANSI mode, the entered data


We can either enter the é by using Ctrl-AltGr-e or by using the numeric keypad and entering Alt-0233. Now we can run our little test program which simply Xlates the row and allows us to inspect the variable in the debugger :-


As you can see, the é character in hex is E9 (Char(233)).

Now time for a brief diversion. It's something that we might not think about on a day-to-day basis , because it's not relevant to us, but Windows doesn't actually use ANSI, it uses UTF-16. But we're SEEING the appropriate character. You'll be relieved to hear that it's not sorcery, the Presentation Server simply says "Windows wants UTF-16 so I'll call ANSI_UNICODE and convert the E9 into the Unicode equivalent for display purposes". Then when we want to save the row, the Presentation server reverses the process by calling UNICODE_ANSI to transform the multi-byte characters into their ANSI form.

Now let's put our OpenInsight into UTF8 mode and repeat the process, again using Ctrl-AltGr of Alt-0233.


Running our program again we see that this time, our é is represented by C3A9.


Whilst we're in UTF-8 mode, let's just go back and look at our row 1, because if the theory above is correct, we should see corrupted data.


Theory failure! The character is still displaying correctly! Let's double check what's actually there using our program again.



It's still an E9. But what happens if we save the row then run again?


The data has been saved with the UTF-8 replacement for E9.

This behaviour is not expected. Speaking to Revelation Software about this it seems that the logic flow that converts the UTF8 to Unicode is being a little more forgiving than it ought and allows the E9 to be processed as an é and converts it to the Unicode equivalent. On writing it converts Unicode to UTF8 and the correct UTF8 is written back.

This may seem like desirable behaviour but stop for a moment and think. When you are working at the engine level (for example, running a program), the conversion doesn't happen. So if you have a database containing a number of E9s and you switch to UTF-8 mode, any rows that haven't been displayed and saved will remain as E9s whilst any new rows, or re-saved rows will be C3A9s. To illustrate what might happen, I populated a test table with 52 records of forms of Beyoncé, 25 using UTF8 é, 25 using ANSI é and two using UTF8 ò. I added a Btree index and sorted the report by name.


You'll see quite obviously that these aren't the results you'd hope to see. We'd have the same issue when using BTREE.EXTRACT. Visually it seems we have 50 Beyoncés, but in reality...


The lookup predictably doesn't retrieve the ANSI values.

The question is though, how can we clean up our data if we have mixed ANSI and UTF8? The answer is, there's no magic button. Let's take our original string "Beyoncé Giselle Knowles-Carter" in both ANSI and UTF8 forms and go back to ANSI with OI. For illustration purposes, we'll try and clean up first the ANSI, then the UTF8. You can see in the image below that the ansi_utf8 routine has successfully converted the ANSI (convertedAnsi) to be equivalent to the UTF8. But you can further see that it has trashed the UTF8(convertedUtf8).

Putting our OI back in to UTF8 mode we can look at the data in the editor and see that line 4 is indeed garbled.


If you're in this unfortunate position then you're going to have to be pretty ingenious to clean up. In a perfect world you'd just go back to a pre-UTF8 copy of the data and convert first!

The obvious moral of the story is that the check-box ISN'T magical and if you're going to check it AND you have existing data above char(127) you MUST run an ANSI_UTF8 conversion first.

As a final coda to all of the above... it is worth noting that UTF-8 supports many locales and some character glyphs are repeated across the locales but at different code points. In other words a certain symbol might have a value of 200, 4000 and 6000. When you come to sort these characters will sort in different places. Normally this wouldn't be an issue BUT if your users are cutting and pasting data from different locales it can be a problem. Like with most things software it's fixable - but that's beyond the scope of this blog post.

By Sprezz | Monday, 11 June 2018 12:41 | 0 Comments
We seem to have had a flurry of UTF-8 queries lately at Sprezz Towers and, as there isn't a lot of information published about this, we thought it might be useful to comment a little about this subject.

For many years, OpenInsight (OI) didn't support UTF-8 and this led to difficulties with people who needed to store foreign language characters; which also just happened to be system delimiters for OI. So, for example:

Delimiter ANSI Character
@Rm 255 ÿ
@Fm 254 þ
@Vm 253 ý
@Svm 252 ü
@Tm 251 û
@Stm 250 ú

Quite the challenge if you're working with European languages.

To get around this issue, Rev introduced something called CHARMAP - a special system wide property that essentially relied upon the developer choosing characters they didn't want to use lower down in the ANSI table and using those to store the affected characters. Then at display time, swapping the lower characters back to the higher.

For example, if you wanted to be able to store ý (and what red-blooded occupant of Český Heršlág wouldn't?), you might decide that you're never going to need to use ™ (ANSI 153) and so instruct the CHARMAP to map 253 to 153. You'd enter Český Heršlág and OI would store Česk™ Heršlág - then when you asked it to display the town name, OI would convert the ™ back to ý before displaying it.

Clunky, but effective.

With UTF-8 we no longer have this problem, ý is actually stored as the multi byte string 00FD and so it doesn't stand a chance of being confused with ANSI 253 (although ironically it is just ANSI 253 with an ANSI 0 stuck in front). BUT, UTF-8 doesn't read minds. It doesn't know that when we ask it to display an ANSI 153 we really mean 00FD. And OI doesn't know that either - you might have intended to store a trademark symbol. So, if you are planning on moving data that previously used CHARMAP into a UTF-8 environment, you need to do some groundwork to avoid tying yourself (and anybody else trying to help you) into a series of complicated knots.

  • Use the SETNOOFDELIMITERS routine to set the value to be used to 0
  • For every data row in the application
    • Read the row
      • Use Loop/Remove to grab the individually delimited pieces of data
      • Convert your CHARMAPPED characters back to their original value
      • Recast the string as UTF-8 using the ANSI_UTF8 function
      • Add back into the new string you are building
    • When complete, write this string back over the original row. This means it will look strange in ANSI I because it will contain multi-byte characters, but it will work properly in the UTF-8 OI app.
  • Fire up your UTF-8 enabled OI app
  • Remove anything to do with CHARMAP
  • Attach your data and all will be well with the world

Hopefully this information will help you in the smooth transition of your data from ANSI to UTF-8.

And if you're new to the ways of UTF-8 don't forget this handy article for programming in UTF-8.
By Sprezz | Wednesday, 9 May 2018 16:39 | 0 Comments
Recently Don Bakke at SRP published a very helpful blog entry highlighting just how much more productive it is possible to be in OI X – and he is SO right. Over the years, as more and more properties were added to the Presentation Server, the Form Designer struggled to keep up. So internally the Presentation Server had its list of properties and the Form Designer – being a separate executable – had its own list of properties which wasn’t really ever added to. The knock-on effect from this was that if you wanted to achieve certain things using the new properties you HAD to cut code, and cutting code takes time, it adds to the maintenance burden and it introduces a potential point of failure.

This post is going to highlight the new properties in just the Windows control that remove some of the coding burden from the OpenInsight developer. To make it easy to follow we’ll deal with the properties in the order in which they’re found in the property panel in the Form Designer. When properties are trivial we’ll describe them en masse to save wasting blog space!

DESIGN SECTION

COMMUTERMODULE

Documented in full in Carl's RevDevX blog.  Conventionally OpenInsight developers have adapted a naming convention of calling their commuter program by a standardised name (such as the Window name or the Window name with a suffix) so that they can easily identify the program for maintenance reasons. By having a COMMUTERMODULE property the developer can use whatever name they want. For example they could use the same commuter program for a logically related set of Windows. By having just one tried and tested code block used across multiple entities, this is yet another area where OpenInsight helps developers to have fewer lines of code to maintain and the minimum possible potential failure points

CONTEXTMENU
Historically attaching context menus to anything has been a… non-straightforward procedure. Now that context menus are the same as normal menus it is possible to design them using the Menu Designer and at design time they simply added to the CONTEXTMENU property.





STATUSLINE
Does what is has always done but it is now exposed in the Form Designer.

STYLESHEET
Does what is has always done but it is now exposed in the Form Designer.

DATA SECTION

MISC
A new property for all controls that just provides a slot for the developer to store stuff at design time and which can easily be accessed and modified at runtime.

TOOLTIP
A new property for all controls that support tooltips. The tooltip designer built into the Form Designer makes it very easy to add tooltips to controls.


Which are then displayed automatically when the user hovers over the control they’re associated with (in this case the Window).




DATA BINDING OPTIONS

Whilst IOOPTIONS still works, its component parts have been exposed as individual properties self-evidently named. Again, the flexibility to set these at design time removes the need for yet more code!

A couple of new properties have been exposed in here as well.

LOADPREVALWAYS
If this is set to “Yes” then PREV will be loaded on READ as well as WRITE. In other words, default OpenInsight behaviour is to only load the PREV property when a row is written – which only happens if a change has been made. If this is set then PREV will always be loaded (as it is loaded when the previous row is read).

WRITEATRECORD
Documented in full in Carl's RevDevX Blog. This property basically, instructs OpenInsight to write out ALL of ATRECORD not just the controls on the screen. If you didn’t know it did this then we’d recommend reading the article 😊.

LAYOUT SECTION

When wishing to size a Window to a specific size it could be difficult using the mouse. Now the LEFT, TOP, WIDTH and HEIGHT properties make this a cinch.

INITIALPOSITION
This removes the need for simple centring code, providing you with the nominatively determinative values of “AsDesigned”, “CenterDesktop” or “CenterParent".

TRACKINGSIZE
This is something which we routinely had to code for – to quote the MSDN documentation “The maximum tracking size is the largest window size that can be produced by using the borders to size the window. The minimum tracking size is the smallest window size that can be produced by using the borders to size the window.”. We used this normally to prevent the user from making any changes to the size of the Window – if we’d wanted a different size we’d have designed it that way 😉. By default, this isn’t set, so the user is free to resize the Window as they want. However, we no longer need to use TRACKINGSIZE to prevent this – see later!

VIRTUALSIZE
Intended to be set at runtime by the system when the screen becomes smaller than its intended size.

APPEARANCE SECTION

TRANSLUCENCY
Documented in full in Carl's RevDevX Blog. This simply sets the percentage translucency of the current Window to anything from 0% (completely opaque) to 100% (where’d my Window go?).

SHOWEFFECT
Upon opening a Window, you can now add a bit of theatre to your application. You can now select from the self-explanatory effects of Default, None, Fade, SlideDown, SlideUp, SlideRight, SlideLeft, SlideDownRight, SlideDownLeft, SlideUpRight, SlideUpLeft. As a PowerPoint user I sort of miss chequered et al but…

Note that this won't work on MDI Children or maximised Windows.

HIDEEFFECT
Upon closing a Window you do the same as with SHOWEFFECT. This is actually very handy for emulating popup menus.

IMAGES SECTION

Effectively defines the image used by the Window in great detail. But don’t worry most of the time you can just accept the defaults. It’s only if you really want your application to continue looking professional under all DPI conditions that you really have to pay attention to this. It is documented in great detail at   https://revdevx.com/2013/02/07/the-image-api  https://revdevx.com/2014/06/09/the-scaled-event  et al

BEHAVIOUR SECTION

ACCEPTDROPFILES
A Boolean property indicating whether the Window can have files dropped onto it during a Drag and Drop operation. No more raw style bits to understand and contend with!

CURSOR
Permits the selection of a different cursor for use with this Window when over an editable control. The cursor will be in a file with extension .CUR.

SCROLLMODE
Permits developers to set whether the system pages multi-page windows, or whether it pans them..

SIZINGMODE
Determines whether the user is allowed to resize the Window. If set to “Never” they aren’t allowed. If set to "Always” then they're always allowed. If set to “Default” then the user is allowed if Windows deems it to be appropriate.

TABWITHINPAGE
In multi-page forms, if set to “No”, tabbing off the last control on a page will move to the next page. If set to “Yes” then tabbing off the last control on a page will move to the first prompt on that page.

WINDOW STYLE SECTION

FRAMESTYLE
This exposes the frame types we were used to in previous versions of OpenInsight. We can have fixed, sizeable or dialog. This doesn’t actually affect the behaviour of the Window, rather it provides the user with a visual clue about the behaviour of the Window. In addition, three new frame styles have been added.

 “None”   No frame at all – good for a panel appearance
 “FixedTool”  Removes the ability to have help/minimise/maximise
 “SizeableTool”  Removes the ability to have help/minimise/maximise

HELPBUTTON
Should a “Help button” be displayed next to the system “Close” button?



Note that this is mutually exclusive with the MAXIMIZEBUTTON and MINIMIZEBUTTON properties.

MAXIMIZEBUTTON/MINIMIZEBUTTON
Whether the Caption should have a Maximize/Minimize button. Note that these are mutually exclusive with the HELPBUTTON property.



SHOWCAPTION/SYSTEMMENU
Exposes these properties both in the User Interface and programmatically.

ADVANCED SECTION

AUTOCOMPOSITED
CLIPCHILDREN
CLIPSIBLINGS
COMPOSITED


If you know what these are you'll know when to use them. If you don't, it's highly unlikely you'll ever need them!

By Sprezz | Wednesday, 2 May 2018 12:14 | 0 Comments
Back in OI 8 the concept of Associated Multi Value (AMV) groups - present in the original Rev C-G - was reintroduced. Basically when designing systems, we, as the developer already know which columns make up an AMV group, so the dictionary was extended to include this information - specifically in columns 36 and 37



One column is defined as being the controlling Multi Value (the "group master") and it (and all other Multi Values in the group) are defined as being part of that group by name. So for example in my CODES table the controlling item CODES looks like this :-


and in the table builder the defined group looks like this


But here is the real beauty of this feature. In X the form designer knows about the group so when painting a new form and selecting the controlling MV I get this prompt


Clicking "Yes" provides me with this entry form


OK, so it's not perfect - the Exclude and Sortkey widths haven't been accounted for correctly BUT it's a vast improvement on the old way of doing things.

There are just so many cute little tweaks in X that are making my life so much easier. I haven't had so much fun in ages!

By Sprezz | Tuesday, 17 April 2018 16:35 | 9 Comments
WW



So OpenInsight X has finally arrived resplendent in all its glory! After a gruelling beta cycle Revelation Software announced yesterday the formal release of OpenInsight 10.0.1.

We've been working with the software internally for quite some time and it has been a pleasure to see it flesh out and come to life. So many small but important details are taken care of. The main executable is now digitally signed - so no more scary warnings when launching the program. The long term goal of developing a tool that's good enough to develop a tool has finally been achieved - no more random executables! These are all of the exe files in the product - not a formdes.exe in sight. Finally form designer is written in OpenInsight itself. This is testimony both to the power and flexibility now delivered in the shipping product.



The first impression on logging in is that of an uncluttered clean interface


Gone is the busy VB3 style front end, replaced with a simple set of choices. Do you want to create something new or open something? If you're just beginning development you're going to want to create an application. As that's such an unusual thing to do it isn't included in the standard "New" dialog. Rather there's an item specially for it on the File/New menu.


The dialog is familiar but as we opted to use the new stronger security during installation our passwords have to conform to the default rules.


There's a lot to look at but in the first instance we're just going to take a flying visit to the form designer. First though let's add some existing tables into the mix. To do this we'll need to turn on the Database Panel.


and now we've done that we can choose to "Add files"


and add in our PROFILES table to give us something to play with


Apply the changes and then a  keystroke shortcut (Ctrl-Alt-D) to toggle the database panel off. All panels have keystroke shortcuts to toggle on and off.

So now let's launch the form designer


for a data bound form.


Just a couple of things to note here. By default the form designer DOESN'T add a menu to the form. This is consistent with good practice in an MDI application. Instead it offers to place OK and Cancel buttons on the form. As with 9.4 the form designer chooses the most suitable control type to place on the form but allows overriding if required.


All the usual suspects and a date/time picker too! Pressing OK generates a default form and opens the form designer.


The first thing that is apparent is that OI X makes intelligent choices for control width! The screen straight out of the box is visually appealing! The screen is now split up into four main areas (The menu and TCL are omnipresent). These areas are the Toolbar at the left, the main design screen in the middle, the properties and events at the right and the status line at the bottom. By default the screen features a grid and snap to grid is set. These options can easily be overridden from the Settings/IDE Settings menu option :-


Note also that right clicking on the screen image provides a context sensitive options menu from which you can override some of these preferences.


As you move from control to control the property panel changes to reflect the properties specific to that control type - so an edit line has a whole different set of properties to a check box.



Similarly an edit line has a whole different set of events to a check box.



But what is really impressive is what happens when you select more than one control of different types :-


The property panel now only shows COMMON properties between the two controls.

As with 9.4 it is possible to add quick events and/or scripts to a control but there has been some significant changes in ease of use for quick events. Let's illustrate this by adding a popup to the options for the user id prompt.

So firstly we add a quickevent to the OPTIONS event on the USERID edit line.



Note a couple of new additions.

Does the quick event have priority? In existing OI, quick events occur at the end of the event chain. So for example, if we are putting on a READ quick event, the system will look for a script, execute it if present, if required execute the actual READ then execute the quick event. This means that the READ quick event is always POST READ. By setting priority, the quick event will be executed before the script and actual READ event. This means that for the first time we can use commuter modules for everything and never actually have to cut a script. In 9.4 sometimes you had to used scripts to get in before the system logic. No more!
Note also the "Finalize" section. When the quick event has completed what should be done (if anything) to wrap up. In the case here where we've returned a row id into the key prompt we obviously would like to read the row. Now we no longer have to cut code to make this possible.

Moving on to linking to this options event - we add a hyperlink control to replace the user Id static.


and we set the tooltip property


and we add a quick event to the CLICK event of the hyperlink control. 


Clicking the button opens the dialog from where we can check that we want a quick event or check that we want a script.



We want a CLICK quick event so we select that


and rather than "General" we want to do something else. Dropping down the combo box we see a number of events, the newest one being "Trigger Options" - i.e. cause the OPTIONS event to be triggered for the control we were just on.


Choosing this leads to what I can only describe as a touch of genius



The quick event sets the SYSTEM FOCUS property to the control we were just on (@PREV - the PREVIOUS property of the current control) then finalizes by sending an OPTIONS event to the control currently having focus.

Running the window we see that the User Id: static is now a hyperlink


and hovering over it with the mouse changes the appearance and displays a tooltip


actually clicking the hyperlink sets the system focus to the USERID edit line and sends an OPTIONS event. The popup displays


a value is chosen


 and the USERID populates and a READ event is issued


No code required at all for such a comprehensive set of actions.

Moving onto the Toolbox section of the screen.


All of our old favourites are there along with a number of new control types.
The attention to detail is just phenomenal. Take as a simple example what is described as the "Up-Down Control". Let's add one into our screen and put it alongside an edit line.


Go into the properties of the up-down control and set a range to operate on and a "BUDDY" control - the edit line to associate with. Set the AUTOUPDATEBUDDY property to True


and now when you're in the entry form, scrolling the arrows will update the edit line and changing the edit line will update the scroll position! Again - no coding.

This short overview hasn't even begun to scratch the surface of what's changed in OI X but hopefully it will have whetted your appetite to get in there and play!
Pixel
Pixel Footer R1 C1 Pixel
Pixel
Pixel
Pixel