logo
Powered by QM on a Rpi server
Home
About OpenQM
Sales and Downloads
Help and Support
About
Login

KnowledgeBase 00100: Partial Select Lists

Last updated: 17 Feb 2017
Applies to: All versions
Search  
Top level index       Full Index Search Tips
Previous article     Next article

This article was originally published as a Tip of the Week.

What is a Partial Select?

Consider an application that needs to process every record in a very large file. This application might do something like

   SELECT FVAR 
   LOOP 
      READNEXT ID ELSE EXIT 
      READ REC FROM FVAR, ID THEN 
         ...processing... 
      END 
   REPEAT 
It appears that the SELECT statement builds a list of all records in the file and the READNEXT then takes record ids one by one from this list to be used in the READ operation. This would require two passes through the file and, because we are discussing a very large file, it is likely that the data must be physically read from disk twice because the data cached by the READNEXT will have been discarded from memory as later groups are processed.

This is not what actually happens. Instead, the SELECT builds a partial select list that contains only the record ids in the first used group. The READNEXT/READ processes these records and when a READNEXT finds the list to be exhausted, a list of all records in the next used group is built. This continues until the final group has been processed.

Because constructing each partial select list is interleaved with the READ operations, there is a very high probability that the read can take the record from cache memory, removing the need for a second physical disk transfer. The application could potentially run twice as fast as would occur if the entire file was selected before execution of the processing loop. If the processing includes display of data derived from the records, there is also the possibility of improved user perception as the first record will appear on the screen almost immediately rather than after a complete scan of the file.


Disadvantages of Partial Selects

The mechanism described above has some important side effects that developers need to understand. The select operation needs to maintain a pointer to its current position in the file. The QMBasic language provides no concept of scope for this internal pointer such that, if the program exits from the loop before the final group has been selected, there is no way in which the partial select mechanism can be terminated automatically.

  • The scanning process requires that the file remains open until the select has been completed. Closing the file in the application is not sufficient. The partial select list must also be terminated, typically by use of CLEARSELECT. Any program that may only partially process a select list should always use CLEARSELECT to ensure that it is not holding the file open.
  • For this pointer to remain valid from one partial select to the next, split and merge operations within a dynamic hashed file must be suspended while a select is in progress on that file. Once all select operations from all processes are terminated, the file reconfiguration will catch up.
  • Writing a new record to the file in the processing loop may result in that record being returned by a READNEXT on a subsequent group. It is usually easy to code around this.

Completion

Some operations require that the scanning process completes building the list of all remaining record ids in the file. These situations include:

  • READLIST to transfer the list to a dynamic array.
  • SELECTINFO() to count items in the list.
  • Using a select list as a string.


QMClient

Use of QMSelect() and QMReadNext() in a QMClient application uses the same mechanism as described above. The API also includes the QMSelectPartial() function to return a list of up to 100 record ids and QMNextPartial() to continue this process. This can imporve performance by replacing many QMReadNext() calls with just one operation whilst still gaining much of the benefit of the underlying partial select mechanism.


The Virtual File System

The VFS also supports partial select lists though a VFS handler can always choose to return the entire select list if the data store to which it connects has no equivalent capability. The skeleton VFS handlers provided with QM show how to implement this feature.


Related Articles

None.



Please tell us if this article was helpful
Very     Slightly     Not at all
Comments
Email (optional)