|Powered by QM on a Rpi server|
KnowledgeBase 00100: Partial Select Lists
This article was originally published as a Tip of the Week.
What is a Partial Select?
Consider an application that needs to process every record in a very large file. This application might do something like
SELECT FVAR LOOP READNEXT ID ELSE EXIT READ REC FROM FVAR, ID THEN ...processing... END REPEATIt appears that the SELECT statement builds a list of all records in the file and the READNEXT then takes record ids one by one from this list to be used in the READ operation. This would require two passes through the file and, because we are discussing a very large file, it is likely that the data must be physically read from disk twice because the data cached by the READNEXT will have been discarded from memory as later groups are processed.
This is not what actually happens. Instead, the SELECT builds a partial select list that contains only the record ids in the first used group. The READNEXT/READ processes these records and when a READNEXT finds the list to be exhausted, a list of all records in the next used group is built. This continues until the final group has been processed.
Because constructing each partial select list is interleaved with the READ operations, there is a very high probability that the read can take the record from cache memory, removing the need for a second physical disk transfer. The application could potentially run twice as fast as would occur if the entire file was selected before execution of the processing loop. If the processing includes display of data derived from the records, there is also the possibility of improved user perception as the first record will appear on the screen almost immediately rather than after a complete scan of the file.
Disadvantages of Partial Selects
The mechanism described above has some important side effects that developers need to understand. The select operation needs to maintain a pointer to its current position in the file. The QMBasic language provides no concept of scope for this internal pointer such that, if the program exits from the loop before the final group has been selected, there is no way in which the partial select mechanism can be terminated automatically.
Some operations require that the scanning process completes building the list of all remaining record ids in the file. These situations include:
Use of QMSelect() and QMReadNext() in a QMClient application uses the same mechanism as described above. The API also includes the QMSelectPartial() function to return a list of up to 100 record ids and QMNextPartial() to continue this process. This can imporve performance by replacing many QMReadNext() calls with just one operation whilst still gaining much of the benefit of the underlying partial select mechanism.
The Virtual File System
The VFS also supports partial select lists though a VFS handler can always choose to return the entire select list if the data store to which it connects has no equivalent capability. The skeleton VFS handlers provided with QM show how to implement this feature.