Powered by QM on a Linux server
Help and Support

KnowledgeBase 00028: Precision, FLTDIFF and INTPREC

Last updated: 27 Dec 2018
Applies to: All versions
Top level index       Full Index Search Tips
Previous article     Next article

This article discusses features of QM that affect floating point (non-integer) arithmetic. The original Pick systems were "integer only" environments that required scaling to handle fractional values. This would be totally unacceptable to most programmers today but leads to the issues described below.


The QMBasic PRECISION statement sets the limit on the number of decimal places when converting a floating point value to a character string. For example, a simple test program:

   n = 12.3456789 
   for i = 0 to 9 
      precision i 
      display i, n 
   next i 
   0    12 
   1    12.3 
   2    12.35 
   3    12.346 
   4    12.3457 
   5    12.34568 
   6    12.345679 
   7    12.3456789 
   8    12.3456789 
   9    12.3456789 

Note that the last two lines of output do not have trailing zeros as the precision value is a limit on the number of decimal places, not a demand for that number of decimal places. The MD conversion code can be used where a fixed number of decimal places is required.

All arithmetic operations are carried out internally to the maximum precision offered by the underlying IEEE floating point representation. The PRECISION value is only applied on conversion to a character string, typically for display or storage in a dynamic array.

On entry to a program, the default precision value is 4. As an alternative to using PRECISION to set a specific value, there is a variant

that will inherit the precision of the program that called the subroutine in which this statement appears.


Just as numbers such as one third cannot be written accurately in decimal form, there are numbers that cannot be represented accurately in the IEEE floating point form used by the computer system. Even an apparently simple value such as 14.2 cannot be stored accurately and becomes something close to 14.19999999999999929. This leads to a problem when comparing floating point values where apparently equal items may found to be unequal due to the slight inaccuracy of data representation.

The FLTDIFF configuration parameter determines how close two floating point values have to be for them to be considered as equal. The default value is 2.91 * 10-11 (which is 2-35).

This tolerance is applied on equality tests between floating point values. This might be, for example...

  • Use of the EQ or NE operator.
  • The limit condition of a FOR statement with a floating point loop counter.
  • In OCONV() or FMT(), when determining whether the Z element (replace zero by null) should be applied.
  • When comparing items in some sort operations.

FLTDIFF is a private configuration parameter. The QM configuration data sets its initial value but it can be modified in an individual process by use of the CONFIG command.


There is a third tolerance value, set by the INTPREC private configuration parameter, which is slightly different from FLTDIFF. It controls the rounding applied when converting a floating point value to an integer.

As an example, a value entered as 17.9 and stored in floating point form is actually stored as something rather closer to 17.8999999999999986. The INT() function is defined to discard the fractional part of a value, effectively rounding towards zero. This is fine for

   x = 17.9 
   display int(x) 
which prints 17 but it is illogical (but technically correct) for
   display int(x * 100) 
which, without INTPREC, would print 1789.

The INTPREC parameter sets the number of decimal places after which rounding occurs in the INT() function. The default value is 11. This rounding ensures that the above example displays the expected result.

Related Articles


Please tell us if this article was helpful
Very     Slightly     Not at all
Email (optional)