Logically Same DOUBLE Precision Assignments, Different Results (42853)

SUMMARY

Assigning SINGLE or DOUBLE precision variables to logically equivalent expressions that use functions and temporary variables differently may return values that vary slightly at the limits of their precision. These variations can range above or below the expected integral value, and can thus affect results returned from the INT function. This is not a software problem, but is due to the way that the compiler optimizes different expressions and rounds off values at the limits of precision differently in different expressions. The binary math used by the compiler cannot precisely represent all floating-point values at each intermediate step in a calculation, and round-off errors are unavoidable. For more information, query on the word IEEETUTR.

This information applies to Microsoft QuickBasic Versions 4.00, 4.00b, and 4.50, to Microsoft Basic Compiler Versions 6.00 and 6.00b, and to Microsoft Basic PDS Version 7.00. This occurs both in the QuickBasic environment (or the QuickBasic Extended environment of Basic PDS 7.00) and in executable programs compiled with BC.EXE.

MORE INFORMATION

The following program demonstrates the behavior. The variable "A#" is assigned to three different expressions that are logically equivalent but yield different floating-point results that vary at the limits of double precision. These slight variations affect the results returned by the INT function if the floating-point value is slightly less than the "expected" integral value.

CLS
Log10# = LOG(10#)
FOR I% = 1 TO 3
   X# = 1#
   SELECT CASE I%
     CASE 1
       PRINT "A# = LOG(X#) / LOG(10#)"
     CASE 2
       PRINT "A# = LOG(X#) / Log10#"
     CASE 3
       PRINT "A# = logX# / Log10#"
     CASE ELSE
   END SELECT
   DO
     X# = X# * 10#
     logX# = LOG(X#)
     ' The following three assignment statements are all logically
     ' identical but all produce slightly different results for
     ' different values of X#.
     ' Values of X# used: 10, 100, 1000, 10000
     SELECT CASE I%
       CASE 1
         A# = LOG(X#) / LOG(10#)
       CASE 2
         A# = LOG(X#) / Log10#
       CASE 3
         A# = logX# / Log10#
       CASE ELSE
     END SELECT
     IntegPart# = INT(A#)
     FractPart# = A# - INT(A#)
     PRINT "A# = "; A#; TAB(27); "INT(A#) = "; IntegPart#;
     PRINT TAB(43); "A# - INT(A#) = "; FractPart#
   LOOP UNTIL X# > 1000#
   PRINT
NEXT I%
END

The program output is as follows:

A# = LOG(X#) / LOG(10#)
A# =  1                   INT(A#) =  1    A# - INT(A#) =  0
A# =  2                   INT(A#) =  2    A# - INT(A#) =  0
A# =  3                   INT(A#) =  3    A# - INT(A#) =  0
A# =  4                   INT(A#) =  4    A# - INT(A#) =  0

A# = LOG(X#) / Log10#
A# =  .9999999999999999   INT(A#) =  0
                          A# - INT(A#) =  .9999999999999999
A# =  2                   INT(A#) =  1
                          A# - INT(A#) =  .9999999999999998
A# =  3                   INT(A#) =  2
                          A# - INT(A#) =  .9999999999999996
A# =  4                   INT(A#) =  3
                          A# - INT(A#) =  .9999999999999996

A# = logX# / Log10#
A# =  1                   INT(A#) =  1    A# - INT(A#) =  0
A# =  2                   INT(A#) =  2    A# - INT(A#) =  0
A# =  3                   INT(A#) =  2    A# - INT(A#) =
                                                   .9999999999999996
A# =  4                   INT(A#) =  4    A# - INT(A#) =  0