Mo logo [home] [lexicon] [problems] [tests] [courses] [auxiliaries] [notes] [staff] german flag

Mathematics-Online lexicon:


A B C D E F G H I J K L M N O P Q R S T U V W X Y Z overview

Standard rounding approximates a real number $ x$ with the nearest even floating point number R$ x$. If $ \vert x\vert$ lies within the admissible floating point range, then

   R$\displaystyle x - x = \delta x,\quad
\vert\delta\vert \le$   eps$\displaystyle = \beta^{1-p}/2,

where $ \beta$ is the base and $ p$ the number of digits of the floating point representation. The constant eps is called the machine precision and equals

$\displaystyle 2^{1-53}/2=2^{-53}

for double precision.


[Annotations] [Links]

  automatically generated 3/ 8/2007