View Single Post
  #1  
Old 06-21-2005, 11:28 AM
BruceZ BruceZ is offline
Senior Member
 
Join Date: Sep 2002
Posts: 1,636
Default Got my calculator to lie

I was under the impression that calculators are supposed to store enough hidden digits so that the digits on the display are accurate. My HP15C evaluates this:

1 / [1 - (1000000/1000001)^(1/37)]

as 37,037,037.04 when it should be 37,000,019. Excel and the windows calculator get it right.

What policy will guarantee that all digits are significant?
Reply With Quote