Tuesday, December 11, 2007

Double v/s Decimal

Just recently, one of my Colleagues approached me with a very strange problem. He had made a function for Rounding the numbers to the specified number of decimal places.

He was facing a problem where his function would Round off 544.435 to 544.43 but would Round off 544.445 to 544.45 which was the intended one.

After a fair bit of research I could conclude the following:

The error was because he was taking input and providing the output as DOUBLE.

Now MSDN says that DOUBLE is a FLOATING POINT VARIABLE Type. It takes up less space in the memory but is prone to some Rounding problems.

So while storing the above two DOUBLES, .NET stored them as follows:

544.435 ------------- 544.43499999999994543031789362430572509765625
544.445 ------------- 544.4450000000000500222085975110530853271484375

So doing a Multiplication with the precision factor (100 in this case) and taking a Math.Floor of that number converts

544.435 to 544.43
&
544.445 to 544.45

More explanation about the Binary Floating point could be found at

http://www.yoda.arachsys.com/csharp/floatingpoint.html

BTW, the solution to the above mentioned problem is using something that is Fixed Point Variable and not a Floating Point Variable like Double.

So using DECIMAL instead of DOUBLE in the application resolves the Rounding Issue.


-- Ashutosh

No comments:

Post a Comment