How to use the straightness and flatness tolerance calculator
Straightness and flatness are not judged by the largest single reading but by the width of the band the surface wanders through. Enter equally spaced indicator readings and this calculator fits a reference line, reports highest residual minus lowest residual as the geometric error, and compares it with the tolerance on the drawing. Readings are entered in ten-thousandths, so a 0.001 in flatness callout is 10 units here and the result is echoed in inches.
Worked check: readings of 0, 3, 5, 4 and 2 ten-thousandths return a least-squares straightness of 4 units (0.0004 in) with a slope of 0.5 per step and residuals of +2.2 and −1.8, the same 4 units from the end-point line, and a peak-to-valley flatness of 5 units (0.0005 in). Against a 0.0008 in tolerance the straightness uses 50% of the band and the flatness 62.5%.
Pick the method your drawing and inspection plan actually call for. ASME Y14.5 GD&T normally intends a minimum-zone interpretation, and a peak-to-valley number from a surface plate is the conservative side of that. Note also that Y14.5 applies Rule #1, the envelope principle, to a feature of size by default, while ISO GPS treats size and form as independent unless the envelope requirement is invoked. Recheck any borderline fail with a minimum-zone evaluation, and confirm that your measurement spacing really is uniform.
Frequently asked questions
The end-point reference line is drawn through the first and last readings, so if those two points sit high or low the error in the middle is exaggerated. The least-squares line minimizes the sum of squared residuals for every point, so it returns an equal or smaller value.
The spread between the highest and lowest indicator reading on a surface plate is always equal to or larger than a true minimum-zone flatness value, so it is conservative. A pass is safe, but a borderline fail should be rechecked with a minimum-zone evaluation on a CMM before the part is rejected.