Statistical Process Control for Resistors, Part 2




Introduction: Statistical Process Control for Resistors, Part 2

About: Jack passed away May 20, 2018 after a long battle with cancer. His Instructables site will be kept active and questions will be answered by our son-in-law, Terry Pilling. Most of Jack's instructables are tu...

A while back I wrote an instructable where I did a capability study to find out if my gold band resistors really held +/- 5% tolerance.

Located here:

In that study I was looking at 560 Ohm +/-5% resistors.

This time I am doing the same study on 560 Ohm +/-10% resistors.

Some of the comments asked some interesting questions:

  • What about the calibration of the meter?
I found a +/-1% resistor and tested it. It measured 559 Ohms.
  • The sample was not random.
In this and the first study the sample was as random as I could make it.
In the first study I pulled the parts out of a bin of hundreds in a local surplus store. I have no idea who manufactured them or if they were from the same lot number.
In this study the parts were ordered from Digikey and they were manufactured by Ohmite. I just accepted the parts they sent me. I did not ask for anything special. I don't know if they are all from the same lot number.

Most capability studies are not random. I worked as a QA manager at a medium sized machine shop that specialized in large quantity orders of small parts. There were two different capability studies we did.

The first time we made a part the buyer wanted a capability study of 30 pieces showing a CPK of 1.3 on all critical features. (I will explain CPK later on in the instructable.)

The other study was in process, timed inspections for the entire run of the part. The last parts off the machine were checked every hour.

To do a study any more random than what I did would require access to the manufacturing plant.

This is one of the more interesting comments from the previous study (Edited by me):

Unless the resistor manufacturing process has changed over the years, the resistors are manufactured, tested, sorted and then marked for tolerance. Therefore there is a very real possibility that your sample is not random.
This brought to mind a co-worker that wanted to get close to a particular resistor value. However all we had in the lab were +/-10% resistors. He figured there was a possibility that there might be one close to the value he wanted and measured them all. It turned out that none of them were within +/-5%, because they would have been marked and sold as such.

Resistors must be manufactured using a more modern process now, all except one of the +/-10% resistors I checked were within +/-5%.

Steps one through four are the same as in my original study of +/-5% resistors.

Teacher Notes

Teachers! Did you use this instructable in your classroom?
Add a Teacher Note to share how you incorporated it into your lesson.

Step 1: Statistical Process Control

To understand SPC it is necessary to understand the following terms:

Nominal is the exact measurement you are trying to achieve. But in reality nothing is ever perfect. There is variation in all processes, the Tolerance sets the limits of how much variation is acceptable. The Lower Specification Limit (LSL) and Upper Specification Limit (USL) are the limits of the tolerance.

The Mean is the arithmetic average of a set of values, or distribution. The Median is the point where half of the values are less and half are more. The Mode is the most common value. In an ideal situation the Nominal, Mean, Median, and the Mode will all be the same.You can get a rough estimate of how consistent your process is running by comparing them.

Process Capability (CP) is the measurement to determine if the process is capable of holding the tolerance allowed. To find the CP first you find the Standard Deviation. The Mean + ( Standard Deviation * 3) gives you the Upper Control Limit (UCL). Next find the Lower Control Limit (LCL), Mean - ( Standard Deviation * 3). The capability is the ratio of the specification limits over the control limits, (CP = (USL - LSL) / (UCL - LCL). If the CP equals one the control limits fit exactly within the specification limits.

You want it to be larger than one to give you some room for error. The reason for using +/- three standard deviations is because in a normal distribution 68.2% of the values will fall within 1 standard deviation. 95.5% will fall within 2 standard deviations, and 99.7% will fall within three. These figures are mathematical constants known as the Empirical Rule. As the amount of variation increases the standard deviation will also increase.

CPK is the measurement of how well centered the Mean is to the Nominal, if they are identical the CPK will equal the CP. More variation between the two in either direction will result in a lower CPK.

The standard most widely used in industry is a CPK of 1.3.

The math mentioned here gets complicated, but it is easy to estimate. With a normal distribution over the center half of the tolerance with the mean centered on the nominal your CPK will be approximately 1.3. An even distribution over the center half of the tolerance with the mean centered on the nominal will give you a CPK of approximately 1.1. If the CP and CPK both equal exactly one 99.7% of the parts will be within the tolerance. The other .3% will be bad.

Step 2: Capable and in Control

In this illustration both of the control limits (LCL and UCL, plus and minus three times the standard deviation) are within the specification limits (LSL and USL) meaning the process is capable.

The vertical line represents the nominal specification and the top of the curve represents the process mean.

The mean is well centered on the nominal so the process is said to be in control.

CP and CPK are equal.

Step 3: In Control But Not Capable

Here the control limits fall outside the specification limits, but the mean is well centered on the nominal.
The process is in control but not capable.

There is to much variation in the process. it must be improved so the control limits fall within the specification limits.

Again CP and CPK are equal.

Step 4: Capable But Out of Control

In this illustration the control limits would fall within the specification limits if the mean was centered on the nominal.
The process is capable but out of control.

Make an adjustment to bring the mean in line with the nominal.

Here the CPK is lower than the CP.

Step 5: Capability Study Data

The following is the data. First the values are for the resistors with the specification limits set at +/-10%. The second set of numbers are the same resistors with the specification limits set at +/-5%.


The files are:

  • 560-10.ods - The data for this study.
  • SPC.ods - An empty template you can use for a capability study of your own.

Capability Study Worksheet

Company Ohmite

Order # 560 Ohm +/-10% 2 watt resistors










































CPK...... 1.1577.....0.5141

LSL.......... 504..........532

LCL 510.8600......510.8605

UCL 597.8728.....597.8728

USL 616.0000.....588.0000

High Sample 588.0000

Low Sample 524.0000



  • 524 @
  • 525
  • 526
  • 527
  • 528
  • 529
  • 530
  • 531
  • 532
  • 533
  • 534 @@
  • 535
  • 536
  • 537 @
  • 538
  • 539 @
  • 540
  • 541
  • 542 @
  • 543 @
  • 544
  • 545 @
  • 546
  • 547 @
  • 548 @
  • 549 @
  • 550 @
  • 551 @@
  • 552
  • 553 @
  • 554 @@
  • 555
  • 556 @
  • 557
  • 558 @@
  • 559
  • 560 @
  • 561 @
  • 562 @
  • 563
  • 564
  • 565
  • 566
  • 567
  • 568 @
  • 569
  • 570 @@
  • 571 @
  • 572
  • 573
  • 574
  • 575
  • 576 @
  • 577
  • 578 @
  • 579
  • 580
  • 581
  • 582
  • 583
  • 584
  • 585
  • 586
  • 587
  • 588 @


This shows a wide distribution over the center half of the tolerance, the mean being about -1%. All except the one that measured 524 Ohms are within +/-5%.

With a 1.3 CPK you can expect 1 in 1,744,278 parts to be out of spec. That's +/- five standard deviations. With the CPK at 1.1577 it's approximately four meaning 1 in 15787 will be out of spec.

If you throw out the 524 measurment the CPK jumps to 1.26, and the CP jumps to 1.38.

By doing timed inspections a small change in the process will make a big difference in the CP and CPK, alerting you to the problem.

Be the First to Share


    • Backyard Contest

      Backyard Contest
    • Silly Hats Speed Challenge

      Silly Hats Speed Challenge
    • Arduino Contest 2020

      Arduino Contest 2020

    2 Discussions


    4 years ago on Introduction

    For more information on statistical process control and capability
    studies on more electronic components please check out my SPC


    5 years ago on Introduction

    Interesting. I have seen the CPK specification used, but never really thought much about it. Thanks.