## Introduction: Do Resistors Really Hold 5% Tolerance?

I just bought a bunch of 560 Ohm gold band resistors from a local surplus store. They were a nickel each. Is it possible to make millions of parts, sell them so cheap, and maintain the +/- 5% tolerance specified by the gold band? The tool used to answer this question is Statistical Process Control (SPC).

You will need:

A digital multimeter

A bunch of resistors the same value. (I measured thirty)

The spreadsheet template located in step five.

## Step 1: Statististical Process Control

To understand **SPC** it is necessary to understand the following terms:

**Nominal** is the exact measurement you are trying to achieve. But in reality nothing is ever perfect. There is variation in all processes, the **Tolerance** sets the limits of how much variation is acceptable. The Lower Specification Limit (**LSL**) and Upper Specification Limit (**USL**) are the limits of the tolerance.

The **Mean** is the arithmetic average of a set of values, or distribution. The **Median** is the point where half of the values are less and half are more. The **Mode** is the most common value. In an ideal situation the Nominal, Mean, Median, and the Mode will all be the same.You can get a rough estimate of how consistent your process is running by comparing them.

Process Capability (**CP**) is the measurement to determine if the process is capable of holding the tolerance allowed. To find the CP first you find the Standard Deviation. The Mean + ( Standard Deviation * 3) gives you the Upper Control Limit (**UCL**). Next find the Lower Control Limit (**LCL**), Mean - ( Standard Deviation * 3). The capability is the ratio of the specification limits over the control limits, (CP = (USL - LSL) / (UCL - LCL). If the CP equals one the control limits fit exactly within the specification limits.

You want it to be larger than one to give you some room for error. The reason for using +/- three standard deviations is because in a normal distribution 68.2% of the values will fall within 1 standard deviation. 95.5% will fall within 2 standard deviations, and 99.7% will fall within three. These figures are mathematical constants known as the Empirical Rule. As the amount of variation increases the standard deviation will also increase.

**CPK** is the measurement of how well centered the Mean is to the Nominal, if they are identical the CPK will equal the CP. More variation between the two in either direction will result in a lower CPK.

The standard most widely used in industry is a CPK of 1.3.

The math mentioned here gets complicated, but it is easy to estimate. With a normal distribution over the center half of the tolerance with the mean centered on the nominal your CPK will be approximately 1.3. An even distribution over the center half of the tolerance with the mean centered on the nominal will give you a CPK of approximately 1.1. If the CP and CPK both equal exactly one 99.7% of the parts will be within the tolerance. The other .3% will be bad.

## Step 2: Capable and in Control

In this illustration both of the control limits (LCL and UCL, plus and minus three times the standard deviation) are within the specification limits (LSL and USL) meaning the process is capable.

The vertical line represents the nominal specification and the top of the curve represents the process mean.

The mean is well centered on the nominal so the process is said to be in control.

CP and CPK are equal.

## Step 3: In Control But Not Capable

Here the control limits fall outside the specification limits, but the mean is well centered on the nominal.

The process is in control but not capable.

There is to much variation in the process. it must be improved so the control limits fall within the specification limits.

Again CP and CPK are equal.

## Step 4: Capable But Out of Control

In this illustration the control limits would fall within the specification limits if the mean was centered on the nominal.

The process is capable but out of control.

Make an adjustment to bring the mean in line with the nominal.

Here the CPK is lower than the CP.

## Step 5: Enter the Data and Get the Results

Download the two spreadsheets at the bottom of the step. 560Ohm.ods is my data, the same as shown here. SPC.ods is a blank SPC template you can use for this or any other SPC study.

Statistical Process Control

Capability Study Worksheet

Company AxMan

Order # 560 Ohm +/-5% resistor

USL - 588

LSL - 532

Sample

1 - 557

2 - 557

3 - 560

4 - 555

5 - 555

6 - 559

7 - 560

8 - 558

9 - 558

10 - 560

11 - 560

12 - 554

13 - 560

14 - 558

15 - 556

16 - 560

17 - 558

18 - 556

19 - 560

20 - 564

21 - 555

22 - 561

23 - 558

24 - 555

25 - 559

26 - 558

27 - 558

28 - 556

29 - 557

30 - 557

Nominal - 560.0000

Mean - 557.9667

Median - 558.0000

Mode - 558.0000

CP - 4.2036

CPK - 3.8983

LSL - 532.0000

LCL - 551.3057

UCL - 564.6277

USL - 588.0000

High Sample - 564.0000

Low Sample - 554.0000

The CP and the CPK are both a long way above 1.3, these resistors are within the specified tolerance with plenty of room to spare.

*The resistors were purchased at Axman, a surplus store in St.Paul MN. I have no financial interest in Axman, I am just a customer.*

### Attachments

Participated in the

Explore Science Contest

## 16 Discussions

5 years ago on Introduction

If you are interested in pursuing this further, you could conduct a gauge repeatability and reproducibility study. This would allow you to analyze how much error is attributable to the operator and the to the test equipment.

5 years ago on Introduction

For more information on statistical process control and capability

studies on more electronic components please check out my SPC

collection:

https://www.instructables.com/id/Statistical-Process-Control-3/

5 years ago on Introduction

Is your meter calibrated?

Reply 5 years ago on Introduction

After much searching I finally found a 560 Ohm 1% tolerance resistor. It's on order now and when it arrives I can check my meter to a higher standard. I will post the results here.

Reply 5 years ago on Introduction

My 1% resistor arrived yesterday. It measured 559 Ohms on my meter.

Reply 5 years ago on Introduction

Hi,

If you really want to test your meters calibration, at least 3 resistor values are needed - one low, one medium and one high, to check the precision over a range.

On say a 200k FSD range, suitable values could be e.g. 20k, 60k and 180k (just use a value near the 60k). Then you can calculate the deviation by inter- and extrapolation.

1% resistors are not gonna cut it, if you want it to be as exact as possible, but you may wanna fetch the datasheet of Vishays E102Z film resistors from here: http://www.vishaypg.com/foil-resistors/list/produc...

They can be had down to 0.005% tolerance (with a tempco of 0.2ppm/°C and the joy doesn't stop here.

Their Burden-Z resistors has got impressive data as well, but only goes to 500 Ohms max - the datasheet is a leisure to read anyway - for anyone with even a remote interest in electronics.

It can be found at: http://www.vishaypg.com/doc?63258

Please know, that I'm not trying to bash your posts in any way, I find them very informative and whether your meter is off or not, it will still be good data, if viewed as relative rather than exact values - even a non-adjusted meter will be precise enough for relative data, as the spread of values are so small. So please just consider it as FYI.

Thanks for posting it - while I don't care for the spread in 5% resistors from a design stand (if I need better, I specify better), it's definitely nice to get a brush up on the statistics behind :)

Reply 5 years ago on Introduction

That sounds interesting. It would be fun to do a calibration check on my meter. I have to decide if I want to spend $14 apiece on those resistors.

Reply 5 years ago on Introduction

Ouch! I didn't see a price for them - sure sounds a bit steep. Personally, I wouldn't bother and most of the people that find it ever so important to have a meter adjusted to µV precision are just kidding themselves ;)

5 years ago on Introduction

Saying nothing about other deficiencies in your methodology, we'll just start with the fact that your sample is invalid.

Reply 5 years ago on Introduction

This was clearly a study done out of personal interest - methodology does not need to be perfect. In fact, I guarantee you many published scientific studies are not perfect due to any number of "deficiencies". The point of any study is to make it as good as you can with the resources and time available.

All these types of comments (criticism without the constructive part) do is make people second guess their want to explore science.

Keep on, JV!

Reply 5 years ago on Introduction

Thank you.

Reply 5 years ago on Introduction

Why do you say my sample is invalid? I pulled the pieces at random from a bin of hundreds.

Reply 5 years ago on Introduction

I can't comment on your SPC methodology - I didn't do very well in my college statistic class, but to comment on the sample being invalid, unless the resistor manufacturing process has changed over the years, the resistors are manufactured, tested, sorted and then marked for tolerance. Therefore there is a very real possibility that your sample is not random.

This brought to mind a co-worker that wanted to get close to a particular resistor value. However all we had in the lab were +-10% resistors. He figured there was a possibility that there might be one close to the value he wanted and measured them all. It turned out that non of them where within +-5% - because they would have been marked and sold as such.

Thank you for the presentation. Perhaps I should see if age has improved my statistical comprehension. I wonder what the probability of that is!

Reply 5 years ago on Introduction

It was as random as I could make it given a bin with 300+ resistors in it. I don't know if they were all from the same manufacturing lot, or different ones.

5 years ago

Good presentation! A complete error analysis would also require the tolerance and accuracy of the multimeter as well as changes in the testing environment conditions. Excellent though!

5 years ago

It's also worth mentioning that the accuracy of your multimeter should be taken into consideration as well as how long is been since it's last calibration check.