Calculating fret positions requires a bit of math, and is actually based in pretty heavy duty theory. There are numerous fret calculators available online, and pre-calculated standard fret distances. These are great, and since they are all measured from the nut, there is little chance of a "cumulative error". Since each fret position is measured from the nut, each has the same tolerance. Using these requires very careful measurement and marking the position, prior to cutting or marking the position.
There is another method, that was often used, which relies on geometry. I have been building a lot of "cigar box" type instruments, ukuleles, and various other instruments (dulcimers, etc), and the scale lengths are all over the place, from around 13" for a ukulele, to around 25" for a dulcimer. This method can be used for any of them.
The rule is sometimes called the "rule of 18". Basically, the position of the next fret, is the scale length, minus the displacement of the previous fret, divided by 18. It was close enough, and is still pretty close. Although the accepted number today is 17.817. This places the 12th fret (the octave) at exactly the mid-point between the nut and bridge.
This method still requires math, and careful measurement. Presented here is a method based on the rule of 18, but laid out geometrically. Basically, a right triangle is constructed. The long leg is the scale length. The short leg is the scale length divided by 17,817. The figures should explain the method better than words can. I enjoy using this technique, and with care, it has yielded great results (as tested by striking the harmonic at the twelveth fret) .
Step 1: Materials
This method requires striking arcs, and drawing exact 90 degree angles. As with anything else, the more precise and accurate the tools, the better the result. One concern with this method is that each fret is measured from the previous fret. If there is a systematic error (ie, measuring a hair too much or a hair too little) fret to fret, the error compounds as you go down the neck. It might seem nit-picky, but sharpen your pencil often. Good, sharp, small lines will minimize the slop in the measurement.
A good compass. The grade school compass can work, but is far from ideal. If you are building anything, spring for a good drafting set from any decent craft store. (around $20). A good mechanical pencil, sharpener, and compass.
A good protractor. Since this will be used only to make 90 degrees, a good 90 degree square ( a precise one, from a drafting set) is probably easier and more precise.
Something to draw on. brown paper bag works alright, but it can actually flex enough and even stretch, to cause problems. The best way is to use a sheet of card stock.
A long straight edge. Every home should have a good, metal yard or meter stick. Perfect for drawing nice straight lines.
Step 2: The Method
Start with a very sharp pencil, and a very straight straight-edge. Remember--cumulative errors. Work hard to minimize this. Even the thickness of a pencil line can cause error.
Start by drawing a straight line, a bit longer than your scale length. Then, on one end, draw an exact 90 degree perpendicular line to it. These form the 2 sides of a right triangle. The first measurement--the one that you should mark on this second line, is the scale length divided by 17.817. Mark this distance very carefully. Now draw the hypotenuse between the above distance, and the long leg of the triangle.
Now, set your compass--with a very sharp pencil, and the point EXACTLY in the intersection of the two lines. Strike an arc. Where this mark crosses the long leg (marked "scale length"), draw a very precise perpendicular line.
Step 3: Continuing....
Set your compass with the point at the intersection of the arc and the long leg, and adjust it, so that it falls exactly on where the perpendicular crosses the hypotenuse. Notice that this distance is slightly shorter than the previous one.... Do this exact procedure, fret to fret, as you go down the neck.
One quick check, to know that you are not suffering a cumulative error, is that you can measure from the nut, to the 12th fret, which should equal exactly 1/2 the scale length. Which should also equal exactly the distance to the bridge.
The method works out very well, again, if you are careful with drawing the angles, striking the arcs....
I have used this for all kinds of projects--lately, mostly "cigar box" type guitars. For a fretless instrument, you can just put a mark on the finger board, along either edge, and draw a line between them, marking where the fret position should be.
To reiterate--sharp pencils, double check by measuring to the 12th fret, and to the bridge. And have fun--it's a cool, and actually very, very old method. Let me know if it works for you. There is a great web resources that can be found here: http://www.liutaiomottola.com/formulae/fret.htm