ChimneySweep®:   A Step-By-Step Look at the Job Creation Process

As we have said, ChimneySweep® uses a “job-oriented approach” to the process of Paradox® table-repair. Everything that ChimneySweep needs in order to do a particular task is fully contained in “a job.” Once a job is built, it can be run with just a double-click. But how do you build a job?

It's easy! The Job Editor, which is available in Traditional and Professional Editions (but not Runtime), is a fully-interactive “wizard” which walks you through the entire process of building (or changing) a job, one step at a time. The Job Editor will prompt you for everything that ChimneySweep needs to know. All of the most-frequently-used options will have been selected for you already by-default. There's also detailed context-sensitive on-line help available in the product for every page, and for every option, in the Job Editor.

Here is a very detailed, step-by-step walkthrough of the entire process.   (Ready?   Fair warning... it's long!   Okay then, here we go!)

Step one:   “What do you want ChimneySweep to do?”

The Job Editor begins by allowing you to select what you'd like ChimneySweep to do with every table that you'll select. You can choose any or all of the following:

Fix “IDAPI In Use” problems:

This option solves an annoying problem that occurs when 16-bit Paradox (or Delphi®) programs crash unexpectedly:   the programs can't be restarted, but instead fail with a message about “IDAPI in use.” ChimneySweep will fix this problem by unloading the IDAPI modules from memory, thereby allowing the applications to once again run normally with no need to restart your computer.

This option does not apply to 32-bit versions of Paradox, which do not exhibit this problem. So, it's mostly obsolete today.
32-bit versions of ChimneySweep simply ignore this option, and future versions of ChimneySweep (which will be 32-bit-only) will omit it altogether.

Remove unused “.LCK” files:

“Lock files” are used by Paradox to manage the locking operations that are needed to prevent two or more users from interfering with one another. If an application crashes, these files might not be removed, causing “ghost locks” to remain. This option removes those files (if they are not in-use), thus safely resolving this problem.

Check tables for internal damage or broken indexes:

You will almost-always select this option, which tells ChimneySweep to check for table-damage. But you could also be running a job, say, that was processing a list of damaged tables that was prepared by a previous job. It's your choice.

The tests that will be performed include:

1. Check table headers for correctness. Make sure that the file is not truncated and that it is completely readable.
2. Check table structural integrity. Make sure that all blocks are complete and that they are correctly linked to one another in both directions with no loops, cross-linked or omitted blocks. All blocks containing data must be on the active chain; those that do not contain data must be on the free-block chain.
3. Check memo-field values and the links to those values. Check space-allocation structures within the memo files.
4. Check primary- and secondary-index integrity. Make sure that the index file structures are completely intact, and that all index-records point to the corresponding data. All records in the table must be properly indexed, and all index-entries must correspond to active records.
5. Check autoincrement fields, if present. Duplicate values will be detected. If the next-value counter in the table header would cause endless key-violations, this will be detected.

Attempt to repair any damaged tables or indexes that are found ...

If you like, a job's sole purpose can be to find problems, not to fix them at this time. A job can also simply be intended to do maintenance only, although such jobs will nonetheless do a cursory amount of structural inspection and will not proceed in any “doubtful” situation.

1. Repairable problems in the table headers will be fixed.
2. Block linkage problems will be repaired so that all blocks containing data are accounted-for, and all blocks not-containing data will be made available for re-use. Any loops or mismatched bi-directional links will be resolved.
3. Optionally (see below), memo-field values will be re-linked to the records which own them, if it is possible to do so.
4. Indexes which do not agree with the table will be re-generated.
5. Autoincrement field-counters will be reset if the present value would cause endless key-violations. (Otherwise, the counter is not disturbed.) Does not attempt to alter any existing duplicated values within records.

... also attempt to fix the value of memo (BLOB) fields:

The storage-format used for memo-type values (memo, formatted-memo, binary, graphic ... so-called “BInary Large OBjects”) is inherently more-uncertain than the other file structures used in the Paradox system, so we give you a separate choice of whether or not to attempt to repair these problems.

When you select this option, ChimneySweep will examine all of the non-empty memo-type fields, verifying that the corresponding data does in fact exist in the memo-file and that it appears to be the same data. ChimneySweep will confirm that two fields do not share the same data, and that the space-allocation structure within the memo file appears to be correct. If a memo-field appears to have become disconnected from its value, ChimneySweep will attempt to locate the value and reconnect it.

ChimneySweep will “err on the side of caution,” electing not to make repairs at all when its choice might be ambiguous or incorrect.
(Unlike other products, it will not “simply remove” values that appear to be corrupt.)

Create a printable report:

A printable report-file summarizing the results of the job in considerable detail can be produced. In ChimneySweep 5.1, this is a simple text-file and there is only one of them, covering the most-recently executed job.

Create an ASCII file containing the names of damaged tables:

This option creates a file containing a list of the names of damaged tables that were found. This file is intended to be used as input to future jobs.

Step two:   where are the tables?

The next thing that the Job Editor wants to know is, where are (or, as the case may be, where will be...) the tables that you want ChimneySweep to look at?

(Most of the time, all of the locations you choose will be ones that exist upon or are accessible to the computer that you're using now. But, strictly speaking, you can also specify locations that you know will exist upon the computer(s) on which the job will eventually be run.)

There are three choices:

(1) In “:WORK: and :PRIV:”:

If you are using Paradox for Windows®, two special aliases are defined, referring to the current working-directory and the current private-directory (respectively) on this computer.

Jobs built using this option will automatically determine the then-current Paradox for Windows settings for these two aliases “on the fly,” each time the job runs.

(2) In aliases and/or directories that you specify:

As you probably know, an “alias” is a symbolic name for a database, which you define either in Paradox or with the BDE (or IDAPI) Configuration Utility. A Paradox “database,” in turn, corresponds to a directory. When you select this option, you'll be presented with a page where you can choose any number of aliases and/or directories.

The default behavior is that all tables within the chosen locations will be processed. If you wish, you can identify specific table-names (or wildcard-patterns such as “sales*.db”) for any of those locations.

You can also specify “pass numbers” for each entry, to specify jobs that run through start-to-finish multiple times:   first for “pass 1”, then “pass 2,” and so on until all passes have been completed.

(3) Obtain the list of table-names from a file:

If (for example) you used a previous ChimneySweep job to build a list of tables that were broken, you can use this option to repair the tables that were identified.

Step three:   what are the master passwords?

Paradox tables can be protected by “master passwords.” ChimneySweep respects Paradox password-security and therefore requires you to provide all of the master-passwords that may be required when the job eventually runs. (It also needs this information to “capture table-information” when building a job that uses any of those options.) If when processing any table (i.e. when the job runs...) ChimneySweep finds that it has not been given the correct master password(s) that are required, it will not process that table. ChimneySweep does not prompt for passwords at runtime.

You should bear in mind that different tables within a database can be protected by different master-passwords. Remember also that databases can be related to other databases, through table-lookups for example, which may have different passwords. All of the master-passwords that may be needed anywhere must be supplied.  

The master-passwords that you enter are scrambled twice to prevent disclosure, even when using a binary file-editor. The Job Editor never reveals your passwords.

Step four:   how should tables be checked for damage?

You can choose between these three options:

1. Try to “open” the table, and if that succeeds, assume the table is okay.
2. Unconditionally verify each table. (This is usually the best choice.)
3. Unconditionally assume that all of the tables are in need of repair.

Step five:   extended repair features

These options allow you to make some specific choices about how you want ChimneySweep to proceed in cases where it needs to attempt table-repair.

Use fast table verification:

Nearly always, you will want to use ChimneySweep's high-speed verification features, but you can, in fact, turn them completely off ... in which case ChimneySweep will only use the Borland-supplied Table Repair (TUTILITY/TUTIL32) facilities instead. (Available, but not recommended.)

Attempt table repairs in-place:

Nearly always, you will want to use ChimneySweep to repair tables, when it can do so, by directly updating the affected files. But once again, you can turn these features completely off. (Available, but not recommended.)

Suppress the use of Table Repair (TUTILITY/TUTIL32):

At various times, ChimneySweep will “fall back” to the use of the Table Repair facility, either for verification or repair or both. You can turn this behavior completely off, and in fact we recommend that you do so. The reason for this recommendation is simple:   TUTILITY is known to have some very serious bugs that will probably never be fixed.

Step six:   table maintenance

As we have said quite repeatedly, ChimneySweep's purpose is not limited to repair. It's intended for maintenance, too. Here's where you can select the most commonly-used maintenance options:

Rebuild table indexes:

Indexes, as you know, are what lets you find information quickly without searching through the entire table. As time goes on, these indexes can sometimes take longer and longer to find things. This option will rebuild those indexes to restore maximum performance.

“Pack” table files:

Paradox tables “recycle” the space used by deleted records. Records are stored in blocks, and when a block becomes completely empty it is put on a separate chain to be re-used. The file itself never shrinks. This process rebuilds the table so that it uses a minimal number of blocks.

Use ChimneySweep-supplied high speed routines:

Both of the above operations can be done either by BDE itself, or by ChimneySweep-supplied replacement routines which may be much faster.

(Please note that you should not select this option in ChimneySweep 5.1 if the “table language” is not conformant with the US-ASCII collating sequence.
This deficiency will be corrected in future versions.)

Step seven:   advanced options

The Job Editor also provides a few “advanced options.” These are choices that don't have to use and you may never need to use. Some of them may only be specified by Professional Edition users, although all Editions will respect them at run-time.

Display the job-status window:

ChimneySweep can display an informative “status window” that shows you the ongoing progress of the job. You can elect to omit this window.

Subsequent releases of ChimneySweep will not display this status-window at all.

On-disk backups:

ChimneySweep can make backup copies of all tables to another location on disk. It can also do this only for tables that it is about to attempt to repair.

Shadow tables:

This highly-advanced option, available to Professional Edition users only, allows you to specify the location of a set of tables which, you assert, are “known good.” ChimneySweep can use this information as an authoritative source for restoring corrupted table headers. (It will, of course, make sure that the “shadow” tables are okay!)

Capture table-information:

An alternative that is much better than “shadow tables” is to allow ChimneySweep to capture information about the database as it is building the job, putting that “known good” information directly into the job for future reference. (Once again, the information will be scrambled.)

The information that can be captured and used includes any or all of the following:

• Referential-integrity information.
• Validity checks. (Min/max/default values, pictures, etc.)
• Index definitions, so that indexes that are entirely missing can be rebuilt.
• Auxiliary passwords and associated rules.

Obviously, you can use this option only when you know that the database you now have does not contain any errors. (And, yes, ChimneySweep will check.)

You must provide the correct master-password(s) when building jobs that use this option, as well as those (if different...) which are needed at runtime.
Note:   With ChimneySweep 5.1, you may (incorrectly...) be presented with a password-prompt at job-build time if you fail to supply all the passwords needed. This deficiency will be corrected in future releases... an error-message should have been returned outright, i.e. with no prompt.

Password-protected jobs:

You can specify a password that must be supplied in order to edit the job, and a separate one that must be supplied in order to run it. Both of these passwords are entirely unrelated to table master-passwords. The passwords are case-sensitive.

“Delivered” (run-only) jobs:

When you save a job, you can also save a copy in the so-called “delivered” format. These jobs cannot be edited, either in the Job Editor or in the (Professional Edition) Script Editor, because they entirely omit the so-called “source code.” All that ChimneySweep will do with such jobs is run them.

Because of this absence of source-code, “delivered” jobs are not backward-compatible. If you supply delivered jobs to your users, and then upgrade their version of ChimneySweep, you should also generate and supply new delivered-jobs to them, using that same version. (Obviously, you will need your non-delivered copies in order to do this.)

 

“Whew!   That's a lot of options!”

Yes, indeed! ChimneySweep is a very powerful product that's designed to be a very all-inclusive solution to the overall problem of database repair and maintenance. But in spite of all this, the Job Editor makes the entire process of building a job very step-by-step and easy.

• The most-common and usually most-appropriate options are already selected for you.
• Once you have built a job, exactly the same process can be used at any time to edit it.
• Best of all, when your users want to run a job, they don't have to think about any of this!   (Nor do you!)   They just double-click the mouse and ... “it just works.” Meanwhile, you know that all of the steps that you specified in the Job Editor will be carried out:   start-to-finish, exactly as you called for, each and every time the job runs.
• ... And you didn't have to “explain” anything!

© Copyright 1996-2008 Sundial Services. All rights reserved. | Terms of Use; Copyright; Trademarks | Privacy Policy | Website Credits | Site last updated:   May 5, 2008.