Table of Contents:
What files SWISH-E indexes and how they are indexed, and where the index is
written can be controlled by a configuration file. The configuration file
is passed to swish as a command line argument by using the -c
switch (see SWISH-RUN).
The configuration file is a text file composed of comments, blank lines, and configuration directives. Order of the directives is not important. Some directives may be used more than once in the configuration file, while others can only be used once (e.g. additional directives will overwrite preceding directives). Case of the directive is not important -- you may use upper, lower, or mixed case.
Comments are any line that begin with a ``#''.
# This is a comment |
Directives may take more than one parameter. Enclose single parameters that include whitespace in quotes (single or double). You may use a backslash to escape the next character.
ReplaceRules append "foo bar" <- define "foo bar" as a single word ReplaceRules append foo\ bar <- same thing |
If you need to include a quote character in the value either use a backslash to escape it, or enclose it in quotes of the other type.
FileFilter .foo foofilter "'%p'" <- unix FileFilter .foo foofilter \'%p\' <- unix same thing FileFilter .foo foofilter '"%p"' <- windows uses double-quotes |
Commented example configuration files are included in the F<config> directory of the SWISH-E distribution. |
Typically, configuration file the directives are grouped together in some logical order -- that is, directives that control the source of the documents would be grouped together first, and directives that control how each document is filtered or its words index in another group of directives. (The directives listed below are grouped in this order).
You may also split your directives up into different configuration files
and specify more than one configuration file. This allows you to have a
master configuration file used for many different indexes, and smaller
configuration files for each separate index. You can specify the different
configuration files when running from the command line with the -c
switch, or you may include other Configuration file with the
IncludeConfigFile directive.
Some command line arguments can override directives specified in the configuration file. Please see also the SWISH-RUN for instructions on running SWISH-E, and the SWISH-SEARCH page for information and examples on how to search your index.
The configuration file is specified to SWISH-E by the -c
switch. For example,
swish-e -c myconfig.conf |
The configuration file directives are listed below in these groups:
Administrative Headers Directives -- You may add administrative information to the header of the index file.
Document Source Directives -- Directives for selecting the source documents and the location of the index file.
Document Contents Directives -- Directives that control how a document content is indexed.
Directives for the File Access method only -- These directives are only applicable to the File Access indexing method.
Directives for the HTTP Access Method Only -- Likewise, these only apply to the HTTP Access method.
Directives for the prog Access Method Only -- These only apply to the prog Access method.
Document Filter Directives -- This is a special section that describes using document filters with Swish-e.
[ TOC ]
AbsoluteLinks [yes|NO]
BeginCharacters *string
of characters*
BumpPositionCounterCharacters *string*
Buzzwords [*list of buzzwords*|File: path]
ConvertHTMLEntities [YES|no]
DefaultContents [TXT|HTML|XML|WML]
Delay *seconds*
DontBumpPositionOnEndTags *list
of names*
DontBumpPositionOnStartTags *list
of names*
EnableAltSearchSyntax [yes|NO]
EndCharacter *string
of characters*
EquivalentServer *server
alias*
ExtractPath *metaname* [replace|remove|prepend|append|regex]
FileInfoCompression [yes|NO]
FileMatch [contains|is|regex] *regular
expression*
FileRules [contains|is|regex] *regular
expression*
FollowSymLinks [yes|NO]
HTMLLinksMetaName *metaname*
IgnoreFirstChar *string
of characters*
IgnoreLastChar *string
of characters*
IgnoreLimit *integer
integer*
IgnoreMetaTags *list
of names*
IgnoreTotalWordCountWhenRanking [YES|no]
IgnoreWords [*list of stop words*|File: path]
ImageLinksMetaName *metaname*
IndexAdmin *text*
IndexComments [YES|no]
IndexContents [TXT|HTML|XML|WML|TXT2|HTML2|XML2] *file
extensions*
IndexDescription *text*
IndexDir [URL|directories or files]
IndexFile *path*
IndexName *text*
IndexOnly *list
of file suffixes*
IndexPointer *text*
IndexReport [0|1|2|3]
MaxDepth *integer*
MaxWordLimit *integer*
MetaNameAlias *meta
name* *list
of aliases*
MetaNames *list
of names*
MinWordLimit *integer*
NoContents *list
of file suffixes*
obeyRobotsNoIndex [yes|NO]
ParserWarnLevel 0|1|2|3
PreSortedIndex *list
of property names*
PropCompressionLevel [0-9]
PropertyNameAlias *property
name* *list
of aliases*
PropertyNames *list
of meta names*
PropertyNames *list
of meta names* =item *
PropertyNames *list
of meta names* =item *
PropertyNamesDates *list
of meta names*
PropertyNamesNumeric *list
of meta names*
ReplaceRules [replace|remove|prepend|append|regex]
ResultExtFormatName name -x format string
SpiderDirectory *path*
StoreDescription [XML <tag>|HTML <meta>|TXT size]
SwishProgParameters *list
of parameters*
SwishSearchDefaultRule [<AND-WORD>|<or-word>]
SwishSearchOperators <and-word> <or-word> <not-word>
TmpDir *path*
TranslateCharacters [*string1 string2*|:ascii7:]
TruncateDocSize *number
of characters*
UndefinedMetaTags [error|ignore|INDEX|auto]
UndefinedXMLAttributes [DISABLE|error|ignore|index|auto]
UseStemming [yes|NO]
UseSoundex [yes|NO]
UseWords [*list of words*|File: path]
WordCharacters *string
of characters*
XMLClassAttributes *list
of XML attribute names*
[ TOC ]
These configuration directives control the general behavior of SWISH-E.
This directive can be used to include configuration directives located in another file.
IncludeConfigFile /usr/local/swish/conf/site_config.config |
This is how detailed you want reporting while indexing. You can specify numbers 0 to 3 - 0 is totally silent, 3 is the most verbose. The default is 3, so you probably should define this.
IndexReport 1 |
This may be overridden from the command line via the -v
switch (see SWISH-RUN).
Sets the error level when using the libxml2 parser for XML and HTML. libxml2 will point out structural errors in your documents.
0 = no report 1 = fatal errors 2 = errors 3 = warnings |
The exception to this is UTF-8 to Latin-1 coversion errors are reported at level 1. This is because words may be indexed incorrecty in these cases.
Note that unlike other errors generated by swish, these errors are sent to stderr.
Index file specifies the location of the generated index file. If not specified, SWISH-E will create the file index.swish-e in the current directory.
IndexFile /usr/local/swish/site.index |
When enabled, swish will not index any HTML file that contains:
<meta name="robots" content="noindex"> |
The default is to ignore these meta tags and index the document. This tag is described at http://www.robotstxt.org/wc/exclusion.html.
Note: This feature is only available with the libxml2 HTML parser.
NOTE: The following items are currently not available. These items require swish to parse the configuration file while searching.
Enable alternate search syntax. Allows the usage of a basic ``Altavista(c)'', ``Lycos(c)'', etc. like search syntax. This means a search query can contain ``+'' and ``-'' as syntax parameter.
Example:
swish-e -w "+word1 +word2 -word3 word4 word5" "+" = following word has to be in all found documents "-" = following word may not be in any document found " " = following word will be searched in documents |
Using this config directive you can change the boolean search operators of swish-e, e.g. to adapt these to your language. The default is: AND OR NOT
Example (german):
SwishSearchOperators UND ODER NICHT |
SwishSearchDefaultRule defines the default Boolean operator to use if none is specified between
words or phrases. The default is AND
.
The word you specify must match one of the available SwishSearchOperators.
Example:
SwishSearchOperators UND ODER NICHT # Make it act like a web search engine SwishSearchDefaultRule ODER |
The output of swish can be defined by specifying a format string with the -x
command line argument. Using ResultExtFormatName you can assign a predefined format string to a name.
Examples:
ResultExtFormatName moreinfo "%c|%r|%t|%p|<author>|<publishyear>\n" |
Then when searching you can specify the the format string's name swish-e ... -x moreinfo ...
See the -x
switch in SWISH-RUN for more information about output formats.
[ TOC ]
SWISH-E stores configuration information in the header of the index file. This information can be retrieved while searching or by functions in the SWISH-E C library. There are a number of fields available for your own use. None of these fields are required:
These variables specify information that goes into index files to help users and administrators. IndexName should be the name of your index, like a book title. IndexDescription is a short description of the index or a URL pointing to a more full description. IndexPointer should be a pointer to the original information, most likely a URL. IndexAdmin should be the name of the index maintainer and can include name and email information. These values should not be more than 70 or so characters and should be contained in quotes. Note that the automatically generated date in index files is in D/M/Y and 24-hour format.
Examples:
IndexName "Linux Documentation" IndexDescription "This is an index of /usr/doc on our Linux machine." IndexPointer "http://localhost/swish/linux/index.html" IndexAdmin "webmaster" |
[ TOC ]
These directives control what documents are indexed. See also Directives for the File Access method only and Directives for the HTTP Access Method Only for directives that are specific to those access methods.
IndexDir defines the source of the documents for SWISH-E. SWISH-E currently supports three file access methods: File system, HTTP (also called spidering), and prog for reading files from an external program.
The -S
command line argument is used to select the file access method.
swish-e -c swish.config -S fs - file system swish-e -c swish.config -S http - internal http spider swish-e -c swish.config -S prog - external program of any type |
For the File system method of access IndexDir is a space-separated list of files and directories to index. Use a forward slash as the path separator in MS Windows.
You may specify more than one IndexDir directive.
Any sub-directories of any listed directory will also be indexed.
Note: While processing directories, swish-e will ignore any files or directories that begin with a
dot (``.''). You may index files or directories that begin with a dot by
specifying their name with IndexDir or -i
.
Examples:
# Index this directory an any subdirectories IndexDir /usr/local/home/http |
# Index the docs directory in current directory IndexDir ./docs |
# Index these files in the current directory IndexDir ./index.html ./page1.html ./page2.html # and index this directory, too IndexDir ../public_html |
For the HTTP method of access specify the URL's from which you want the spidering to begin.
Example:
IndexDir http://www.my-site.com/index.html IndexDir http://localhost/index.html |
Obviously, using the HTTP method to index is much slower than indexing local files. Be well aware that some sites do not appreciate spidering and may block your IP address. You may wish to contact the remote site before spidering their web site. More information about spidering can be found in Directives for the HTTP Access Method Only below.
For the prog method of access IndexDir specifies the path to the program(s)
to execute. The external
program must correctly format the documents being passed back to swish.
Examples of external programs are provided in the prog-bin directory.
Note: Not all directives work with all methods.
Files with these suffixes will not have their contents indexed.
If the file's type is HTML (as set by IndexContents or DefaultContents) then the file will be parsed for a HTML title and that title will be
indexed. Note that you must set the file's type: .html and .htm
are NOT type HTML by default.
If a title is found, it will still be checked for FileRules title
, and the file will be skipped if a match is found. See FileRules.
If the file's type is not HTML, or it is HTML and no title is found, then the file's path will be indexed. For example, you might wish to search for image files by file name.
Example:
NoContents .gif .xbm .au .mov .mpg .pdf .ps |
Note: This feature only works for files that would be indexed in absense of the NoContents directive. That is, if you use IndexOnly then you must specify in IndexOnly the same suffixes listed in NoContents.
A -S prog
program may set the No-Contents: header (to anything) to enable this feature for a specific document
(althought it would be smarter for the -S prog
program to simply only send the pathname or title to be indexed.
ReplaceRules allows you to make changes to file pathnames before they're indexed. These changed file names or URLs will be returned in search results.
For example, you may index your files locally (with the File system indexing method), yet return a URL in search results. This directive can be used to map the file names to their respective URLs on your web server.
There are five operations you can specify: replace, append, remove, prepend, and regex They will parse the pathname in the order you've typed these commands.
This directive uses C library regex.h regular expressions.
replace "the string you want replaced" "what to change it to" remove "a string to remove" prepend "a string to add before the result" append "a string to add after the result" regex "/search string/replace string/options" |
Regex is an Extended Regular Expression. The first character found is is the delimiter (but it's not smart enough to use matched chars such as [], (), and {}).
The replace string may use substitution variables:
$0 the entire matched (sub)string $1-$9 returns patterns captured in "(" ")" pairs $` the string before the matched pattern $' the string after the matched pattern |
The options change the behavior of expression:
i ignore the case when matching g repeat the substitution for the entire pattern |
Examples:
ReplaceRules replace "testdir/" "anotherdir/" ReplaceRules replace "[a-z_0-9]*_m.*\.html" "index.html" |
ReplaceRules remove "testdir/" |
ReplaceRules prepend "http://localhost/" ReplaceRules append ".html" |
ReplaceRules regex "!^/web/(.+)/!http://$1.domain.com/!" replaces a file path: /web/search/foo/index.html with http://search.domain.com/foo/index.html |
ReplaceRules regex "#^#http://localhost/www# ReplaceRules prepend "http://localhost/www" (same thing) |
# Remove all extensions from C source files ReplaceRules remove .c # ERROR! That "." is *any char* ReplaceRules remove \.c # much better... |
The IndexContents directive assigns one of Swish's document parsers to a document, based on the its extension. Swish currently knows how to parse TXT, HTML, and XML documents.
The XML2, HTML2, and TXT2 parsers are currently only available when swish is configured to use libxml2.
Documents that are not assigned a parser with IndexContents will, by default, use the HTML parser. The DefaultContents directive may be used to assign a parser to documents that do not match a file extension defined with the IndexContents directive.
Example:
IndexContents HTML .htm .html .shtml IndexContents TXT .txt .log .text IndexContents XML .xml |
HTML is the default type for all files, unless otherwise specified (and
this default can be changed by the DefaultContents directive. Swish parses titles from HTML files, if available, and keeps
track of the context of the text for context searching (see -t
in SWISH-RUN). HTML and XML files use different tag formats for MetaNames and PropertyNames.
If using filters to convert documents you should include those extensions, too. For example, if using a filter to conver .pdf to .html, you need to tell swish that .pdf should be indexed by the internal HTML parser: FileFilter .pdf pdf2html IndexContent HTML .pdf
See also Document Filter Directives.
Note: Some of this may be changed in the future to use content-types instead of file extensions. See SWISH-3.0
This sets the default parser for documents that are not specified in IndexContents. If not specified the default is HTML.
The XML2, HTML2, and TXT2 parsers are currently only available when swish is configured to use libxml2.
Example:
DefaultContents HTML |
The DefaultContents directive should be used when spidering, as HTML files may be returned without a file extension (such as when requesting a directory and the default index.html is returned).
** This directive is currently not supported **
Setting FileInfoCompression to yes
will compress the index file to save disk space. This may result in longer
indexing times. The default is no
.
Also see the -e
switch in SWISH-RUN for saving RAM during indexing.
[ TOC ]
These directives control what information is extracted from your source documents, and how that information is made available during searching.
ASCII entities can be converted automatically while indexing documents of type HTML. For
performance reasons you may wish to set this to no
if your documents do not contain HTML entities. The default is yes
.
If ConvertHTMLEntities is set no
the entities will be indexed without conversion.
NOTE: Entities within XML files and files parsed with libxml2 are converted regardless of this setting.
META names are a way to define ``fields'' in your XML and HTML documents.
You can use the META names in your queries to limit the search to just the
words contained in that META name of your document. For example, you might
have a META tagged field in your documents called subjects
and then you can search your documents for the word ``foo'' but only return
documents where ``foo'' is within the subjects
META tag.
swish-e -w subjects=foo |
(See also the -t
switch in SWISH-RUN for information about context searching in HTML documents.)
The MetaNames directive is a space separated list. For example:
MetaNames meta1 meta2 keywords subjects |
You may also use UndefinedMetaTags to specify automatic extraction of meta names from your HTML and XML documents, and also to ignore indexing content of meta tags.
META tags can have two formats in your HTML source documents:
<META NAME="meta1" CONTENT="some content"> |
and
<meta1> some content </meta1> |
But this second version is invalid HTML, and will generate a warning if ParserWarningLevel is set (libxml2 only).
And in XML documents, use the format:
<meta1> Some Content </meta1> |
Then you can limit your search to just META meta1 like this:
swish-e -w 'meta1=(apples or oranges)' |
You may nest the XML and the start/end tag versions:
<keywords> <tag1> some content </tag1> <tag2> some other content </tag2> <keywords> |
Then you can search in both tag2 and tag2 with:
swish-e -w 'keywords=(query words)' |
Swish indexes all text as some metaname. The default is swishdefault
, so these two queries are the same:
swish-e -w foo swish-e -w swishdefault=foo |
When indexing HTML swish indexes the HTML title as default text, so when searching swish will find matches in both the HTML body and the HTML title. Swish also, by default, indexes content of meta tags. So:
swish-e -w foo |
will find ``foo'' in the body, the title, or any meta tags.
Currently, there's no way do prevent swish from indexing the title contents along with the body contents, but see UndefinedMetaTags for how to control the indexing of meta tags.
If you would like to search just the title text, you may use:
MetaNames swishtitle |
This will index the title text separately under the built-in swish internal meta name ``swishtitle''. You may then search like
swish-e -w foo -- search for "foo" in title, body (and undefined meta tags) swish-e -w swishtitle=foo -- search for "foo" in title only |
In addition to swishtitle, you can limit searches to documents' path with:
MetaNames swishdocpath |
Then to search for ``foo'' but also limit searches to documents that include ``manual'' or ``tutorial'' in thier path:
swish-e -w foo swishdocpath=(manual or tutorial) |
See also ExtractPath.
MetaNameAlias assigns aliases for a meta name. For example, if your documents contain meta tags ``description'', ``summary'', and ``overview'' that all give a summary of your documents you could do this:
MetaNames summary MetaNameAlias summary description overview |
Then all three tags will get indexed as meta tag ``summary''. You can then search all the fields as:
-w summary=foo |
The Alias work at search time, too. So these will also limit the searh to the ``summary'' meta name.
-w description=foo -w overview=foo |
Allows indexing of HTML links. Normally, HTML links (href tags) are not indexed by swish-e. This directive defines a metaname, and links will be indexed under this meta name.
Example:
HTMLLinksMetaName links |
Now, to limit searches to just the links on pages:
-w links='"home.html"' |
To make swish-e index links as normal text, you may use:
HTMLLinksMetaName swishdefault |
This feature is only available with the libxml2 HTML parser.
Allows indexing of image links under a metaname. Normally, image URLs are not indexed.
Example:
ImagesLinksMetaName images |
Now, if you would like to find pages that include a nice image of a beach:
-w images='"beach"' |
To make swish-e index links as normal text, you may use:
ImageLinksMetaName swishdefault |
This feature is only available with the libxml2 HTML parser.
If this is set true then swish-e will attempt to convert relative URIs extracted from HTML documents for use with HTMLLinksMetaName and ImageLinksMetaName into absolute URIs. Swish will use any <BASE> tag found in the document, otherwise swish will use the file's pathname. The pathname used will be the pathname *after* ReplaceRules has been applied to the document's pathname.
For example, say you wish to index image links under the metaname ``images''.
ImageLinksMetaName images |
if an image is located in http://localhost/vacations/france/index.html and AbsoluteLinks is set to no, then a image within that document:
<img src="beach.jpeg"> |
will only index ``beach.jpeg''.
But, if you want more deatil when searching, you can enable AbsoluteLinks and swish-e will index ``http://localhost/vacations/france/beach.jpeg''. You can then look for images of beaches, but only in France:
-w images=(beach and france) |
This also means you can search for any images within France:
-w images=(france) |
This feature is only available with the libxml2 HTML parser.
This directive defines the behavior of swish during indexing when a meta name is found but is not listed in MetaNames. There are four choices:
error - If a meta name is found that is not listed in MetaNames then indexing will be halted and an error reported.
ignore - The contents of the meta tag are ignored and not indexed unless a metaname has been defined with the MetaNames directive.
index - The contents of the meta tag are indexed, but placed in the main index unless there's an enclosing metatag already in force. This is the default.
auto - This method create meta tags automatically for HTML meta names and XML elements. Using this is the same as specifying all the meta names explicitly in a MetaNames dirictive.
This is similar to UndefinedMetaTags, but only applies to XML documents (parsed with libxml2). This allows indexing of attribute content, and provides a way to index the content under a metaname. For example, UndefinedXMLAttributes can make
<person age="23"> John Doe </person> |
look like the following to swish:
<person> <person.age> 23 </person.age> John Doe </person> |
What happens to the text ``23'' will depend on the setting of UndefinedXMLAttributes:
disable - XML attributes are not parsed and not indexed. This is the default.
error - If the concatenated meta name (e.g. person.age) is not listed in MetaNames then indexing will be halted and an error reported.
ignore - The contents of the meta tag are ignored and not indexed unless a metaname has been defined with the MetaNames directive.
index - The contents of the meta tag are indexed, but placed in the main index unless there's an enclosing metatag already in force.
auto - This method will create meta tags from the combined element and attributes (and XML Class name) This options should be used with caution as it can generate a lot of metaname entries.
See also the example below XMLClassAttribues
.
Combines an XML class name with the element name to make up a metaname. For example:
XMLClassAttributes class |
<person class="first"> John </person> <person class="last"> Doe </person> |
Will appear to swish as:
<person> <person.first> John </person.first> </person> <person> <person.last> Doe </person.last> </person> |
How the data is indexed depends on MetaNames and UndefinedMetaTags.
Here's an example using the following configuation which combines the two directives XMLClassAttributes and UndefinedXMLAttributes.
XMLClassAttributes class UndefinedMetaTags auto UndefinedXMLAttributes auto IndexContents XML2 .xml The source XML file looks like: |
<xml> <person class="student" phone="555-1212" age="102"> John </person> <person greeting="howdy">Bill</person> </xml> |
Swish parses as:
./swish-e -c 2 -i 1.xml -T parsed_tags parsed_text -v 0 Indexing Data Source: "File-System" <xml> (MetaName) <person> (MetaName) <person.student> (MetaName) <person.student.phone> (MetaName) 555-1212 </person.student.phone> <person.student.age> (MetaName) 102 </person.student.age> John </person> <person> (MetaName) <person.greeting> (MetaName) howdy </person.greeting> Bill </person> </xml> Indexing done! |
One thing to note is that the first <person> block finds a class name ``student'' so all metanames that are created from attributes use the combined name ``person.student''. The second <person> block doesn't contain a ``class'' so, the attribute name is combinded directly with the element name (e.g. ``person.greeting'').
This directive can be used to index extracted parts of a document's path. A common use would to to limit searches to specific areas of your file tree.
The extracted string will be indexed under the specified meta name.
See ReplaceRules for a description of the various pattern replacement methods, but you will use the regex method.
For example, say your file system (or web tree) was organized into departments:
/web/sales/foo... /web/parts/foo... /web/accounting/foo... |
And you wanted a way to limit searches to just documents under ``sales''.
ExtractPath department regex !^/web/([^/]+)/.*$!$1! |
Which says, extract out the department name (as substring $1) and index it
as meta name department
. Then to limit a search to the sales department:
swish-e -w foo AND department=sales |
Note that the regex
method uses a substitution pattern, so to index only a sub-string match the entire
document path in the regular expression, as shown above.
See the ExtractPathDefault option for a way to set a value if not patterns match.
Although unlikely, you may use more than one ExtractPath direcive. More than one directive of the same meta name will operate successively (in order listed in the configuration file) on the path. This allows you to use regular expressions on the results of the previous pattern substitution (as if piping the output from one expression to the patter of the next).
ExtractPath foo regex !^(...).+$!$1! ExtractPath foo regex !^.+(.)$!$1! |
So, the third letter is indexed as meta name ``foo'' if both patterns match.
ExtractPath foo regex !^X(...).+$!$1! ExtractPath foo regex !^.+(.)$!$1! |
Now (not the ``X''), if the first pattern doesn't match, the last character of the path name is indexed. You must be clear on this behavior if you are using more than one ExtractPath directive with the same metaname.
The document path operated on is the real path swish used to access the document. That is, the ReplaceRules directive has no effect on the path used with ExtractPath.
The full path is used for each meta name if more than one ExtractPath directive is used. That is, changes to the path used in ExtractPath foo
do not effect the path used by ExtractPath bar
.
This can be used with ExtractPath to set a default string to index under the given metaname if none of the ExtractPath patterns match.
For example, say your want to index each document with a metaname ``department'' based on the following path examples:
/web/sales/foo... /web/parts/foo... /web/accounting/foo... |
But you are also indexing documents that do not follow that pattern and you want to search those seperately, too.
ExtractPath department regex !^/web/([^/]+)/.*$!$1! ExtractPathDefault department other |
Now, you may search like this:
-w foo department=(sales) - limit searches to the sales documents -w foo department=(parts) - limit searches to the parts documents -w foo department=(accounting) - limit searches to the accounting documents -w foo department=(other) - everything but sales, parts, and accounting. |
This basically is a shortcut for:
-w foo not department=(sales or parts or accounting) |
but you don't need to keep track of what was extracted.
SWISH-E allows you to specify certain META tags that can be used as document properties. The contents of any META tag that has been identified as a document
property can be returned as part of the search results along with the rank,
file name, title, and document size (see the -p
and -x
switches in SWISH-RUN).
Properties are useful for returning additional data from documents in search results -- this saves the effort of reading and parsing the source files while reading SWISH-E search results, and is especially useful when the source documents are no longer available or slow to access (e.g. over http).
Another feature of properties is that SWISH-E can use the PropertyNames for
sorting the search results (see the -s
switch).
PropertyNames author subjects |
Two variations are available. PropertyNamesCompareCase
and PropertyNamesIgnoreCase
. These tell swish to either ignore or compare case when sorting results.
The default for
PropertyNames is to ignore the case.
PropertyNames subject PropertyNamesIgnoreCase subject <-- same as previous PropertyNamesCompareCase keyword <-- sort |
The defaults for ``internal'' properties are:
swishtitle -- ignore the case swishdocpath -- compare case swishdescription -- compare case |
These can be overridden with PropertyNamesCompareCase
and PropertyNamesIgnoreCase
.
PropertyNamesCompareCase swishtitle |
Use of PropertyNames will increase the size of your index files, sometimes significantly. Properties will be compress if swish is compiled with zlib as described in the INSTALL manual page.
If swish finds more than one property of the same name in a document the property's contents will be concatinated for strings, and a warning issues for numeric (or date) properties.
This directive is similar to PropertyNames, but it flags the property as being a string of digits that will be stored
as binary data instead of a string. This allows sorting with -s
and limiting with -L
to sort and limit the property correctly.
Swish uses strtoul(3)
to convert the string into an unsigned long integer. Therefore, only
positive integers can be stored.
Future versions of swish may be able to store different property types (such as negative integers and real numbers). This directive may change in future releases of Swish.
This directive is exactly like PropertyNamesNumeric, but it also flags the number as a machine timestamp (seconds since
epoch), and will print a formatted date when returning this property. See -x
in SWISH-RUN.
Swish will not parse dates when indexing; you must use a timestamp.
This allows aliases for a property name. For example, if you are indexing HTML files, plus XML files that are written in English, German, and Spanish and thus use the tags ``title'', ``titel'', and ``título'' you can use:
PropertyNameAlias swishtitle title titel título titulo |
Not that ``swishtitle'' is the built-in property used to store the title of a document, and therefore you do not need to specify it as a PropertyName before use.
By default Swish generates presorted tables while indexing for each property name. This allows faster sorting when generating results. On large document collections this presorting may add to the indexing time, and also adds to the total size of the index. This directive can be used to customize exactly which properties will be presorted.
If PreSortedIndex it is not present in the config file (default action), all the properties will be presorted at indexing time. If it is present without any parameter, no properties will be presorted. Otherwise, only the property names specified will be presorted.
For example, if you only wish to sort results by a property called title
:
PropertyNames title age time PreSortedIndex title |
StoreDescription allows you to store a document description in the index file, and this
description is returned in your search results when the -x
switch is used to include the swishdescription for extended results.
For text documents you specify the type TXT
and the number of characters to capture.
StoreDescription TXT 20 |
For HTML, and XML file types, specify the the tag to use for the description, and optionally the number of characters to capture. If not specified will capture the entire contents of the tag.
StoreDescription HTML <body> 20 StoreDescription XML <desc> 40 |
Note that documents must be assigned a document type with IndexContents or DefaultContents to use this feature.
This directive sets the compression level used when storing properties to disk. A setting of zero is no compression, and a setting of nine is the most compression.
The default depends on the default setting compiled with zlib, but is typicaly six.
This option is useful when using StoreDescription to store a large amount text in properties (or if using PropertyNames with large property sizes).
Properties must be over a value defined in config.h (100 is the default) before compression will be attempted. Swish will never store the results of the compression if the compressed data is larger than the original data.
This option is only available when swish is compiled with zlib support.
TruncateDocSize limits the size of a document while indexing documents and/or using filters. This config directive truncates the numbers of read bytes of a document to the specified size. This means: if a document is larger, read only the specified numbers of bytes of the document.
Example:
TruncateDocSize 10000000 |
The default is zero, which means read all data.
Warning: If you use TruncateDocSize, use it with care! TruncateDocSize is a safty belt only, to limit e.g. filteroutput, when accessing databases, or to limit ``runnaway'' filters. Truncating doc input may destroy document structures for swish-e (e.g. swish may miss closing tags for XML or HTML documents).
TruncateDocSize does not currently work with the prog input source method.
Put yes to apply word stemming algorithm during indexing, else no.
UseStemming no UseStemming yes |
When UseStemming is set to yes
every word is stemmed before placing it in to the index.
The stemming function does not convert words to their root, rather programmatically removes endings on words in an attempt to make similar words with different endings stem to the same string of characters. It's not a perfect system, and searches on stemmed indexes often return curious results. For example, two entirely different words may stem to the same word.
Stemming also can be confusing when used with a wildcard (truncation). For example, you might expect to find the word ``running'' by searching for ``runn*''. But this fails when using a stemmed index, as ``running'' stems to ``run'', yet searching for ``runn*'' looks for words that start with ``runn''.
It's a good idea to create both a stemmed and non-stemmed index and allow your search interface select which index to use.
When UseSoundex is set to yes
every word is converted to a Soundex code before placing it in to the
index.
Soundex was developed in the 1880s so records for people with similar sounding names could be found more readily. Soundex is a coded surname based on the way a surname sounds rather than spelling. Surnames that sound similar, like Smith and Smyth, are filed together under the same Soundex code. This is mostly useful for US English.
Soundex should not be used to search for sound-alike words. Metaphone would be more appropriate for generic sound matching of words. Soundex should only be used where you need to search multiple documents for proper names which sound similar. This is primarily used for indexing genealogical records. This may be useful for indexing other collections of data consisting mostly of names. Many common name variations are matched by Soundex. The only notable exception is the first letter of the name. The first letter is not matched for sound.
It may be a good idea to create both a Soundex and non-Soundex index and allow your search interface select which index to use.
Put yes to ignore the total number of words in the file when calculating ranking. Often better with merges and small files. Default is yes.
IgnoreTotalWordCountWhenRanking no |
The default was changed from no to yes in version 2.2.
Set the minimum length of an word. Shorter words will not be indexed. The default is 1 (as defined in src/config.h).
MinWordLimit 5 |
Set the maximum length of an indexable word. Every longer word will not be indexed. The Default is 40 (as defined in src/config.h).
These settings define what a word consists of to the SWISH-E indexing engine. Compiled in defaults are in src/config.h.
When indexing SWISH-E uses WordCharacters to split up the document into words. Words are defined by any string of non-blank characters that contain only the characters listed in WordCharacters. If a string of characters includes a character that is not in WordCharacters then the word will be spit into two or more separate words.
For example:
WordCharacters abde |
Would turn ``abcde'' into two words ``ab'' and ``de''.
Next, of these words, any characters defined in IgnoreFirstChar are stripped off the start of the word, and IgnoreLastChar characters are stripped off the end of the word. This allows, for example, periods within a word (www.slashdot.com), but not at the end of a word. Characters in IgnoreFirstChar and IgnoreLastChar must be in WordCharacters.
Finally, the resulting words MUST begin with one of the characters listed in BeginCharacters and end with one of the characters listed in EndCharacters. BeginCharacters and EndCharacters must be a subset of the characters in WordCharacters. Often, WordCharacters, BeginCharactes and EndCharacters will all be the same.
Note that the same process applies to the query while searching.
Getting these setting correct will take careful consideration and practice.
It's helpful to create an index of a single test file, and then look at the
words that are placed in the index (see the -v 4
, -D
and -k
searching switches).
Currently there is only support for eight-bit characters.
Example:
WordCharacters .abcdefghijklmnopqrstuvwxyz BeginCharacters abcdefghijklmnopqrstuvwxyz EndCharacters abcdefghijklmnopqrstuvwxyz IgnoreFirstChar . IgnoreLastChar . |
So the string
Please visit http://www.example.com/path/to/file.html. |
will be indexed as the following words:
please visit http www.example.com path to file.html |
Which means that you can search for www.example.com
as a single word, but searching for just example will not find the document.
Note: when indexing HTML documents HTML entities are converted to their
character equivalents before being processed with these directives. This is
a change from previous versions of SWISH-E where you were required to
include the characters 0123456789&#;
to index entities. See also ConvertHTMLEntities
The Buzzwords option allows you to specify words that will be indexed regardless of WordCharacters, BeginCharacters, EndCharacters, stemming, soundex and many of the other checks do on words while indexing.
Buzzwords are case insensitive.
Buzzwords should be separated by spaces and may span multiple directives.
If the special format File:filename
is used then the Buzzwords will be read from an external file during
indexing.
Examples:
Buzzwords C++ TCP/IP |
Buzzwords File:./buzzwords.lst |
If a Buzzword contains search operator characters they must be backslashed when searching. For example:
Buzzwords C++ TCP/IP web=http |
./swish-e -w 'web\=http' |
The IgnoreWords option allows you to specify words to ignore, called stopwords. The default is to not use any stopwords.
Words should be separated by spaces and may span multiple directives. If
the special format File:filename
is used then the stop words will be read from an external file during
indexing.
In previous versions of swish you could use the directive
IgnoreWords swishdefault - obsolete! |
to include a default list of compiled in stopwords. This keywords is no longer supported.
Examples:
IgnoreWords www http a an the of and or |
IgnoreWords File:./stopwords.de |
UseWords defines the words that swish will index. Only the words listed will be indexed.
You can specify a list of words following the directive (you may specify
more than one UseWords directive in a config file), and/or use the File:
form to specify a path to a file containing the words:
UseWords perl python pascal fortran basic cobal php UseWords File: /path/to/my/wordlist |
Please drop the swish-e list a note if you actually use this feature. It may be removed from future versions.
This automatically omits words that appear too often in the files (these words are called stopwords). Specify a whole percentage and a number, such as ``80 256''. This omits words that occur in over 80% of the files and appear in over 256 files. Comment out to turn off auto-stopwording.
IgnoreLimit 50 1000 |
SWISH-E must do extra processing to adjust the entire index when this feature is used. It is recommended that instead of using this feature that you decided what words are stopwords and add them to IngoreWords in your configuration file. To do this, use IgnoreLimit one time and note the stop words that are found while indexing. Add this list to IgnoreWords, and then remove IgnoreLimit from the configuration file.
IgnoreMetaTags defines a list of metantags to ignore while indexing XML files (and HTML files if using libxml2 for parsing HTML). All text within the tags will be ignored -- both for indexing (MetaNames) and properties (PropertyNames). To still parse properties, yet do not index the text, see UndefinedMetaTags.
This option is useful to avoid indexing specific data from a file. For example:
<person> <first_name> William </first_name> <last_name> Shakespeare </last_name> <updated_date> April 25, 1999 </updated_date> </person> |
In the above example you might not want to index the updated date, and therefore prevent finding this record by searching
-w 'person=(April)' |
This is solved by:
IgnoreMetaTags updated_date |
See also UndefinedMetaTags.
This option allows the user decide if to index the contents of HTML comments. Default is no. Set to yes if comment indexing is required.
IndexComments yes |
Note: This is a change in the default behavior prior to version 2.2.
The TranslateCharacters directive maps the characters in string1 to the characters listed in string2.
For example:
# This will index a_b as a-b and ámo as amo TranslateCharacters _á -a |
TranslateCharacters :ascii7:
is a predefined set of characters that will translate eight bit characters
to ascii7 characters. Using the :ascii7: rule will translate ``Ääç'' to
``aac''. This means: searching ``Çelik'', ``çelik'' or ``celik'' will all
match the same word.
TranslateCharacters is done early in the indexing process, after converting HTML entities but before splitting the input text into words based on WordCharacters. So characterters you are translating from do not need to be listed in word characters.
The same character translations take place when searching.
When indexing SWSIH-E assigns a word position to each word. This enables phrase searching. There may be cases where you would like to prevent phrase matching. The BumpPositionCounterCharacters directive allows you to specify a set of characters that when found in the text will increment the word position -- effectively preventing phrase matches across that character.
For example, if you have a tag:
<subjects> computer programming | apple computers </subjects> |
You might want to prevent matching ``programming apple'' in that meta name.
BumpPositionCounterCharacters | |
There is no default, and you may list a string of characters.
Since metatags are typically separate data fields, the word position counter is automatically bumped between metatags (actally, bumpted when a start tag is found and when an end tag is found). This prevents matching a phrase that spans more than one metaname. DontBumpPositionOnEndTags and DontBumpPositionOnStartTags disables this feature for the listed metanames.
For example,
<person> <first_name> William </first_name> <last_name> Shakespeare </last_name> <updated_date> April 25, 1999 </updated_date> </person> |
In the conifuration file:
DontBumpPositionOnEndTags first_name DontBumpPositionOnStartTags last_name |
This configuration allows this phrase search
-w 'person=("william shakespeare")' |
but this phrase search will fail
-w 'person=("shakespeare april")' |
[ TOC ]
Some directives have different uses depending on the source of the documents. These directives are only valid when using the File system method of indexing.
This directive specifies the allowable file suffixes (extensions) while indexing. The default is to index all files specified in IndexDir.
# Only index .html .htm and .q files IndexOnly .html .htm .q |
IndexOnly checks that the file end in the characters listed. It does not check ``extensions''. IndexOnly is tested right before FileRules is processed.
Put ``yes'' to follow symbolic links in indexing, else ``no''. Default is no.
FollowSymLinks no FollowSymLinks yes |
Note that when set to no
extra stat(2)
system calls must be made for each file. For
large number of files you may see a small reduction in indexing time by
setting this to yes
.
See also the -l
switch in SWISH-RUN.
FileRules and FileMatch are used to, respectively, exclude and include files and directories to index. Since, by default, swish indexes all files and recurses all directories you will typically only use FileRules. FileMatch is useful, for example, to override the behavior of IndexOnly. Some examples are included below.
Except for FileRules title ...
, this feature is only available for file access method (-S fs), which is
the default. Also, any pathname modification with ReplaceRules
happens after the check for FileRules. (it's unlikly that you would exclude files with FileRules based on text you added with
ReplaceRules!)
The regular expression is a C regex.h extended regular expression. You may supply more than one regular expression per line, or use separate directives. Preceeding the regular expression with the word ``not'' negates the match.
The regular expression is compared against [type] as described below.
For historical reasons, you can specify contains
or is
. is
simply forces the regular expression to match at the start and end of the
string (by internally prepending ``^'' and appending ``$'' to the regular
expression).
The regex
option requires delimiter characters:
FileRules title regex /^private/i |
The only advantage of regex
is if you want to do case insensitive matches, or simply like your regular
expressions to look like perl regular expressions. You must use matching
delimiters; (), {}, and [], are not currently supported for no good reason.
Use double quotes around a pattern if it contains any white space; otherwise, it will be considered as two or more patterns. For example, these sets generate the same regular expressions.
FileRules title is hello FileRules title contains ^hello$ FileRules title regex /^hello$/ |
FileRules title is "hello there" FileRules title contains "^hello there$" FileRules title regex !^hello there$! |
Matching Types
The following types of match strings my be supplied:
FileRules pathname FileRules dirname FileRules filename FileRules directory FileRules title FileMatch pathname FileMatch filename FileMatch dirname FileMatch directory |
pathname matches the regular expression against the current pathname. The pathname may or may not be absolute depending on what you supplied to IndexDir.
Example:
# Don't index paths that start with private or hidden FileRules pathname contains "^/(private|hidden)" |
# Don't index exe files FileRules pathname contains "\.exe$" |
dirname and filename split the path name by the last delimiter character into a directory name, and a file name. Then these are compared against the patterns supplied.
The delimiter used by swish to split the path is set in config.h, and is platform dependent. Using pathname
will avoid any possible mismatches from what swish thinks your delimter
should be vs. what it actually is. dirname and filename will probably fail miserably on many platforms. Patches are welcome.
Example:
# Same as last example - don't index *.exe files. FileRules filename contains "\.exe$" |
# Don't index any file called test.html files FileRules filename contains "^test\.html$" # Same thing FileRules filename is "test\.html" |
# Don't index any directoires that contain "old" (/usr/local/myold/docs) FileRules dirname contains old |
# Don't index any directories that contain the path segment "old" (/usr/local/old/foo) FileRules dirname contains /old/ |
# Index only .htm, .html, plus any all-digit file names IndexOnly .htm .html FileMatch filename contains "^\d+$" |
# Same as previous, but maybe a little slower FileRules filename regex not !\.(htm|html)$! FileMatch filename contains "^\d+$" |
Swish checks these settings in the order of pathname
, dirname
, and
filename
, and FileMatch patterns are checked before FileRules, in general. This allows you to exclude most files with FileRules, yet allow in a few special cases with FileMatch. For example:
# Exclude all files of .exe, .bin, and .bat FileRules filename contains "\.(exe|bin|bat)$" # But, let these two in FileMatch filename is "baseball\.bat" "incoming_mail.bin" |
# Same, but as a single pattern FileMatch filename is (baseball\.bat|incoming_mail\.bin) |
The directory
type is somewhat unique. When swish recurses into a directory it will
compare all the files in the directory with the pattern and then decide if that entire directory
should or should not be indexed (or recursed). Note that you are matching
against file names in a directory -- and some of those names may be
directory names.
A FileRules directory
match will cause swish to ignore all files and sub-directories within the
directory where the match occured.
Warning: FileMatch directory
says to index everything in the current directory and ignore any FileRules for this directory.
Example:
# Don't index any directories (and sub directories) that contain # a file (or sub-directory) called "index.skip" FileRules directory contains "^index\.skip$" |
# Don't index directories that contain a .htaccess file. FileRules directory contains ^\.htaccess |
Note: While processing directories, swish-e will ignore any files or directories that begin with a
dot (``.''). You may index files or directories that begin with a dot by
specifying their name with IndexDir or -i
.
title
checks for a pattern match in an HTML title.
Example:
FileRules title contains construction example pointers |
# This example says to ignore case FileRules title regex "/^Internal document/i" |
Note: FileRules title
works for any input method (fs, prog, or http) that is parsed as HTML, and
where a title was found in the document.
In case this seems a bit confusing, processing a directory happens in the followin order. Remember, directory names will have a trailing path delimiter.
First the directory name is checked:
FileRules dirname - reject entire directory if matches |
Next the directory is scanned and each file (or sub-directory) is checked: FileRules directory - reject entire dir if any files match FileMatch directory - accept *entire* dir if any files match
Then, unless FileMatch directory
matched, each file is tested with FileMatch. A match says to index the file
without further testing (i.e. overrides FileRules and IndexOnly):
FileMatch pathname \ FileMatch dirname - file is accepted if any match FileMatch filename / |
otherwise
IndexOnly - file is checked for the correct file extension |
FileRules pathname \ FileRules dirname - file is rejected if any match FileRules filename / |
finally, the file is indexed.
Files (not directories) listed with IndexDir or -i
are processed in a similar way: FileMatch pathname \ FileMatch dirname -
file is accepted if any match FileMatch filename /
otherwise, the file is rejected if it doesn't have the correct extension or a FileRules matches.
IndexOnly - file is checked for the correct file extension |
FileRules pathname \ FileRules dirname - file is rejected if any match FileRules filename / |
[ TOC ]
These directives are available when using the HTTP Access Method of indexing.
MaxDepth defines how many links the spider should follow before stopping. A value of 0 configures the spider to traverse all links. The default is MaxDepth 5.
MaxDepth 5 |
The number of seconds to wait between issuing requests to a server. This setting allows for more friendly spidering of remote sites. The default is 60 seconds.
Delay 1 |
The location of a writable temp directory on your system. The HTTP access
method tells the Perl helper to place its files in this location, and the -e
switch causes swish to use this directory while indexing. There is not
default.
TmpDir /tmp/swish/ |
If this directory does not exist or is not writable SWISH-E will fail with an error during indexing.
Note, the environment variables of TMPDIR
, TMP
, and TEMP
(in that order) will override this setting.
The location of the Perl helper script called swishspider. If you use a relative directory, it is relative to your directory when
you run SWISH-E, not to the directory that SWISH-E is in. The default is ./
SpiderDirectory /usr/local/swish/ |
Often times the same site may be referred to by different names. A common example is that often http://www.some-server.com and http://some-server.com are the same. Each line should have a list of all the method/names that should be considered equivalent. Multiple EquivalentServer directives may be used. Each directive defines its own set of equivalent servers.
EquivalentServer http://library.berkeley.edu http://www.lib.berkeley.edu EquivalentServer http://sunsite.berkeley.edu:2000 http://sunsite.berkeley.edu |
[ TOC ]
This section details the directives that are only available for the ``prog'' document source feature of swish. The ``prog'' access method runs an external program that ``feeds'' documents to swish. This allows indexing and filtering of documents from any source.
See prog - general purpose access method in the SWISH-RUN man page for more information.
A number of example programs for use with the ``prog'' access method are provided in the prog-bin directory. Please see those example if you have questions about implementing a ``prog'' input program.
This is a list of parameters that will be sent to the external program when running with the ``prog'' document source method.
SwishProgParameters /path/to/config hello there IndexDir /path/to/program.pl |
Then running:
swish-e -c config -S prog |
swish will execute /path/to/program.pl
and pass
/path/to/config hello there
as three command line arguments to the program. This directive makes it
easy to pass settings from the swish-e configuration file to the external
program.
For example, the spider.pl
program (included in the prog-bin
directory) uses the SwishProgParameters to specify what file to read for configuation information.
SwishProgParameters spider.config IndexDir ./spider.pl |
The spider.pl
program also has a default action so you can avoid using a configuration
file:
SwishProgParameters default http://www.swishe.org/ http://some.other.site/ IndexDir ./spider.pl |
And the spider program will use default settings for spidering those sites.
[ TOC ]
Internally, SWISH-E knows how to parse only text, HTML, and XML documents. With SWISH-E filters you can index other types of documents. For example, if all your web pages are in gzip format a filter can uncompress these on the fly for indexing.
A filter is an external program that swish executes while processing a document of a given type. SWISH-E will execute the filter program for each file that matches the file extension set in the FileFilter directive.
SWISH-E calls the external program passing as default arguments:
the name of the filter program
the physical path name of the file to read. This may be a temporary file location if indexing by the http method.
When indexing under the file system this will be the same as
$1
(the path to the source file), but when indexing under the
http method this will be the URL of the source document.
SWISH-E can also pass other parameters to the filter program. These parameters can be defined using the FileFilter directive. See Filter Options below.
The filter program must open the file, process its contents, and return it to SWISH-E by printing to STDOUT.
Note that this can add a significant amount of time to the indexing process if your external program is a perl or shell script. If you have many files to filter you should consider writing your filter in C instead of a shell or perl script, or using the ``prog'' Access Method.
This is the path to a directory where the filter programs are stored. SWISH-E looks in this directory to find the filter specified in the FileFilter directive. If this directive is omitted, you have to specify the full path to the filterscript on each FileFilter directive.
Example:
FilterDir /usr/local/swish/filters |
This maps file extensions to a filter program. If filter-prog starts with a directory delimiter (absolute path), SWISH-E doesn't use the FilterDir settings, but uses the given filter-prog path directly.
Filter options:
Filter options are a string passed as arguments to the filter-prog. Filter options can contain variables, replaced by SWISH-E. If you ommit filter-options SWISH-E will use default parameters for the options listed above.
Default: "'%p' '%P'" Which means: pass "workfile path" and "documentfile path" to filter (each quoted). |
Variables in filter options:
%% = % %P = Full document pathname (e.g. URL, or path on filesystem) %p = Full pathname to work file (maybe a tmpfile or the real document path on filesystem) %F = Filename stripped from full document pathname %f = Filename stripped from "work" pathname %D = Directoryname stripped from full document pathname %d = Directoryname stripped from full "work" pathname |
Examples of strings passed:
%P = document pathname: http://myserver/path1/mydoc.txt %p = work pathname: /tmp/tmp.1234.mydoc.txt %F = mydoc.txt %f = tmp.1234.mydoc.txt %D = http://myserver/path1 %d = /tmp |
Important hint for security:
When using variable substitution, use quotes to ensure filename integrity.
e.g. "'%f'" --> 'file name with spaces.doc'. |
If you don't use this, your system security may be compromised, or filtering may not work for these files.
Notes when using MS Windows
Windows uses double quotes to escape shell metacharacters, so reverse the example above. e.g.:
'"%f"' --> "file name with spaced.doc" |
The filter program is opened with popen()
which passes the command through the shell. This means that paths to the
filter program must be in DOS format and use the backslash as the path
separator. Swish, like many shells, use the backslash as an escape
character, so you will need to specify the filter program like:
FileFilter .mydoc "c:\\some\\path\\mydocfilter.exe" '-d "%d" -example -url "%P" "%f"' |
Examples of filters:
FileFilter .doc /usr/local/bin/catdoc "-s8859-1 -d8859-1 '%p'" FileFilter .pdf pdftotext "'%p' -" FileFilter .html.gz gzip "-c '%p'" FileFilter .mydoc "/some/path/mydocfilter" "-d '%d' -example -url '%P' '%f'" |
The above examples are running a binary filter program. For more complicated filtering needs you may use a scripting language such as Perl or a shell script. Here's some examples of calling a shell and perl script.
FileFilter .pdf pdf2html.sh FileFilter .ps ghostscript-filter.pl |
Using a scripting language (or any language that has a large startup cost)
can greatly increase the indexing time. For small indexing jobs, this may not be an issue, but for large
collections of files that require processing by a scripting language, you
may be better off using the -S prog
access method where the script will only be compiled once, instead of for
each document.
Filters are probably easier to write than a -S prog
program. Which you decide to use depends on your requirements. Examples of
filter scripts can be found in the filter-bin directory.
[ TOC ]
$Id: SWISH-CONFIG.pod,v 1.43 2001/12/30 17:53:44 whmoseley Exp $
. [ TOC ]