Table of Contents:
SWISH-E 2.0 is written in C, and, up to this time, it has been tested on Solaris 2.6, AIX 4.3.2, OpenVMS 7.2-1 AXP, RedHat Linux 6.2 (and other Linux distributions) and Win32 platforms.
Unless you are using the included Win32 binary, a C compiler is needed. Pretty much any standard compiler should do, although you will probably have best luck with a current version of gcc. If you are using something else (such as HP-UX or AIX) you may see more warnings during the build process. Any problems should be sent to the swish-e discussion list after searching the list archives.
libxml2
Swish-e 2.2 can (and probably should) use the libxml2 library for parsing HTML and XML files. This library can be found at http://www.xmlsoft.org/. Instructions for installing and enabling the library are described below.
Currently, the libxml2 library is not required, but is a much better parser than the tired old swish-e html parser (html.c). Please see the Swish-e FAQ SWISH-FAQ for more discussion of the use of libxml2.
Swish's old xml.c paser has been rewritten to use James Clark's Expat library (included with the swish-e distribution), but Swish's old html.c code is still broken in a number of ways. Libxml2 is comparable to Expat, but offers a much better HTML parser that swish-e's html.c parser. Use libxml2 if possible for parsing HTML and XML.
Currently, setting a content type (IndexContents or DefaultContents) of HTML uses swish's html.c parser, where HTML2 uses libxml2's HTML parser. Likewise, a setting of XML uses the included Expat library, where XML2 uses libxml2 for parsing XML. All this may change in future releases.
zlib compression
Swish can make use of zlib to compress document properties. This is recommended if you are using StoreDescription
Memory
Swish needs quite a bit of memory while indexing. How much depends on what
you are indexing. The index is portable between platforms, so you can index
on a machine that has lots of memory available and move the index
file(s)
to another machine for searching. Use the -e
switch if you are short on memory.
Perl modules
The HTTP document source method uses a Perl helper script that requires the LWP, HTTP, and HTML modules. (Note: depending on your perl installation, you might need to install additional modules required by LWP; for requirements and downloads check http://www.cpan.org or http://search.cpan.org). The Perl helper script was tested with perl 5.005 although should probably work with any version 5 release. Do note that the LWP, HTTP, and HTML modules are updated often for bug fixes and such -- do check for upgrades.
[ TOC ]
A configure
script is used to determine platform specific details for building swish.
Please contact the SWISH-E discussion list if you notice any platform
specific problems while building SWISH-E.
Specific information for various platforms can be found in subdirectories
of the src
directory. For example, the Win32 binary for Windows can be found in src/win32
, and instructions for building under VMS can be found in src/vms
.
[ TOC ]
[ TOC ]
Daniel Veillard's libxml2 is a well supported library for working with HTML and XML documents. Swish-e can now use libxml2 to parse HTML and XML documents.
The libxml2 source code can be downloaded from http://www.xmlsoft.org.
Installing the libxml2 library is not required at this time, but is recommended, espeically if you are parsing HTML. As mentioned above, the XML parser that is included with swish uses James Clarks's Expat library and works well. The HTML parser in swish has been in use for years, but the parser provided by libxml2 is preferred. The libxml2 HTML parser offers more features (and more features for parsing XML), and is more accurate. If you are running Linux it may already be installed (look for libxml2.so.2.4.5 or higher).
The library can be downloaded from http://www.xmlsoft.org/. Installation directions are included in the INSTALL file in the libxml2 package. Uncompressing, building, and installation of libxml2 is very similar to the way Swish-e is built.
If libxml2 complains during compilation that it can not find zlib then you may need to specify the location of libz. This happens (on Solaris) when the ./configure script finds the zlib header files, but the compiler and linker do not know to look in /usr/local/lib for the library. You may see an error like:
ld: fatal: library -lz: not found ld: fatal: File processing errors. No output written to .libs/libxml2.so.2.4.5 *** Error code 1 |
In this case, try specifying where zlib can be found. For example, if libz was located in /usr/local/lib you would use this when building libxml2:
./configure --with-zlib=/usr/local |
Swish-e doesn't use libxml2 uncompression features, so you *should* be able to disable zlib.
./configure --without-zlib |
NOTE: But, that doesn't seem to work at this time (as of version libxml2-2.4.5).
If you do not have root access you can specify a prefix when building libxml2:
./configure --prefix=$HOME/local |
This will install the headers and library files in $HOME/local/include and $HOME/local/lib. You will need to inform the swish-e build process of this non-standard directory location (explained below).
Once you run the configure script you build and install the library as the libxml2 INSTALL page instructs:
make make install |
Building Swish with libxml2
To enable swish to use libxml2 you must run swish-e's configure script as:
./configure --with-libxml2 |
If libxml2 was installed in a non-standard location then specify the path where libxml2 was installed. For example,
./configure --with-libxml2=$HOME/local |
If libxml2 is installed in a non-standard location, swish needs to know
where that library is at run time. There seems to be a number of ways to do
this. First, you can set the environment variable LD_RUN_PATH
*before* running make to create swish. This will add the path directly to
the swish-e executable file.
For example, under Bourne type shells:
LD_RUN_PATH=$HOME/local/lib make |
Other shells may require
setenv LD_RUN_PATH $HOME/local/lib make |
Another option is to use the LD_LIBRARY_PATH
environment variable. This is a list of directories to search for libraries
when a program is run. See the ld(8)
man page for more info
Note that libxml2 will be linked as a shared library, so once you compile Swish-e to use the library, the libxml2 library must not be deleted or moved.
[ TOC ]
Building with zlib is similar to the instructions for building swish-e with libxml2 above. To select support for zlib run configure with:
./configure --with-zlib |
zlib is common on many systems, but may be out of date. You should run at least version 1.1.3. To link with zlib in a non-standard location use, for example:
./configure --with-zlib=$HOME/zlib |
Again, you may need
[ TOC ]
If you are reading this INSTALL document, then you probably already have downloaded and unpacked the distribution. But just in case...
Make sure you are using the current stable release from http://swish-e.org. How you download is up to you: lynx, lwp-download, wget are all common methods.
gzip -dc swish-e.x.x.tar.gz | tar xof - |
or on some versions of tar, simply
tar zxof swish-e.x.x.tar.gz |
Uncompressing should create the following directories:
swish-e-x.x/ configure script and top-level Makefile swish-e-x.x/pod/ swish-e documentation swish-e-x.x/html/ HTML version of the documentation swish-e-x.x/src/ source code swish-e-x.x/conf/ example configuration files and stopword files swish-e-x.x/example/ working example CGI scripts swish-e-x.x/filter-bin/ filter samples swish-e-x.x/prog-bin/ -S prog a web spider and other examples swish-e-x.x/perl/ perl interface to the SWISH-E C library swish-e-x.x/src/expat/ James Clark's Expat XML parser swish-e-x.x/src/win32/ win32 binary and buid files swish-e-x.x/src/vms/ files required for building under VMS swish-e-x.x/tests/ tests used for running "make test" swish-e-x.x/doc/ directory used or building the documentation |
Compile-time configuration settings are adjusted in the file src/config.h. Most of the settings may also be specified in the configuration file used during indexing. You probably will not need to change this file, but it's helpful to become familiar with the default compiled-in settings.
Building swish-e on most systems is a simple procedure. In the swish-e-x.x/ top level directory type the following commands
./configure make make test |
Note: If you wish to use libxml2 or zlib please see the previous section.
The above will create the swish-e executable src/swish-e and test that the executable is working correctly. make test
will generate an index file in the tests directory and run a number of searches against this index.
You may optionally ``build'' the swish-search executable. This is a version of swish-e that cannot write to the index file. This version may provide somewhat improved security in a CGI environment. The binaries swish-e and swish-search are the same files -- the additional security is enabled when the binary is named swish-search. swish-search is not a substitute for good file system and CGI security.
Again, this is an optional step:
make swish-search |
which simply copies the file swish-e to swish-search.
Move the swish-e (and/or swish-search) executable to its final location (normally /usr/local/bin). You may
simply copy the program anywhere you see fit, or you may use the make install
command to install it to the location defined by the configure script:
You may need to superuser privileges:
su root make install exit |
IMPORTANT: Do not *run* swish as the superuser (root).
The bin directory may be set when first running ./configure. For example:
./configure --bindir=$HOME/bin |
sets the installation directory to $HOME/bin and make install
will install the program in that location.
[ TOC ]
The SWISH-E discussion list is the place to ask questions about installing and using swish-e, see or post bug fix or security announcements, and a place where you can offer help to others.
The list is typically very low traffic, so it won't overload your inbox. Please take time to subscribe. See http://swish-e.org.
If you are using SWISH-E on a public site, please let the list know so it can be added to the list of sites that use SWISH-E!
[ TOC ]
Swish 2.2 creates the C library libswish-e.a during the build. Install this library if you wish to embed SWISH-E into another application. For example, the library should be installed before using the high level Perl SWISH modules located on CPAN. http://search.cpan.org/search?mode=module&query=SWISH
This is an optional step.
To install the library issue the following commands (again, you may need to su root)
su root make install-lib exit |
By default this will install the library in /usr/local/lib, but this directory can be set when running ./configure with the --libdir option. For example:
./configure --bindir=$HOME/bin --libdir=$HOME/lib |
So make install
will install the swish-e binary in $HOME/bin and make install-lib
will install the libswish-e.a library in $HOME/lib.
Note: You may wish to run make realclean
before running ./configure again.
[ TOC ]
The SWISH-E documentation in HTML format was created with Pod::HtmlPsPdf, a package of Perl modules written and/modified by Stas Bekman to automate the conversion of documents in pod format (see perldoc perlpod) to HTML, Postscript, and PDF. A slightly modified version of this package is include with the SWISH-E distribution and used for building the HTML.
If your system has the necessary tools to build Postscript and the converter ps2pdf installed, you may be able to build the Postscript and PDF versions of the documentation. After you have run ./configure, type from the top-level directory of the distribution:
make pdf |
And with any luck you will end up with the these two files in the top-level directory:
swish-e_documentation.pdf swish-e_documentation.ps |
[ TOC ]
Part of the included SWISH-E documentation can be installed as system
man(1)
pages. Only the reference related pages are installed
(it's assumed that you don't need to install the README or INSTALL
documents as man pages). You must have the pod2man program installed on
your system (which you probably do if you have Perl).
To build the man pages and install them into your system, type from the top-level directory (after running ./configure):
su root make install-man exit |
You will need to su root
if you do not have write access to the man directory. The man pages are
installed in the system man directory. This directory is determined by
running ./configure and can be set by passing the directory when running
./configure.
For example,
./configure --mandir=/usr/local/doc/man |
Information on running ./configure can be found by typing:
./configure --help |
The pod source files used to create the man files were written running under perl 5.6.0. Older version of Perl may complain slightly about the formatting of the pod files. This shouldn't be a problem, but please let the swish-e list know if otherwise.
[ TOC ]
Please search the SWISH-E list archive before posting a question, and check the SWISH-FAQ to see if your question hasn't already been asked.
Support for installation, configuration and usage is available via the SWISH-E discussion list. Visit http://swish-e.org for information. Do not contact developers directly for help -- always post your question to the list.
Before posting use tools available to narrow down the problem.
SWISH-E has the -T, -v, and -k switches that may help resolve issues. If
possible find a single document that shows the problem, then index with -T
INDEXED_WORDS and watch the exact words that are indexed. Use -H 9 when
searching and look at Parsed Words:
to make sure you are searching the correct words.
You can also use programs like gdb
to help find segfaults and other run-time errors, and programs like truss
or strace
can often provide interesting information, if you are adventurous.
[ TOC ]
The exact version of SWISH-E that you are using. Running swish-e with the -V
switch will print the version number. Also, supply the output from uname -a
or similar command that identifies the operating system you are running on.
If you are running an old version of swish be prepared for a response to
your question of ``upgrade.''
A summary of the problem. This should include the commands issued (e.g. for indexing or searching) and their output, and why you don't think it's working correctly. Please cut-n-paste the exact commands and their output instead of retyping to avoid errors.
Include a copy of the configuration file you are using, if any. Swish has reasonable defaults so in many cases you can run swish without using a configuration file. But, if you need to use a configuration file, reduce it down to the absolute minimum number of commands required to demonstrate your problem. Again, cut-n-paste.
A small copy of a source document that demonstrates the problem.
If you are having problems spidering a web server, use lwp-download or wget to copy the file locally to make sure you can index the document using the file system method.
If you do need help with spidering, don't post fake URLs, as it makes it impossible to help. If you don't want to expose your web page to the people on the swish-e list, find some other site to test spidering on. If that works, but you still cannot spider your own site then post your real URL if you want help.
If you are having trouble building SWISH-E please cut-n-paste the output from make (or from ./configure if that's where the problem is).
[ TOC ]
This section should give you a basic overview of indexing and searching with SWISH-E. Other examples can be found in the conf directory, and you should review the SWISH-FAQ.
SWISH-E reads a configuration file (see SWISH-CONFIG) for directives that control what and how SWISH-E indexes files. Then running SWISH-E is controlled by command line arguments (see SWISH-RUN). Swish does not require a configuration file, but most people need to change the default behavior of swish by settings in a configuration file.
To try the examples below change to the tests subdirectory of the distribution. The tests will use the *.html files in this directory when creating the test index. You may wish to review these *.html files to get an idea of the various native file formats that SWISH-E supports.
[ TOC ]
The configuration file controls what and how SWISH-E indexes. The configuration file consists of directives, comments, and blank lines. The configuration file can be any name you like.
This example will work with the documents in the tests directory. You may wish to review the tests/test.config configuration file used for the make test
tests.
For example, a simple configuration file (swish-e.conf):
# Example SWISH-E Configuration file |
# Define *what* to index # IndexDir can point to a directories and/or a files |
# Here it's pointing to the current directory IndexDir . |
# But only index the .html files IndexOnly .html |
# Show basic info while indexing IndexReport 1 |
And that's a simple configuration file. It says to index all the .html files in the current directory, and provide some basic output while indexing.
The complete list of all configuration file directives are described in SWISH-CONFIG.
[ TOC ]
Now, make sure you are in the tests directory and save the above example configuration file as swish-e.conf. Then run swish using the -c
switch to specify the name of the configuration file.
../src/swish-e -c swish-e.conf |
Indexing Data Source: "File-System" Indexing "." Removing very common words... no words removed. Writing main index... Sorting words ... Sorting 55 words alphabetically Writing header ... Writing index entries ... Writing word text: Complete Writing word hash: Complete Writing word data: Complete 55 unique words indexed. Writing file list ... Property Sorting complete. Writing sorted index ... 5 files indexed. 1252 total bytes. Elapsed time: 00:00:00 CPU time: 00:00:00 Indexing done! |
This created the index file index.swish-e. This is the default index file name unless the IndexFile directive is specified in the configuration file:
IndexFile ./website.index |
[ TOC ]
You specify your search terms with the -w
switch. For example, to find the files that contain the word sample you would issue the command:
../src/swish-e -w sample |
This example assumes that you are in the tests directory, and the swish-e binary is in the ../src directory. Swish returns in response to that command the following:
../src/swish-e -w sample |
# SWISH format: 2.2 # Search words: sample # Number of hits: 2 # Search time: 0.000 seconds # Run time: 0.005 seconds 1000 ./test_xml.html "If you are seeing this, the METATAG XML search was successful!" 159 1000 ./test.html "If you are seeing this, the test was successful!" 437 . |
So the word sample was found in two documents. The first number shown is the relevance or rank of the search term, followed by the file containing the search term, the title of the document, and finally the length of the document.
The period (``.'') alone at the end marks the end of results.
Much more information may be retrieved while searching by using the -x
and -H
switches (see SWISH-RUN) and by using Document Properties (see SWISH-CONFIG).
[ TOC ]
To search for a phrase in a document use double-quotes to delimit your search terms. (The phrase delimiter is set in src/swish.h.)
You must protect the quotes from the shell.
For example, under Unix:
swish-e -w '"this is a pharase" or (this and that)' swish-e -w 'meta1=("this is a pharase") or (this and that)' |
Or under Windows command.com shell.
swish-e -w \"this is a pharase\" or (this and that) |
The phrase delimiter can be set with the -P
switch.
[ TOC ]
You can use the Boolean operators and, or, or not in searching. Without these Boolean, SWISH-E will assume you're anding the words together.
Here are some examples:
../src/swish-e -w 'apples oranges' ../src/swish-e -w 'apples and oranges' ( Same thing ) |
../src/swish-e -w 'apples or oranges' |
../src/swish-e -w 'apples or oranges not juice' -f myIndex |
retrieves first the files that contain both the words ``apples'' and ``oranges''; then among those the ones that do not contain the word ``juice''
A few others to ponder:
../src/swish-e -w 'apples and oranges or pears' ../src/swish-e -w '(apples and oranges) or pears' ( Same thing ) ../src/swish-e -w 'apples and (oranges or pears)' ( Not the same thing ) |
See SWISH-SEARCH for more information.
[ TOC ]
The -t
option in the search command line allows you to search for words that exist
only in specific HTML tags. Each character in the string you specify in the
argument to this option represents a different tag in which the word is
searched; that is you can use any combinations of the following characters:
H means all <HEAD> tags B stands for <BODY> tags t is all <TITLE> tags h is <H1> to <H6> (header) tags e is emphasized tags (this may be <B>, <I>, <EM>, or <STRONG>) c is HTML comment tags (<!-- ... -->) |
For example:
# Find only documents with the word "linux" in the E<lg>TITLEE<gt> tags. ./swish-e -w linux -t t |
# Find the word "apple" in titles or comments ./swish-e -w apple -t tc |
[ TOC ]
For the last example we will instruct swish to use META tags to define fields in your documents.
META names are a way to define ``fields'' in your documents. You can use
the META names in your queries to limit the search to just the words
contained in that META name of your document. For example, you might have a
META tagged field in your documents called subjects
and then you can search your documents for the word ``foo'' but only return
documents where ``foo'' is within the subjects
META tag.
Document Properties are somewhat related to meta tags: Properties allow the contents of a META tag in a source document to be stored within the index, and that text to be returned along with search results.
META tags can have two formats in your documents.
<META NAME="keyName" CONTENT="some Content"> |
And in XML format
<keyName> Some Content </keyName> |
If using libxml, you can optionally use a non-html tag as a metaname:
<html> <body> Hello swish users! <keyName> this is meta data </keyName>. </body> |
This, of course, is invalid HTML.
To continue with our sample swish-e.conf file, add the following lines:
# Define META tags MetaNames meta1 meta2 meta3 |
Reindex to include the changes:
../src/swish-e -c swish-e.conf |
Now search, but this time limit your search to META tag ``meta1'':
../src/swish-e -w 'meta1=metatest1' |
Again, please see SWISH-RUN and SWISH-CONFIG for complete documentation of the various indexing and searching options.
[ TOC ]
The above example indexes local files using the file system access method -S fs
. You may also index files that are located on a local or remote web server
by using the HTTP access method -S http
, or via the prog input method -S prog
. These are described in SWISH-RUN
and example configuration files for using these methods can be found in the conf directory of the swish-e distribution.
The -S prog
access method can be used to index any type of document, such as documents
stored in a database (RDBMS), or documents that need to be processed before
they can be indexed. Examples for using the
-S prog
method are shown in the prog-bin directory>.
Swish can also use filters to convert documents as they are processed by swish. For example, MS-Word
or PDF documents can be converted and indexed by swish-e by using filters.
See the section on filters in SWISH-CONFIG, and the examples shown in the filter-bin
directory.
[ TOC ]
$Id: INSTALL.pod,v 1.14 2001/12/26 18:33:52 whmoseley Exp $
. [ TOC ]