There is no such thing as bug free software. There are always bugs, nasty behaviour etc. This document shall help to solve these problems. The usual symptom will be that a client cannot connect to the server or the server is not responding. Or error messages show up. This section helps to solve such problems.
The first thing to do, especially when confronted with confusing statements from either users or instrument scientists, is to look at the SICS servers log files. The last 1000 lines of the instrument log are accessible from any SICS client or through the WWW interface. The SICS commands:
firstname.lastname@example.orgThis means the log file has been started at August, 8, 2001 at 00:01:01. There is a new log file daily. Load appropriate files into the editor and look what really happened.
Another good ideas is to use the unix command grep on assorted log files. A grep for the strings ERROR or WARNING will more ofteh then not give an indication for the nature of the problem.
The log files show you all commands given and all the responses of the system. Additionally there are hourly time stamps in the file which allow to narrow in when the problem started. Things to watch out for are:
All of SICS can be restarted through the command:
monit restart all
An essential prerequisite of SICS is that the server is up and running. The system is configured to restart the SICServer whenever it fails. Only after a reboot or when the keepalive processes were killed (see below) the SICServer must be restarted. This is done for all instruments by typing:
monitat the command prompt. startsics actually starts two programs: one is the replicator application which is responsible for the automatic copying of data files to the laboratory server. The other is the SICS server. Both programs are started by means of a shell script called keepalive. keepalive is basically an endless loop which calls the program again and again and thus ensures that the program will never stop running.
When the SICS server hangs, or you want to enforce an reinitialization of everything the server process must be killed. This can be accomplished either manually or through a shell script.
All SICS processes can be stopped through the commands:
monit stop all monit quitgiven at the unix command line. You must be the instrument user (for example DMC) on the instrument computer for this to work properly.
If nothing seems to work any more, no connections can be obtained etc, then the next guess is to restart everything. This is especially necessary if mechanics or electronics people were closer to the instrument then 400 meters.
Sometimes it happens that the SICServer hangs while starting up or hardware components are not properly initialized. In such cases it is useful to look at the SICS servers startup messages. On the instrument account issue the commands:
monit stop sicsserver cd inst_sics ./SICServer inst.tcl | moreReplace inst with the name of the appropriate instrument in lower case. For example, from the home directory of the hrpt account on the computer hrpt:
cd monit stop sicsserver cd hrpt_sics ./SICServer hrpt.tcl | moreThis allows to page through SICS startup messages and will help to identify the troublesome component. The proceed to check the component and the connections to it.
The SICS server hanging or crashing should not happen. In order to sort such problems out it is very helpful if any available debugging information is saved and presented to the programmers. Information available are the log files as written continously by the SICS server and posssible core files lying around. They have just this name: core.pid, where pid is the process identification number. In order to save them create a new directory (for example dump2077) and copy the stuff in there. This looks like:
/home/DMC> mkdir dump2077 /home/DMC> cp log/*.log dump2077 /home/DMC> cp core.2077 dump2077The /home/DMC> is just the command prompt. Please note, that core files are only available after crashes of the server. These few commands will help to analyse the cause of the problem and to eventually resolve it.