SICSOrg

Organization of SINQ Data Acquisition

Client Server Data Acquisition

SINQ uses SICS as its data acquisition program. SICS is a client server system. Thus at any given time at least two programs are required to operate SICS:

1. A SICS client. SICS clients can run on any computer
2. A SICS server. The SICS server runs on the instrument computer.

The two programs communicate with each other through the TCP/IP network and a simple ASCII command protocol.

The Instrument Computer

For each instrument there is an instrument computer. This computer is named like the instrument. For example, for the instrument AMOR the instrument computer is amor.psi.ch. Instrument computers are HP workstations equipped with two disks set up in RAID 1 (mirroring) configuration. For most instruments the instrument computer is located in the instruments cabin. The instrument computers run linux. At the time of writing Scientific Linux 6.

Instrument Computer Accounts

On each instrument computer there are two important local user accounts:

1. The instrument user account
2. The instrument guest account

The instrument account is used to run the SICS server. Its name is the same as the name of the instrument. The password follows a scheme: instrument name + lns, all in capitals. Thus for amor the credentials for this account are: amor/AMORLNS. Ordinary users are not meant to touch this account.

The instrument guest account is named inst + lnsg. The password is the last two digits of the year + lns1. Thus, again for the amor example, the credentials are amorlnsg/14lns1. The instrument guest account supposed to be used by SINQ users and holds user data such as batch files and else. Permssions and umasks are set appropriately that the instrument account can read and write the guest account and that the guest account can read data files.

This is the theory, in practice some instrument do everything through the instrument account.

This scheme allows us to accommodate external SINQ users without going through the pains of allocating AFS accounts for each user group. So far, security posed no problem. The local accounts on the instrument computer are not easily accessible from outside of PSI. To access those local instrument accounts the following steps are required:

1. Logon to balrog.psi.ch with your AFS username
2. Logon to any other linux machine, for example, lnsl15.psi.ch, with your AFS username
3. From there logon to the local accounts on the instrument computer

This is so complicated because each instrument has its own instrument network segment.

The Instrument Computer in the Instrument Network

For each instrument in the SINQ hall there is a separate network segment, each with space for 32 TCP/IP devices. Each instrument network is separated from the main PSI network through a firewall maintained by PSI. The purpose of this setup is to separate the instruments from each other and protect them against malicious access. This setup has a number of consequences:

• Devices in the instrument network are only accessible from the instrument computer
• The instrument computer is the gateway to the instrument. To this purpose, ssh, AFS and the SICS ports have been openened in the firewall. If additional services are required, the corresponding ports must be openened in the firewall by AIT.
• Ping is one of the blocked protocols; thus instrument computers cannot be pinged from the outside.

There is no DNS in the instrument network. TCP/IP addresses are managed locally by way of the /etc/hosts file on the instrument computer. Inspect this file for TCP/IP addresses of instrument devices. As yet another complication, the PSI SL6 standard setup tries to manage the /etc/hosts file automatically. In order to prevent overwriting of the /etc/hosts file an obscure ext3 file system feature, file attributes, is used. Thus, when a change to /etc/hosts is required the following procedure must be adhered too:

1. Become root
2. Remove the immutable attribute from /etc/hosts: chattr -i /etc/hosts
3. Edit /etc/hosts to your hearts delight
4. Set the immutable attribute again: chattr +i /etc/hosts

The SICS Server in the Instrument Account

The instrument account contains the necessary files to run the SICS server. The basic command to start the SICS server is:

	SICServer inst.tcl

Where inst.tcl is a program in the server side scripting language Tcl which configures the SICS server for this particular instrument. All instruments use the same SICS server executable. Inst.tcl is named after the instrument, for example amor.tcl. Inst.tcl will load further Tcl files as required. I will use the string inst as the place holder for the instrument name in the following discussion.

The instrument account has to contain the following directories:

inst_sics

The instrument specific Tcl programs and the SICS server executable. For example: amor_sics

sicscommon

The Tcl files shared among many instruments

log

Log files

data

Data files

monitfiles

Files for the monit system, see later

The rest is user generated obfuscation.....

Some important files in the inst_sics directory:

inst.tcl

the main configuration program for the instrument

instcom.tcl

the definition of instrument specific commands

nxinst.tcl

the programs for writing data files.

inst.dic

The dictionary file which controls the placement of data items in the data file

The data directory contains the data collected at the instrument. There is a directory for each year of operation. Such year directories contain:

DataNumber

a file to persist the current data file number. Do not touch

statusHistory

a directory holding the SICS parameter history files

000-0nn

directories containing 1000 data files each. This organisation was choosen because an instrument

Hardware Access

All hardware is controlled by the SICS server from the instrument computer. Some hardware communicates directly via TCP/IP. But a great many devices speak only RS232 and are connected to the network through terminal servers. There is a terminal server per instrument.

Data Replication and Backup

The experimental data files are the most valuable commodity produced by a SINQ instruments. The following scheme is in place to keep them safe:

1. The SICS server writes the data initially to the local disk. This works even when the network breaks down.
2. The SICS server also sends a UDP message to a little java server running on the instrument computer. This server essentially starts an rsync process which makes sure that the data file is duplicated to the AFS filesystem.

The AFS storage area for data files is: /afs/psi.ch/project/sinqdata There are further subdirectories per year which in turn contains subdirectories per instrument. Which in turn contain the 000-0nn directories containing the data files, the DataNumber and status history files. The directory for the current year is covered by a nightly backup by AIT.

Older data lives on a non backuped area on AFS. But the old data file years are also archived in the PSI archiving system.

For the local instrument accounts the following scheme is in place:

• The sicscommon and inst_sics directories are under version control using git.
• There is a nightly rsync backup to lnsl15 triggered by a cron job. This protect against disk crashes; it is no full journaled backup.

We do not give guarantees that user files on the instrument computers can be recovered at all times.