| Line 1 was replaced by line 1 |
| - ! Organization of SINQ data Acquisition |
| + !!! Organization of SINQ Data Acquisition |
| At line 10 added 119 lines. |
| + !! The Instrument Computer |
| + |
| + For each instrument there is an instrument computer. This computer is named like the instrument. For example, for the instrument AMOR the instrument computer is amor.psi.ch. Instrument computers are HP workstations equipped with two disks set up in RAID 1 (mirroring) configuration. For most instruments the instrument computer is located in the instruments cabin. The instrument computers run linux. At the time of writing Scientific Linux 6. |
| + |
| + ! Instrument Computer Accounts |
| + |
| + On each instrument computer there are two important local user accounts: |
| + |
| + # The instrument user account |
| + # The instrument guest account |
| + |
| + The instrument account is used to run the SICS server. Its name is the same as the name of the instrument. The password follows a scheme: instrument name + lns, all in capitals. Thus for amor the credentials for this account are: amor/AMORLNS. Ordinary users are not meant to touch this account. |
| + |
| + The instrument guest account is named inst + lnsg. The password is the last two digits of the year + lns1. Thus, again for the amor example, the credentials are amorlnsg/14lns1. The instrument guest account supposed to be used by SINQ users and holds user data such as batch files and else. Permssions and umasks are set appropriately that the instrument account can read and write the guest account and that the guest account can read data files. |
| + |
| + This is the theory, in practice some instrument do everything through the instrument account. |
| + |
| + This scheme allows us to accommodate external SINQ users without going through the pains of allocating AFS accounts for each user group. So far, security posed no problem. The local accounts on the instrument computer are not easily accessible from outside of PSI. To access those local instrument accounts the following steps are required: |
| + # Logon to balrog.psi.ch with your AFS username |
| + # Logon to any other linux machine, for example, lnsl15.psi.ch, with your AFS username |
| + # From there logon to the local accounts on the instrument computer |
| + |
| + This is so complicated because each instrument has its own instrument network segment. |
| + |
| + ! The Instrument Computer in the Instrument Network |
| + |
| + For each instrument in the SINQ hall there is a separate network segment, each with space for 32 |
| + TCP/IP devices. Each instrument network is separated from the main PSI network through a firewall |
| + maintained by PSI. The purpose of this setup is to separate the instruments from each other and |
| + protect them against malicious access. This setup has a number of consequences: |
| + * Devices in the instrument network are only accessible from the instrument computer |
| + * The instrument computer is the gateway to the instrument. To this purpose, ssh, AFS and the SICS ports have been openened in the firewall. If additional services are required, the corresponding ports must be openened in the firewall by AIT. |
| + * Ping is one of the blocked protocols; thus instrument computers cannot be pinged from the outside. |
| + |
| + There is no DNS in the instrument network. TCP/IP addresses are managed locally by way of the /etc/hosts |
| + file on the instrument computer. Inspect this file for TCP/IP addresses of instrument devices. As yet |
| + another complication, the PSI SL6 standard setup tries to manage the /etc/hosts file automatically. |
| + In order to prevent overwriting of the /etc/hosts file an obscure ext3 file system feature, file attributes, |
| + is used. Thus, when a change to /etc/hosts is required the following procedure must be adhered too: |
| + # Become root |
| + # Remove the immutable attribute from /etc/hosts: chattr -i /etc/hosts |
| + # Edit /etc/hosts to your hearts delight |
| + # Set the immutable attribute again: chattr +i /etc/hosts |
| + |
| + ! The SICS Server in the Instrument Account |
| + |
| + The instrument account contains the necessary files to run the SICS server. The basic command to |
| + start the SICS server is: |
| + |
| + {{{ |
| + SICServer inst.tcl |
| + }}} |
| + |
| + Where inst.tcl is a program in the server side scripting language Tcl which configures the SICS server for |
| + this particular instrument. All instruments use the same SICS server executable. Inst.tcl is named after |
| + the instrument, for example amor.tcl. Inst.tcl will load further Tcl files as required. I will use the |
| + string inst as the place holder for the instrument name in the following discussion. |
| + |
| + The instrument account has to contain the following directories: |
| + |
| + ;inst_sics: The instrument specific Tcl programs and the SICS server executable. For example: amor_sics |
| + ;sicscommon: The Tcl files shared among many instruments |
| + ;log: Log files |
| + ;data: Data files |
| + ;monitfiles: Files for the monit system, see later |
| + |
| + The rest is user generated obfuscation..... |
| + |
| + Some important files in the inst_sics directory: |
| + |
| + ;inst.tcl: the main configuration program for the instrument |
| + ;instcom.tcl: the definition of instrument specific commands |
| + ;nxinst.tcl: the programs for writing data files. |
| + ;inst.dic: The dictionary file which controls the placement of data items in the data file |
| + |
| + The data directory contains the data collected at the instrument. There is a directory for each year |
| + of operation. Such year directories contain: |
| + ;DataNumber: a file to persist the current data file number. Do not touch |
| + ;statusHistory: a directory holding the SICS parameter history files |
| + ;000-0nn: directories containing 1000 data files each. This organisation was choosen because an instrument |
| + may collect more then hundred thousand experimental data files per year. When these are all in one |
| + directory file system operations break down. |
| + |
| + The SICS server and its associated processes are managed by a supervisor program called monit. This is |
| + described in more detail in the section on starting and stopping SICS. The directory monitfiles contains |
| + script belonging to this system. |
| + |
| + |
| + !! Hardware Access |
| + |
| + All hardware is controlled by the SICS server from the instrument computer. Some hardware communicates |
| + directly via TCP/IP. But a great many devices speak only RS232 and are connected to the network through |
| + terminal servers. There is a terminal server per instrument. |
| + |
| + !! Data Replication and Backup |
| + |
| + The experimental data files are the most valuable commodity produced by a SINQ instruments. The following |
| + scheme is in place to keep them safe: |
| + # The SICS server writes the data initially to the local disk. This works even when the network breaks down. |
| + # The SICS server also sends a UDP message to a little java server running on the instrument computer. This |
| + server essentially starts an rsync process which makes sure that the data file is duplicated to the AFS |
| + filesystem. |
| + |
| + The AFS storage area for data files is: /afs/psi.ch/project/sinqdata There are further subdirectories per |
| + year which in turn contains subdirectories per instrument. Which in turn contain the 000-0nn directories |
| + containing the data files, the DataNumber and status history files. The directory for the current year |
| + is covered by a nightly backup by AIT. |
| + |
| + Older data lives on a non backuped area on AFS. But the old data file years are also |
| + archived in the PSI archiving system. |
| + |
| + |
| + For the local instrument accounts the following scheme is in place: |
| + |
| + * The sicscommon and inst_sics directories are under version control using git. |
| + * There is a nightly rsync backup to lnsl15 triggered by a cron job. This protect against disk crashes; |
| + it is no full journaled backup. |
| + |
| + We do not give guarantees that user files on the instrument computers can be recovered at all times. |
LIN