NAME:
	l1a
 PURPOSE:
	Script for processing the SMEI L1A data
 CATEGORY:
	smei/gen/python
 CALLING SEQUENCE:
	l1a.py <site> <rdir> <user> <password>
 INPUTS:
	If a remote server is specified an attempt at downloading by (s)ftp is made:

	<site>		remote server
	<rdir>		directory on remote site with L1A files
	<user>		remote user
	<password>	password
 OPTIONAL INPUT PARAMETERS:
	-source=source	default: /mnt/cdrom
			source directory to look for *.buf or *.buf.gz files.
			For ftp this is also the directory where downloaded
			*.buf.gz are put (write access to the source directory
			is needed)
			If L1A files are retrieved from a local L1A mirror
			(-mirror is set) then this is the directory where the
			L1A files taken from the mirror are copied to.
	-destination=destination
			destination directory for CCD frames:
			default: $SMEIDB
	-badname=badname
			destination directory for L1A files with suspected
			corrupted file names.

	-gunzip		look for gzipped *.buf.gz files and unzip them,
			instead of looking for *.buf files
	-maxftp=maxftp	maximum number of l1a*.buf.gz files downloaded by ftp,
			or copied from local L1A mirror (if -mirror is set)

	-nounpack	only download and unzip L1A files. Don't unpack them.
			The names of successfully downloaded and unzipped files are
			WILL BE ADDED to the list of locally processed L1A files.
	-detach		only used for ftp downloads.
			If set then an L1A file is unpacked immediately after
			download is complete, in parallel with the download of the
			next L1A file. If not set, all L1A files are downloaded
			first, and are then unpacked in a single operation.

	-l1a=l1a	name of specific L1A file to be processed
	-all_l1a	by default only L1A files are processed that are not yet
			listed in file 'file_list' (under the assumption that
			these have already been dealt with). Set this keyword to
			unconditionally process L1A files.
	-file_list=file_list
			fully-qualified name of file with names of already processed
			L1A files. The file must exist already, so if necessary
			create an empty one before running this script.
	-nowrite	suppress writing of CCD frames (useful for testing only)
	-overwrite	force ovewriting of existing CCD frames.
			By default frames are not written to drive if a file for the
			frame already exists. This is currently the primary control
			for picking up either the first or second telemetry dump
			(setting -overwrite speeds up processing by a factor 2).
	-nodups		if set then an attempt is made to remove duplicate frames
			before starting to write individual frames.
	-zipdir		zip CCD frames when leaving subdirectory
	-remove		delete *.buf files after processing
	-move=move	move *.buf files to directory 'move' after processing
	-noclones	skip clone frames (see smei_buf_prep)
	-unblock	skips test for running instances of l1a.py in ftp mode.
	-nic		write .nic files instead of .fts files (OBSOLETE)
	-kblob		buffer size for reading remote L1A files for sftp.
			default is 64 kB. If set to zero the file is downloaded
			with a single read (this is very slow for some reason).
	-sftp		download using sftp instead of ftp.
	-mirror		looks for L1A files in locally maintained mirror of
			remote L1A files (instead of downloading by ftp or sftp)
 OUTPUTS:
	Updated data in <destination>
	Exit code is number of L1A files processed
 CALLS:
	l1a_ftp, l1a_gunzip, l1a_idl, l1a_near, l1a_redo
 SEE ALSO:
	smei_buf
 PROCEDURE:
 >	Only one instance of the script is allowed to run.
 >	The list of L1A files found locally or (when using ftp) on the remote site
	are compared against the list of l1a file names in file 'file_list'
	Only files not on this local list will be downloaded. Override this
	behaviour using keyword all_l1a.
 MODIFICATION HISTORY:
	MAR-2003, Paul Hick (UCSD/CASS)
	APR-2003, Paul Hick (UCSD/CASS)
		Added -overwrite, -nodups, -zipdir, -remove keywords.
	OCT-2003, Paul Hick (UCSD/CASS)
		Added -alive keyword.
	FEB-2004, Paul Hick (UCSD/CASS)
		Added -nounpack keyword.
		Added -noclones keyword to l1a_idl
		Added -move keyword
	MAR-2004, Paul Hick (UCSD/CASS)
		Added keyword 'file_list' instead of hardcoding file name $HOME/l1a_files.txt.
	FEB-2005, Paul Hick (UCSD/CASS)
		Removed -sparseftp option (never used it).
		Added -sftp switch to use sftp instead of plain ftp (requires
		packages paramiko and pycrypto).
		Removed all references to the module string.
	SEP-2005, Paul Hick (UCSD/CASS)
		After installing Paramiko 1.4 on 64-bit AMD (running Fedora Core)
		download would fail for every other file (first downloaded OK,
		second failed, third OK, etc.). Modified code to open a new
		ssh session for each download. Also introduced kblob keyword.
	DEC-2005, Paul Hick (UCSD/CASS)
		Added keyword -mirror to be able to process L1A files from a
		locally-maintained mirror of the L1A data base.
	FEB-2008, Paul Hick (UCSD/CASS)
		Added check for existence of env var $L1A.
		Added check to prevent two copies of l1a.py to run at the
		same time.
	DEC-2009, Paul Hick (UCSD/CASS)
		Exit code now is number of L1A files processed (was always zero)
	APR-2011, Paul Hick (UCSD/CASS; pphick@ucsd.edu)
		Added fix to l1a_mirror to detect L1A files with corrupted
		file names. These are now copied to the directory specified
		in keyword -badname

 NAME:
	l1a_ftp
 PURPOSE:
	Download .buf.gz files
 CALLING SEQUENCE:
	buf_files = l1a_ftp( sys_argv, ldir, redo_list, all_l1a, ddir, unblock )
 INPUTS:
	sys_argv	command line list sys.argv
			contains the info to log into the ftp site
	ldir		local directory where to put the downloaded files
	redo_list	list of l1a files to process
			Only used if len(redo_list) != 0
	all_l1a		if set, process all *.buf files
			if not set, only *.buf files not processed locally yet will be
			downloaded.
 OUTPUTS:
	buf_files	list *.buf.gz to be processed further
			if detach is set then the return value buf_files is an empty list.
 CALLS:
	l1a_idl, l1a_near, l1a_redo
 CALLED BY:
	l1a
 PROCEDURE:
 > The presence of ftp-specific keywords -detach, -sparseft and -maxftp is tested
	for inside this routine.
 MODIFICATION HISTORY:
	APR-2003, Paul Hick (UCSD/CASS)
	FEB-2004, Paul Hick (UCSD/CASS; pphick@ucsd.edu)
		Local L1A file names are now in lower case (l1a* instead of L1A*)

 NAME:
	l1a_gunzip
 PURPOSE:
	Unzip list of *.buf.gz files (usually files downloaded from Hanscom)
 CALLING SEQUENCE:
	buf_files = l1a_gunzip(gz_list, ldir, redo_list)
 INPUTS:
	gz_list		list of *.buf.gz files (only file name, plus extension)
	ldir		directory where *.buf.gz files are located
	redo_list	only entries in gz_list also on redo_list are processed
			(if redo_list is empty all files in gz_list are processed)
 OUTPUTS:
	buf_files	list of successfully unzipped *.buf files
			(fully-qualified file names, located in ldir)
 CALLED BY:
	l1a
 PROCEDURE:
	Loops over all files in far_list, spawning gunzip for each file individually.
 MODIFICATION HISTORY:
	APR-2003, Paul Hick (UCSD/CASS; pphick@ucsd.edu)

 NAME:
	l1a_idl
 PURPOSE:
	Unpack *.buf files writing separate files for each frame
 CALLING SEQUENCE:
	l1a_idl( sys_argv, wait, buf_files, ddir)
 INPUTS:
	buf_files	list of *.buf files or *.buf.gz (fully-qualified names)
			Must have at least one element.
	ddir		destination directory
 OUTPUTS:
	(none)
 CALLED BY:
	l1a, l1a_ftp
 PROCEDURE:
 >	Creates main program for starting up the IDL procedure smei_buf
 >	Writes ascii file with input information for smei_buf,
 >	Then spawns the main IDL program
 >	Both the IDL main program and the ascii file will be deleted by smei_buf
 MODIFICATION HISTORY:
	APR-2003, Paul Hick (UCSD/CASS)
	FEB-2004, Paul Hick (UCSD/CASS)
		Added -noclones keyword
	MAR-2005, Paul Hick (UCSD/CASS; pphick@ucsd.edu)
		Added -nic keyword. By default Fits files are written to disk.

 NAME:
	l1a_near
 PURPOSE:
	Function to update / retrieve list of already processed L1A files
 CALLING SEQUENCE:
	near_list = l1a_near( buf_files )
 INPUTS:
	buf_files		list of file names to be added to the list of already
					processed L1A files stored in file 'file_list'
					(this may be an empty list)
 OUTPUTS:
	near_list		list of processed L1A files currently
 CALLED BY:
	l1a, l1a_ftp
 PROCEDURE:
	The file names (name and type only) of all processed files are stored
	in the ascii file 'file_list
 MODIFICATION HISTORY:
	APR-2003, Paul Hick (UCSD/CASS; pphick@ucsd.edu)

 NAME:
	l1a_redo
 PURPOSE:
	Check list of available files against list of files to be reprocessed.
 CALLING SEQUENCE:
	list = l1a_redo( byftp, buf_files, redo_list, negate )
 INPUTS:
	byftp	if set, then buf_files is a list of *.buf.gz files (obtained from ftp nlist cmd)
		if not set, then buf_files is a list of *.buf files available locally.

	buf_files
		byftp set: list of *.buf.gz files (file names, plus extension only)
		byftp not set: list of *.buf files (fully-qualified file names)

	redo_list
		list of *.buf files (only file name, plus extension)

	negate	negate set: return subset of buf_files NOT present in redo_list
		negate NOT set: return subset of buf_files also present in redo_list
 OUTPUTS:
	list	list of files will be a subset of buf_files.
 CALLED BY:
	l1a, l1a_ftp
 PROCEDURE:
	Compares entries in buf_files against entries in redo_list.
 MODIFICATION HISTORY:
	APR-2003, Paul Hick (UCSD/CASS; pphick@ucsd.edu)

 NAME:
	l1a_summary
 PURPOSE:
 CALLING SEQUENCE:
	status = l1a_summary( wait )
 INPUTS:
 OUTPUTS:
	(none)
 PROCEDURE:
 MODIFICATION HISTORY:
	JUL-2003, Paul Hick (UCSD/CASS; pphick@ucsd.edu)

 NAME:
	list_of_top_level_domains
 PURPOSE:
	Provides a hash with translations of top-level domains to
	full names. These are mostly the standard 2-letter country
	abbreviations to country names, with a few additions.
 CALLING SEQUENCE:
 	tld_name = list_of_top_level_domains()
 OUTPUTS:
	tld_name   dictionary with as key the tld (country abbreviation)
	and as value the full name (country name).
 NOTE:
	There also is a Perl version of this routine
 PROCEDURE:
	Reads the file $ECLIPSE/general/etc/top_level_domains.txt.
	(if not found the function dies).

 The content of the file comes from http://www.thrall.org/domains.htm
 (Dec 2012) with a few manual additions:
	me (Montenegro), sr (Serbia), cat (Catalonia), ps (Palestine)
	eu (European Union)
 MODIFICATION HISTORY:
	DEC-2012, Paul Hick (UCSD/CAIDA; pphick@caida.org)

 NAME:
	login
 PURPOSE:
	Sets up common environment for all user accounts.
 CALLING SEQUENCE:
	This script is probably best added to the global startup
	file /etc/bashrc as:

	. /full_dir/login smei_account smei_dir dat_dir ssw_dir sswdb_dir

	full_dir	full directory path to this script
	smei_account	the user name of the SMEI software account
			(default: soft)
	smei_dir	directory assigned to env var $SMEI
			(default: /home/soft/smei)
	dat_dir		directory assigned to env var $DAT
			(default: undefined)
	ssw_dir		directory assigned to SSW tree
			(default: undefined)
	sswdb_dir	directory assigned to SSWDB data base directory
			(default: undefined)
 OUTPUTS:
	Several other env variables are defined to navigate the SMEI
	software tree: $COM, $EXE, $HTM

	Equivalent env variables are set up to navigate a
	personal software tree: $FOR, $EXE, $PRO, $COM

	Defines IDL_STARTUP=$SMEI/pro/idl_startup.pro

	If the directory $HOME/soft does not exist, it is assumed
	that a new user account is accessed for the first time and
	a few initializations occur:
	- the directory $HOME/soft is created with in it
	  the subdirectories $for, etc.
	- the file ~/LOGFIL.TXT is initialized
	- the startup file $HOME/.bash_profile is modified by
	  adding $EXE and $COM to the path.
 CALLS:
	alias_bash, intel_setup, mkramdisk, newaccount, update_logfile
 PROCEDURE:
	The call to this script should be the only place
	where directories need to be hardcoded.
 MODIFICATION HISTORY:
	FEB-2001, Paul Hick (UCSD/CASS)
	AUG-2002, Paul Hick (UCSD/CASS)
		Introduced command line arguments to set env variables
		instead of hardcoding them for each Linux box separately.
	JAN-2003, Paul Hick (UCSD/CASS)
		Added env var PYTHONPATH
	JUN-2003, Paul Hick (UCSD/CASS)
		Added env vars SMEIDB, SMEIDB1, SMEIDB2, SMEIDB3
	FEB-2004, Paul Hick (UCSD/CASS; pphick@ucsd.edu)
		Added env var for CVS depository
		Added experimental setup for running ssh-agent2
       JUN-2006, Zachary Vaughan (UCSD/CASS; vaughan.zachary@gmail.com
               Added argument for java home directory
	JUN-2006, Java is now set up by sourcing script java_setup.
		Java-related aliases have been added to alias_bash.

 NAME:
	login_csh
 PURPOSE:
	Sets up common environment for all user accounts.
 CALLING SEQUENCE:
	This script is probably best added to the global startup
	file /etc/csh.login as:

	source /full_dir/login_csh smei_account smei_dir dat_dir temp_dir ssw_dir sswdb_dir

	full_dir	full directory path to this script
	smei_account	username of SMEI software account
	smei_dir	directory assigned to env var $SMEI
	dat_dir		directory assigned to env var $dat
	temp_dir	directory assigned to global temp dir
			(will usually be replaced by directory in users
			home directory)
	ssw_dir		directory assigned to SSW tree
	sswdb_dir	directory assigned to SSWDB data base directory
 OUTPUTS:
	Several other env variables are defined to navigate the SMEI
	software tree: $COM, $EXE, $HTM

	Equivalent env variables are set up to navigate a
	personal software tree: $EXE

	Defines IDL_STARTUP=$pro/idl_startup.pro

	If the directory $HOME/soft does not exist, it is assumed
	that a new user account is accessed for the first time and
	a few initializations occur:
	- the directory $HOME/soft is created with in it
	  the subdirectories $myfor, etc.
	- the file ~/LOGFIL.TXT is initialized
	- the startup file $HOME/.bash_profile is modified by
	  adding $EXE and $COM to the path.
 CALLS:
	alias_csh, newaccount, update_logfile
 PROCEDURE:
	The call to this script should be the only place
	where directories need to be hardcoded.
 MODIFICATION HISTORY:
	FEB-2001, Paul Hick (UCSD/CASS)
	AUG-2002, Paul Hick (UCSD/CASS; pphick@ucsd.edu)
		Introduced command line arguments to set env variables
		instead of hardcoding them for each Linux box separately.