;+ ; NAME: ; smei_mkorb ; PURPOSE: ; Calculates approximations for parameters needed to ; create orbital "on-the-fly" patterns ; CATEGORY: ; ucsd/camera/idl/toolbox ; CALLING SEQUENCE: PRO smei_mkorb, ut_frac , $ lastpat = lastpat , $ firstorb= firstorb , $ lastorb = lastorb , $ generate= generate , $ truncpat= truncpat , $ filecmd = filecmd , $ threshold=threshold , $ force = force , $ silent = silent ; OPTIONAL INPUTS: ; lastpat=lastpat ; scalar; type: integer; default: nr cal patterns, minus one. ; defines the pair of "closed-shutter" calibration patterns ; where to start calculating, i.e. the last "lastpat" pairs ; of calibration patterns are processed. ; ; lastpat=0: look at orbits more recent than last cal pattern ; this is used in the real-time data pipeline ; lastpat=1: look at orbits between last pair of cal patterns ; lastpat>1: look further back for last 'lastpat' pairs of cal patterns ; ; firstorb=firstorb ; scalar; type: integer ; overrides the internally determined start orbit ; lastorb=lastorb ; scalar; type: integer ; overrides the internally determined stop orbit ; /generate (only used if lastpat=1, and if run on ; SMEI server smei.ucsd.edu) ; if set then generate the "on-the-fly" orbital ; patterns. ; /filecmd (only used if lastpat=1, and if run on ; SMEI server smei.ucsd.edu) ; append result to file $SMEIDB/cat/list/c3orb ; threshold=threshold ; scalar; type: integer; default: 800 ; minimum on nr of frames required for "on-the-fly" pattern ; (if there are less frames no pattern is created) ; INCLUDE: @compile_opt.pro ; On error, return to caller ; CALLS: ; flt_read, FindAllFiles, GetFileSpec, smei_hdr_get ; timeposn, smei_orbit_get ; TimeSet, TimeUnit, TimeOp, TimeLInterpol, TimeLimits, TimeGet ; RESTRICTIONS: ; Needs access to ; $SMEIDB/cat/list/c3orb ; file with previously used smeidb_orb calls ; (the first 3-years of calls are used to get values for ; min and max orbital fractions) ; $SMEIDB/cat/list/smei_frm_orb_m1.txt ; contains frame counts per orbit ; and directory $SMEIDB/cal ; contains the "closed-shutter" calibration patterns. ; PROCEDURE: ; For camera 3 a sequence of "on-the-fly" patterns needs to be defined ; between each pair of successive "closed-shutter" calibration patterns. ; Each sequence is created by a call to the Fortran program ; href=smei_orb=. The main parameters needed for this program are ; minimum orbit, start orbit, stop orbit, minimum orbital fraction, and ; maximum orbital fraction. ; ; minimum orbit: usually the first full science mode orbit following ; a "closed-shutter" calibration pattern. ; NOTE that this probably will be wrong near anneals, when the ; proper minimum patterns may be the one close to the trailing ; calibration pattern. ; start orbit: first orbit for which an "on-the-fly" patterns is needed ; Usually the same as the minimum orbit) ; stop orbit: last orbit for which an "on-the-fly" pattern is needed. ; Usually the last science mode orbit preceding the next-following ; calibration pattern. ; minimum orbital fraction (between 0 and 1) ; maximum orbital fraction (between 0 and 1) ; restricts the range of CCD frames to be used in the construction ; of "on-the-fly" patterns. ; ; For the first three years of SMEI these parameters were determined ; "by hand". This information is stored in the bash script $SMEIDB/cat/list/c3orb ; in the form of calls to smei_orb. More recent data in this script are based ; on the output of this program. ; ; Minimum, start and stop orbit are set using the frame counts stored ; in $SMEIDB/cat/list/smei_frm_orb_m1.txt. The main constraint is that the ; number of valid frames needs to be above a specified threshold. ; ; The minimum and maximum fraction are calculated by mapping back to ; the first three assuming that the fractions have a periodicity of 1 year. ; ; MODIFICATION HISTORY: ; APR-2007, Paul Hick (UCSD/CASS) ; JAN-2010, Paul Hick (UCSD/CASS) ; Prevent last_orbit return value to be larger than ; orbit number of following cal ; Bugfix. Do not use redirection over ssh to send modified ; c3orb script back to soft@smei. ; Changed definition for min_orbit. It now is one orbit later ; than first_orbit to avoid using a partial orbit as min_orbit. ; JUN-2011, Paul Hick (UCSD/CASS; pphick@ucsd.edu) ; Added check for multiple cal patterns in the same orbit. ; Script will abort when this happens. ; Bugfix: the c3orb script was not updated correctly. ; Updates are now done to file $SMEIDB/cat/list/c3orb_pipeline ; Improved algorithm to calculate start, stop and minimum orbit. ; The old algorithm put the start and minimum orbit very often ; one orbit too high. ;- InitVar, generate , /key InitVar, filecmd , /key InitVar, threshold, 800 InitVar, force , /key InitVar, silent , 0 is_remote = getenv('HOSTNAME') NE 'smei.ucsd.edu' uday = TimeUnit(/day ) uyr = TimeUnit(/year) cal_dir = smei_filepath(mode='cal' ) cat_dir = smei_filepath(mode='cat' ) lst_dir = smei_filepath(mode='list') sts_dir = smei_filepath(mode='sts' ) orb0_dir = smei_filepath(mode='orb') orb3_dir = smei_filepath(mode='orb_003') min0_dir = smei_filepath(mode='min') min3_dir = smei_filepath(mode='min_003') tub_dir = getenv('TUB') ; Read c3orb bash script c3orb = filepath(root=lst_dir,'c3orb_pipeline') IF NOT flt_read(c3orb, atleast=5, d, /silent, comment_char='#') THEN $ RETURN d = reverse(d,2) ; d[0,*] minimum orbit ; d[1,*] first difference orbit ; d[2,*] last difference orbit ; d[3,*] low fraction ; d[4,*] high fraction nr_min_orb = reform(d[0,*]) ut_min_orb = TimeSet(smei=nr_min_orb) ; The line for minimum orbit 16276 was the last one set up by Zach ; The lines up to this one are used to calculate the later ones. loc = (where( nr_min_orb EQ 16276))[0] nr_min_orb = nr_min_orb[0:loc] ; Minimum orbit number ut_min_orb = ut_min_orb[0:loc] ; Start time of minimum orbit ; Time of last minimum orbit set up by Zach ut_last = ut_min_orb[loc] lo_frac = reform(d[3,0:loc]) ; Low orbit fraction hi_frac = reform(d[4,0:loc]) ; High orbit fraction IF IsTime(ut_frac) THEN BEGIN print, 'low fraction=', TimeLInterpol(lo_frac,ut_min_orb,ut_frac) print, 'high fraction=', TimeLInterpol(hi_frac,ut_min_orb,ut_frac) RETURN ENDIF ;plotcurve, ut_min_orb, lo_frac, xstyle=1 ;plotcurve, ut_min_orb, hi_frac, linestyle=3, /oplot ; Read the catalogue containing the number of frames ; per orbit for camera 3 in mode 1. ; Remember that this file is in reverse chronological order f = filepath(root=lst_dir,'smei_frm_orb_m1.txt') IF NOT flt_read(f, atleast=11, c3m1) THEN $ RETURN c3m1_orb = round( reform(c3m1[ 0,*] ) ) ; Orbit number c3m1_cnt = round( reform(c3m1[10,*] ) ) ; Nr of frames in orbit with mode=1 ;c3m1_cnt = round( reform(c3m1[14,*] ) ) ; Pick up a list all "closed shutter" calibration patterns cals = FindAllFiles('c3cal*.fts.gz',path=cal_dir,count=count) IF IsType(truncpat,/string) THEN BEGIN i = (where(cals EQ truncpat))[0] IF i EQ -1 THEN message, 'truncation pattern not found, '+hide_env(truncpat) message, /info, 'truncation pattern, '+hide_env(truncpat) cals = cals[0:i] count = n_elements(cals) ENDIF message, /info, strcompress(count,/rem)+' calibration patterns for camera 3' IF count EQ 0 THEN RETURN InitVar, lastpat, count-1 lastpat <= count-1 cals = GetFileSpec(cals, part='name',/strict) cal_ut = timeposn(cals,/extract) ; Time of calibration patterns cal_orb = TimeGet(/smei, cal_ut) ; Orbit nr of calibration patterns cal_orb = smei_orbit_get(cal_orb,/number); Integer orbit number for cal patterns one_year = TimeSet(/diff,1,uyr) ; lastpat=0: look at orbits more recent than last cal pattern ; (loop goes from count-1 to count-1) ; lastpat=1: look at orbits between last pair of cal patterns ; (loop goes from count-2 to count-2) ; lastpat>1: look further back for last 'lastpat' pairs of cal patterns ; (loop goes from count-2 to count-1-lastpat) message, /info, '# frms/orbit threshold is'+strcompress(threshold) FOR i=count-2+(lastpat EQ 0), count-1-lastpat,-1 DO BEGIN ; Loop over calibration patterns ; Find entries in c3m1 between subsequent calibration patterns ; Since c3m1_orb is in reverse chronological order loc_last < loc_first ; First find the line in c3m1_orb with orbit number cal_orb[i] loc_first = (where(cal_orb[i] EQ c3m1_orb))[0] IF loc_first EQ -1 THEN BEGIN IF silent LE 1 THEN $ message, /info, 'no matching c3m1 orbit for 1st pattern, '+ $ hide_env(cals[i])+', orbit '+strcompress(/rem,cal_orb[i]) continue ; Cannot happen ?? ENDIF ; loc_first now points to the orbit in which the cal was taken. ; If this orbit contains science mode frames taken before the cal ; we need to increase the orbit number by one (to the orbit ; following the cal). IF c3m1_cnt[loc_first] NE 0 THEN BEGIN ; Get times for science mode frames in cal orbit smei_hdr_get, c3m1_orb[loc_first]+[0,1],time=c3m1_time, camera=3, mode=1, silent=2 c3m1_time = TimeLimits(c3m1_time,/mint) IF TimeOp(/subtract,c3m1_time,cal_ut[i],uday) LT 0 THEN BEGIN loc_first -= 1 ; Next higher orbit IF loc_first EQ -1 THEN BEGIN IF silent LE 1 THEN $ message, /info, 'no matching post-cal c3m1 orbit for 1st pattern, '+ $ hide_env(cals[i])+', orbit '+strcompress(/rem,cal_orb[i]) continue ; Cannot happen ?? ENDIF ENDIF ENDIF ; loc_first now points to an orbit that definitely does not contain pre-cal ; science mode frames. Usually it will be the cal orbit itself, or the ; one following. The orbit may or may not contain post-cal science ; mode frames. The first "on-the-fly" orbit can be no lower than this one. IF i EQ count-1 THEN BEGIN loc_last = -1 ; Only happens if lastpat=0: there is no cal_orb[i+1] ENDIF ELSE BEGIN loc_last = (where(cal_orb[i+1] EQ c3m1_orb))[0] IF loc_last NE -1 THEN BEGIN IF c3m1_cnt[loc_last] NE 0 THEN BEGIN ; Get times for science mode frames in cal orbit smei_hdr_get, c3m1_orb[loc_last]+[0,1],time=c3m1_time, camera=3, mode=1, silent=2 c3m1_time = TimeLimits(c3m1_time,/maxt) IF TimeOp(/subtract,c3m1_time,cal_ut[i+1],uday) GT 0 THEN BEGIN loc_last += 1 ; Next lower orbit IF loc_last EQ n_elements(c3m1_cnt) THEN BEGIN IF silent LE 1 THEN $ message, /info, 'no matching pre-cal c3m1 orbit for 2nd pattern, '+ $ hide_env(cals[i+1])+', orbit '+strcompress(/rem,cal_orb[i+1]) continue ENDIF ENDIF ; loc_last now points to an orbit that definitely does not contain ; post-cal science mode frames. Usually it will be the cal orbit ; itself, or the one preceeding. The orbit may or may not ; contain pre-cal science mode frames. ENDIF ENDIF ELSE BEGIN IF silent LE 1 THEN $ message, /info, 'no matching c3m1 orbit for 2nd pattern, '+ $ hide_env(cals[i+1])+', orbit '+strcompress(/rem,cal_orb[i+1]) ENDELSE ENDELSE ;print, cals[i], cal_orb[i] ; If no matching c3m1 orbit was found for 2nd pattern, or there is no ; 2nd pattern (lastpat=1) then cover orbits after loc_first, ; but do not go outside the range covered by c3m1_orb loc_last >= 0 ; This will be triggered if two cal patterns in the same orbit exist ; Probably means a new pattern was created without cleaning up the old one. IF loc_last GE loc_first THEN BEGIN IF silent LE 1 THEN BEGIN IF i LT count-1 THEN BEGIN message, /info, 'unrecoverable error between cal patterns ' + $ cals[i ]+' (orbit '+strcompress(/rem,cal_orb[i ])+') and '+ $ cals[i+1]+' (orbit '+strcompress(/rem,cal_orb[i+1])+')' ENDIF ELSE BEGIN message, /info, 'unrecoverable error after last cal pattern ' + $ cals[i ]+' (orbit '+strcompress(/rem,cal_orb[i ])+')' ENDELSE ENDIF continue ENDIF ; Pick up nr of frames for orbit between subsequent cal patterns ; Pick up a list of orbits with more than 1000 frames enough = where( c3m1_cnt[loc_last:loc_first] GT threshold, n ) IF n EQ 0 THEN continue ; First orbit is first orbit after cal pattern with ; more than 1000 frames ; Last orbit is last orbit preceding subsequent cal pattern ; with more than 1000 frames ; The +1 for last_orbit is required because smei_orb wants ; the time at the end of the orbit for stop time. ; (Remember c3m1_orb is in reverse chronological order) first_orbit = (c3m1_orb[loc_last:loc_first])[enough[n-1]] last_orbit = (c3m1_orb[loc_last:loc_first])[enough[ 0]]+1 ;IF i LT count-1 THEN last_orbit <= cal_orb[i+1] IF silent LE 0 THEN BEGIN IF i LT count-1 THEN BEGIN print, ' first cal pattern: '+cals[i ]+' (orbit '+strcompress(/rem,cal_orb[i ])+')' print, ' second cal pattern: '+cals[i+1]+' (orbit '+strcompress(/rem,cal_orb[i+1])+')' ENDIF ELSE BEGIN print, ' last cal pattern: '+cals[i ]+' (orbit '+strcompress(/rem,cal_orb[i ])+')' ENDELSE print, ' first orbit is '+strcompress(/rem, first_orbit) print, ' last orbit is '+strcompress(/rem, last_orbit) ENDIF ; The minimum orbit is the first complete orbit of science ; mode frames. First find the first orbit, starting at ; loc_first, that contains post-cal science mode frames. ; Then take the orbit after that. The minimum orbit cannot ; be lower than this orbit. loc_min = -1 FOR n=loc_first,loc_last,-1 DO BEGIN IF c3m1_cnt[n] NE 0 THEN BEGIN loc_min = n break ENDIF ENDFOR IF loc_min Eq -1 OR loc_min EQ loc_last THEN BEGIN IF silent LE 0 THEN message, /info, 'unable to determine candidate minimum orbit' continue ENDIF loc_min -= 1 ; The actual minimum orbit is the first orbit on the 'enough' list ; that is >= c3m1_orb[loc_min] n = where( (c3m1_orb[loc_last:loc_first])[enough] GE c3m1_orb[loc_min] ) IF n[0] EQ -1 THEN BEGIN IF silent LE 0 THEN message, /info, 'unable to determine minimum orbit' continue ENDIF min_orbit = min( (c3m1_orb[loc_last:loc_first])[enough[n]] ) ;min_orbit = first_orbit ;min_orbit = (c3m1_orb[loc_last:loc_first])[enough[(n-2) > 0]] IF silent LE 0 THEN $ print, ' minimum orbit is '+strcompress(/rem, min_orbit) IF IsType(firstorb,/defined) THEN BEGIN first_orbit >= firstorb IF silent LE 0 THEN print, ' first orbit is '+strcompress(/rem, first_orbit)+' (requested)' ENDIF IF IsType(lastorb ,/defined) THEN BEGIN last_orbit <= lastorb IF silent LE 0 THEN print, ' last orbit is '+strcompress(/rem, last_orbit)+' (requestd)' ENDIF IF last_orbit LE first_orbit THEN BEGIN IF silent LE 0 THEN $ message, /info, 'orbit range '+strjoin(strcompress([first_orbit,last_orbit],/rem),' - ')+'; nothing left to do' continue ENDIF ; Step back from start time for minimum orbit ; by one-year steps until we are in the range of times ; extracted from the c3orb script. ut = TimeSet(smei=min_orbit) REPEAT ut = TimeOp(/subtract,ut,one_year) $ UNTIL TimeOp(/subtract,ut,ut_last,uday) LT 0 ; Interpolate to get lo_frac and hi_frac lo_frac_prev = TimeLInterpol(lo_frac,ut_min_orb,ut) hi_frac_prev = TimeLInterpol(hi_frac,ut_min_orb,ut) ;FOR n=first_orbit,last_orbit DO BEGIN ; tmp = smei_filename(n,mode='orb',camera=3) ;ENDFOR ;print, cals[i], cal_orb[i], min_orbit, first_orbit, last_orbit, lo_frac_prev,hi_frac_prev cmd = '$orb -min=' +string(min_orbit ,format='(I5.5)') + $ ' -start=' +string(first_orbit ,format='(I5.5)') + $ ' -stop=' +string(last_orbit ,format='(I5.5)') + $ ' $arg' + $ ' -low=' +string(lo_frac_prev,format='(F5.3)') + $ ' -high=' +string(hi_frac_prev,format='(F5.3)') + $ (['',' -force'])[force] print, cmd ; Continue past this point only if a single range of maps is processed IF lastpat GT 1 THEN continue IF NOT generate THEN continue ; Read the file $SSWDB_SMEI/cat/c3orb containing all smeidb_orb ; calls. The first couple of lines define a couple of env vars ; (e.g. $orb) IF NOT txt_read(c3orb,orb) THEN $ break ; Find first (most recent) smeidb_orb line first_line = (where( strpos(orb,'$orb -min=') EQ 0 ))[0] IF first_line EQ -1 THEN $ break ; Create a scratch bash script to be spawned later tmp = orb[0:first_line-1] IF is_remote THEN BEGIN ; The c3orb script sets the destination directory for the "on-the-fly" ; patterns as dest="$SMEIDB/orb" and dest="$SMEIDB/orb_003". ; On smeidb we want to create the files in $TUB/orb and $TUB/orb_003 ; instead, and then scp them to $SMEIDB on smei. ; Modify the destinations here. j = (where(strpos(tmp, 'dest="$SMEIDB/orb"') NE -1))[0] IF j EQ -1 THEN message, 'bad c3orb script: dest="$SMEIDB/orb" not found' tmp[j] = 'dest="'+tub_dir+'/orb"' j = (where(strpos(tmp, 'dest="$SMEIDB/orb_003"') NE -1))[0] IF j EQ -1 THEN message, 'bad c3orb script: dest="$SMEIDB/orb_003" not found' tmp[j] = 'dest="'+tub_dir+'/orb_003"' ENDIF tmp = [tmp, cmd, 'exit'] c3orb_tmp = filepath(root=tub_dir,'c3orb_tmp') openw, /get_lun, iu, c3orb_tmp FOR j=0,n_elements(tmp)-1 DO printf, iu, tmp[j] free_lun, iu ; If lastpat=1 we just set up a smeidb_orb call for the ; the stretch of data between the two most recent ; cal patterns. Save this cmd to $SSWDB_SMEI/c3orb. ; The new line will become the first smeidb_orb cmd in ; this file. IF lastpat EQ 1 AND filecmd THEN BEGIN ; Don't try to redirect c3orb to soft@smei over ssh like this if remote is set ;spawn, 'echo "'+txt+'" | ssh soft@smei "cat - > '+f+'"' ; Shell interpolation will mess up the file. Found this out the hard way. orb = [orb[0:first_line-1],cmd,orb[first_line:*]] f = ([c3orb,filepath(root=tub_dir,'c3orb_dummy')])[is_remote] openw, /get_lun, iu, f FOR j=0,n_elements(orb)-1 DO printf, iu, orb[j] free_lun, iu IF is_remote THEN spawn, 'scp -q '+f+' soft@smei:'+c3orb+'; rm -v '+f ENDIF ; Create the "on-the-fly" orbital patterns by executing the ; scratch bash script created above in $TUB. ; The script creates all files in $SSWDB_SMEI/orb and ; $SSWDB_SMEI/orb_003, including the minimum patterns. ; Move the minimum patterns once done. out = filepath(root=tub_dir,'mkorb_'+GetFileSpec(cals[i],part='name')+'.txt') CASE is_remote OF 0: BEGIN spawn, 'chmod u+x '+c3orb_tmp +'; '+ $ ; Make script executable c3orb_tmp +' > ' +out+'; '+ $ ; Execute script c3orb_tmp+' 3' +' >> '+out ; Execute script for 3-day smoothed dark f = 'c3min*.fts.gz' spawn, 'mv -v '+filepath(root=orb0_dir,f)+' '+min0_dir ; Move *min* files to min directory spawn, 'mv -v '+filepath(root=orb3_dir,f)+' '+min3_dir ; Move *min* files to min directory END 1: BEGIN ; Create the destination directories in $TUB tmp0_dir = filepath(root=tub_dir,'orb' ) tmp3_dir = filepath(root=tub_dir,'orb_003') spawn, 'mkdir -v '+tmp0_dir+'; mkdir -v '+tmp3_dir IF NOT CheckDir( tmp0_dir ) OR NOT CheckDir( tmp3_dir ) THEN $ message, 'failed to create tmp directories' spawn, 'chmod u+x '+c3orb_tmp +'; '+ $ ; Make script executable c3orb_tmp +' > ' +out+'; '+ $ ; Execute script c3orb_tmp+' 3' +' >> '+out ; Execute script for 3-day smoothed dark ; Now scp the patterns to the appropriate directories in $SMEIDB f = filepath(root=tmp0_dir, 'c3orb*.fts.gz') if (file_search(f))[0] NE '' THEN BEGIN spawn, 'scp -q '+f+' soft@smei:'+orb0_dir+'; rm '+f f = filepath(root=tmp0_dir, 'c3min*.fts.gz') spawn, 'scp -q '+f+' soft@smei:'+min0_dir+'; rm '+f ENDIF f = filepath(root=tmp3_dir, 'c3orb*.fts.gz') IF (file_search(f))[0] NE '' THEN BEGIN spawn, 'scp -q '+f+' soft@smei:'+orb3_dir+'; rm '+f f = filepath(root=tmp3_dir, 'c3min*.fts.gz') spawn, 'scp -q '+f+' soft@smei:'+min3_dir+'; rm '+f ENDIF ; The temp directories should now be empty; delete them spawn, 'rmdir -v '+tmp0_dir+'; rmdir -v '+tmp3_dir END ENDCASE tmp = do_file(/delete,c3orb_tmp) ; Delete script ENDFOR RETURN & END