CLAS12 Simulation Portal to OSG

Submitting jobs to OSG

osg_submit.py drives the full submission pipeline. Run it with:

~/venv/pymysql/bin/python3 osg_submit.py [-b ID] [--devel] [--test] [--print-nodescript] [--print-condor-card]

Steps executed

1. Capacity check Queries HTCondor for the number of running + idle jobs owned by gemc. If the count is at or above --max-submitted-jobs (default 80 000), submission is aborted. Requires the htcondor2 Python package (available on the submit node); use --test to skip.

2. Fetch job from the database Connects to CLAS12OCR (or CLAS12TEST with --devel) and retrieves the first job whose status is not submitted. Use -b ID to target a specific user_submission_id.

3. Parse the submission card The scard field stored in the database is parsed into an SConfiguration object that drives all subsequent generation steps.

3a. Mark job as Processing The run_status column in the database is immediately updated from Not Submitted to Processing, preventing a concurrent osg_submit.py instance from picking up the same job.

3b. Create the job staging directory A per-submission directory is created on the submit node:

~/osgOutput/<username>/job_<user_submission_id>/
    log/              ← HTCondor per-job .err / .out / .log files

4. For type-2 (lund-file) submissions: list and stage lund files

The generator field in the scard must start with /volatile/clas12/ — this is enforced before any file operations begin.

pelican object ls is called on the OSDF mirror of that path to enumerate all lund files (.dat, .txt, .lund). Each OSDF URI is written as one line to ~/osgOutput/<username>/job_<id>/lund_files. HTCondor reads this file with queue lundFile from lund_files and creates one subjob per entry.

In test mode (--test) a three-file mockup is used when pelican is not available.

5. Generate the HTCondor submit file generate_condor_card() assembles a complete .sub file in nine sections and writes it to ~/osgOutput/<username>/job_<id>/clas12.condor:

Section	Content
Header	Universe, container image, site ranking
Retry policy	on_exit_remove / on_exit_hold / periodic_release rules
Requirements	Singularity, CVMFS, kernel version, glidein constraints
Undesired sites	+UNDESIRED_Sites exclusion list
Authentication	OAuth token for Pelican / OSDF transfers
Hardware	CPUs, memory, scratch disk per slot
Executable	nodescript.sh, log/output/error paths, +ProjectName
File transfer	Input staging and output retrieval
Queue	Arguments line per subjob and Queue N (or queue lundFile from lund_files)

6. Generate nodescript.sh and stage all scripts generate_nodescript() assembles the bash script that runs on each OSG worker node and writes it directly to the staging directory. The following files are then staged there:

~/osgOutput/<username>/job_<id>/
    clas12.condor                     ← HTCondor submit file
    nodescript.sh                     ← simulation script (executable on the worker node)
    generators/bash/functions.sh      ← shared bash helpers (path required by transfer_input_files)
    lund_files                        ← one OSDF URI per line (type-2 jobs only)
    log/                              ← HTCondor per-job log files

Every pipeline step in nodescript.sh follows the same pattern: a # input/output comment, an explicit cmd=(...) array, a descriptive echo "Running …: ${cmd[@]}", and run_timed <fn> "${cmd[@]}".

Full pipeline (default):

preamble → environment variables → clean_and_check_environment → setup_job_files
  → Pelican environment setup
  → [fetch_background_file]            # only when bkmerging is set
  → lund_or_generator                  # pelican fetch (type-2) or run_generator cmd array
  → run_gemc                           # gemc.hipo
  → [merge_background]                 # gemc.merged.hipo — only when bkmerging is set
  → [run_denoiser]                     # gemc_denoised.hipo — only for coatjava < 14
  → run_reconstruction                 # recon.hipo
  → test_hipo_file                     # integrity check
  → create_dst                         # $OUTPUT_FILE
  → write_to_jlab                      # upload to OSDF
  → print_timing_summary

nodescript.sh sets OSRELEASE=almalinux9-gcc11 before loading modules so CVMFS modulefiles use the platform directory that contains GEMC, JDK, HIPO, and denoise. For coatjava 14 and newer, the denoiser step is skipped and reconstruction reads gemc.merged.hipo when background merging was requested, otherwise gemc.hipo.

GEMC-only pipeline (output_type=1): steps up to (and including) merge_background run as normal; denoising, reconstruction, and DST are skipped; the gemc output file is renamed to $OUTPUT_FILE and uploaded directly via write_to_jlab.

7. Submit to HTCondor (not yet implemented)

Useful flags

Flag	Effect
--test	Skip HTCondor capacity check; use a pelican mock for lund-file lookups
--devel	Use CLAS12TEST database and the devel container image
-b ID	Process a specific user_submission_id instead of the next pending job
--print-nodescript	Print the generated nodescript.sh to stdout
--print-condor-card	Print the generated HTCondor submit file to stdout
--target-site SITE	Pin all jobs to one GLIDEIN_Site (e.g. CNAF)

Priority systems

The portal uses two independent priority values:

• Pending-submission priority is the sequential submissions.priority value in MySQL. The calculation is in db_io/priority_submissions.py; priority 1 is the next Not Submitted row selected by osg_submit.py. Use --write-to-db to store a newly calculated queue in MySQL. Historical load decays from server_time, falling back to client_time. Interleaved recalculations continue the most recently served user's burst instead of restarting it.
• HTCondor runtime priority is JobPrio for jobs that have already been submitted. It is calculated and applied by condor_io/run_priority_map.py and does not change the pending MySQL queue.

Production applies runtime priorities every ten minutes with:

~/venv/pymysql/bin/python3 condor_io/run_priority_map.py -p -a -m 20

update_simgrid.sh only pulls repository updates; it does not calculate either priority value.

This selects the 20 oldest Condor clusters and assigns priorities from -5 to 5 based on each cluster's running job count relative to the selected-cluster average. The calculation balances clusters, not portal users, because all clusters use the shared Condor owner gemc.

The submissions table displayed by condor_io/list_owner_submission.py shows the MySQL submissions.priority for matched database rows. It does not show the JobPrio applied by run_priority_map.py.

Condor_io