RTN-064: JOR review recommendations

  • William O'Mullane

Latest Revision: 2023-07-26

JOR recommendations

Review recommendations extracted from Jira.

Download jor.csv here.

2023-12-01 14:28

Issue key

Rec#

Summary

Report date

Due Date

Implementation Status

Description

Response

Implementation Status Description

PREOPS-3542

M-III.6-25

Hold a risk register scrub/workshop

JOR_2023_02

2023-09-01

Implemented

Hold a dedicated risk register scrub/workshop with subsystem leads and risk owners as well experts in risk management, in addition to the upcoming cost scrub to ensure focus on risk and complete the risk register. This activity should engage all levels of management that will be involved in operations [by September 2023].

This was completed in May 2023. The scrub is done and the RDO is updating budgets and Risks.

A pre-scrub risk workshop was held in April 2023, and served to homogenize and finalize the cost and schedule impact analysis of the risk register, as well as surface some new candidate risks. This workshop was attended by the Risk Board, which includes several internal experts in risk management.

PREOPS-3541

M-III.6-24

Develop, document, and socialize detailed decision-making processes

JOR_2023_02

2024-06-01

In Progress

Develop, document, and socialize the detailed decision-making processes for both short-term tactical choices and longer-term strategic operations choices that impact scientific results. This should include the specific lines of communication and decision-making authority within the operations team as well as how feedback is solicited and incorporated from science collaborations and other stakeholders [by one year before operations start].

We will do this. A start has been made in rtn-005.

Work has been scheduled for FY24 Q1 and Q2 to partially address this recommendation.

PREOPS-3540

M-III.6-23

Change management plan and resourcing

JOR_2023_02

2024-03-01

In Progress

Ensure that sufficient resourcing is allocated during the pre-operations phase to allow all future operations staff to become engaged, regardless of their ongoing construction activities [by March 2024]. This includes but is not limited to: • Develop a plan to transition the existing internal project Science Team to operations, including detailing the processes for this group within the operations plan. • Continue work to solidify the processes that allow delegating responsibility to lower-level managers and enabling them to make appropriate decisions. • Deliver the change management plan that is understood and accepted by all staff with high priority.

We will do the first two bullets. It is not clear we need a change management plan so much as a transition activities that includes regular communication down to the individual team member, clarity and sign off on individual roles and responsibilities, etc.

Work is being scheduled on this for FY24 Q1 and Q2.

PREOPS-3539

M-III.6-22

Best hiring practices for all Rubin hiring committees

JOR_2023_02

2024-03-01

In Progress

Complete and implement a set of best hiring practices for all Rubin hiring committees, regardless of parent organization for the hire, that includes within it best practices to ensure diversity, equity, and inclusion [by March 2024].

We will do this.

Work on a Rubin Hiring Toolkit that addresses this recommendation has been scheduled for FY24 Q1 and Q2.

PREOPS-3538

M-III.6-21

Formalize advisory committee processes

JOR_2023_02

2024-03-01

In Progress

Formalize the make-up and meeting cadence of the advisory committees, particularly for the case of the science collaborations, to ensure the science collaborations are appropriately represented and that the process for acknowledging and responding to their advice is clear. Additionally, create a single unified organization chart that shows all advisory committees and the communication lines to management [by March 2024].

We will do this. Most of it already is documented in RDO-018.

Work is being scheduled to update RDO-018, the Rubin Operations Plan, in FY24 Q1 and Q2 prior to the next review.

PREOPS-3537

M-III.6-20

Develop salary equity process for AURA & SLAC

JOR_2023_02

2024-03-01

In Progress

Develop a process to ensure AURA and SLAC review salary and benefits at least annually to ensure similar compensation for similar positions and experience across the organizations [by March 2024].

This will depend on AURA and SLAC HR.

Work on this is scheduled for FY24 Q1 and Q2.

PREOPS-3536

M-III.6-19

Policy to allow employees to work from any state

JOR_2023_02

2024-02-01

In Progress

Make a policy change at AURA allowing for the hiring of employees to work in any state, without consideration for AURA’s existing presence in that state [by the next review].

Rubin will work with NOIRLab leadership to enable this change with AURA HQ. Rubin has already worked with NOIRLab to put in a formal request to add Oregon, NY, and North Carolina. This request is in AURAs hands.

As of end of November, AURA is working towards registering Oregon as a state from which AURA employees may work.

PREOPS-3535

ESH-III.5-18

Lessons learned folded into ops

JOR_2023_02

2023-06-01

Implemented

Ensure that the lessons learned and recommendations from construction are folded into operations. • Complete the Winter Operations Plan and Response to Weather Emergency Situations plan well in advance of the 2023 winter season to allow time for equipment procurement, dry run exercises, testing of equipment [by June 2023]. • Develop a Fatigue Mitigation Plan. Suggest including fatigue observation checklists and assessments [by June 2023]. • Perform an annually quantitative measure of the impact of implementing recommendations from the external traffic safety company evaluation. [Complete the first one prior to the next operations review.]

Winter plan 2023 on place Fatigue Mitigation program on progress Safety Road Enginnerering Study (done) Corrective Measures in progress

Winter plan 2023 on place

Fatigue Mitigation program on progress

Safety Road Enginnerering Study (done) Corrective Measures in progress

PREOPS-3534

ESH-III.5-17

Document lessons learned

JOR_2023_02

2023-06-01

Implemented

Document the lessons learned recovering from a major setback such as resuming work after 2 months of shutdown following the incident that took place at SLAC in December 2022 and share with the Rubin team [by June 2023].

Safety Lesson Learned has been distributed

[^Final_ORPS_NR_LOTO_procedural_violation.pdf]

PREOPS-3533

ESH-III.5-16

Plan to execute recommendations

JOR_2023_02

2023-09-01

In Progress

Create a plan for executing the recommendations from the ES&H review of the Rubin Observatory which took place on 10/6/22. Focus on electrical safety, confined spaces, ladder safety/working at heights and earthquake bracing [by September 2023].

Reinforce of :

Electrical Safety ; LOTO training update

Confined Sapce : Develop of a Confine Space Training

Working on height and Fall protecction : review of current traning and add of ladder safety

Hello

Understood . We will develop a summary report of training provided and updates . Plan is basically add those training to Safety Orienattion Package (for new hired) according to the hazards and risks analysis of the Job Position (i.e. An admin. positition will not take the confine space training)

PREOPS-3532

ESH-III.5-15

Implementation of Road Safety Study

JOR_2023_02

2023-07-01

In Progress

Deliver a timeline for implementing the recommendations from the Road Safety Study (Estudio de Seguridad Vial by Ambitrans Ingenieros Consultadores, SpA) report [by July 2023].

Minimun Commuting Time Test , March 2023 , done ,

Submit new Minimum Timing Table to AURA-O and RSLT (done)

GPS monitoring Submitted to AURA-O for approval (May 2023)

Distribution of new Minumum time comutting , June 2023

Hi, Got It , I will prepare a summary report and submited in order to close this one

PREOPS-3531

CS-III.4-14

Training on new tools

JOR_2023_02

2024-01-31

In Progress

Provide sufficient training on new tools and budget and the budget planning process for ADs and team leads [prior to the launch of next annual planning cycle].

We will train the ADs on all the tools necessary to do their part of planning and tracking.

Training on the budget planning process is being provided “on the job” via the annual scrub process. ADs and Team Leaders were walked through the Scrub Sandbox workbook (which imports dynamically from the planning tools) at the annual scrub launch in May, and will be debriefed on the scrub implementation at a closeout meeting on December 12.

PREOPS-3530

CS-III.4-13

Consolidated report Plan vs Actual

JOR_2023_02

2023-08-31

In Progress

Develop a consolidated report showing plan versus actuals by WBS by month for the Rubin Operations team and agencies [by summer 2023].

We will make this report. The NSF side is essentially already done in collaboration with Program Operations and NOIRLab Management Services.

This work is scheduled for FY24 Q1, so as to feed into the post-scrub implementation, top-down budget planning phase of the annual cycle. While formally late, we expect to have the tool in use before the next review.

PREOPS-3529

SP-III.3-12

Track RSP usage

JOR_2023_02

2024-06-28

Not Started

Continue to find ways to track how broadly the Rubin data and RSP are being used by the community. In particular, assess the impact of the LSST survey data reaching historically underrepresented groups [before Data Preview 1].

The RPF Community Science team will investigate ways to track RSP usage

Added Tina Adair as a watcher. Within CST she’s leading development of our “annual reports” which might (if it’s possible) include tracking this kind of usage.

PREOPS-3528

SP-III.3-11

Visibility of change procedures for the survey cadence, scheduling and strategy

JOR_2023_02

2024-02-29

Not Started

Increase the visibility of the change procedures of the survey cadence, scheduling, and strategy to inform the community and agencies regarding ongoing compliance with the SRD [by September 2023].

A process will be developed to make the change procedures of the survey cadence, scheduling, and strategy more visible and inform the community and agencies regarding ongoing compliance with the SRD.

We have a documentation site set up but it is down at the moment. More content needs to be added.

PREOPS-3527

SP-III.3-10

Add a formal review and sign off following pilot processing runs

JOR_2023_02

2024-02-01

Not Started

Add a formal review and sign off for the transition to data release processing. The signoff should occur between the end of the pilot run processing and the start of production processing [before the next review].

A formal review and sign off following pilot processing runs will be added

[~lguy] maybe we can delegate this to Colin?

PREOPS-3526

SP-III.3-9

Mitigate system engineering work overload

JOR_2023_02

2023-12-31

In Progress

Mitigate the upcoming system engineering work overload in the handoff between Rubin construction and RPF operations. One possible mitigation is the use of external contractors [by the end of 2023].

Consulting help will be hired to mitigate system engineering work overload

[~lguy] let’s get back to CAS about external contractors.

PREOPS-3525

SP-III.3-8

Implementation of the Operations CCB

JOR_2023_02

2023-12-31

Not Started

Complete the implementation of the RPF CCB and RB [by the end of 2023].

The Operations CCB will be implemented by the RPF Systems Engineering team. The Systems Engineering team has few resources to dedicate to pre-operations work so consulting help will be needed

[~lguy] this is a big one. Let’s strategize on how to make progress.

PREOPS-3524

DM-III.2-7

Metric of success

JOR_2023_02

2024-06-28

Not Started

Define a metric of success that enables the RDM team to evaluate their chosen user support model, and the efficacy of the community support [by Data Preview 1].

One or more metrics of success will be defined to enable evaluation of the chosen user support model, and the efficacy of the community support

This recommendation  is specific to the user support model. I suspect that the reviewer mistakenly put RDM when it should have been RPF.

PREOPS-3523

DM-III.2-6

Concise and complete set of performance metrics

JOR_2023_02

2024-06-28

In Progress

Develop and agree on a concise and complete set of performance metrics that are tracked by the team and reported to operations management [by Data Preview 1].

We do need to document a set of (non science science) performance metrics - it has not been done yet.

This says implementation is in progress but the issue is still at “to do”. Can you please update the issue status, [~womullan]. Thanks

PREOPS-3522

DM-III.2-5

Agreement with SLAC

JOR_2023_02

2023-06-01

In Progress

Re-evaluate the agreement with SLAC offering new personnel software staff positions and consider instead continuing existing software staff that could transition from the NSF construction project [by June 2023].

The FY24 Scrub allowed us to explore this topic and make some advances. SLAC has been working with us to retain more staff from construction. It not obvious we can re-valuate the agreement with SLAC, DOE need to fund half the project. We will keep pressing our case.

I am ok with the progress on this. I think we have made progress. Beyond FY24 there is still detail scrubbing to do. Do we close this out for now. I think so. Will approve and Wil can reopen if needed.

PREOPS-3521

DM-III.2-4

Decision to split software developer

JOR_2023_02

2024-02-01

In Progress

Re-evaluate the decision to split software developer management between different departments [by the next operations review].

We will convene a discussion among the ADs to see how to best move forward.

We will convene a discussion among the ADs to see how to best move forward.

PREOPS-3520

OO-III.1-3

Plan for SLAC maintenance

JOR_2023_02

2023-10-01

In Progress

Ensure that the plan for SLAC maintenance of the LSSTcam is fully described for future reviews [by FY 2024].

Both the construction and operations teams (they are essentially one in the same) continue to work closely with the team at SLAC developing LSSTCam. Maintenance plans are continually being updated as specific details are learned during the final test and verification action phase for the instrument. These details are being folded into the Rubin Observatory Operations (ROO) strategic maintenance plan and include reassessment of both technical and personnel resource needed to maintain LSSTCam through the 10-year LSST program. The ROO Strategic Maintenance PLan do men tis a key deliverable for FY23 and will be a featured discussion point in subsequent reviews.

Blum and Marshall met with Camera team in November at SLAC to discuss maintenance and OPS supports needs. Claver visited SLAC the following week to follow up and further advance planning.

PREOPS-3519

OO-III.1-2

Re-consider the on-site presence

JOR_2023_02

2023-10-01

Not Started

Re-consider the on-site presence of engineering and technical support staff during nighttime for the first years of operations, until steady-state operation with nominal unplanned downtime has been reached [by FY 2024].

As the construction project progresses through its commissioning phase it is clearly evident that significant technical presence will be needed in the early stages of operations. The Rubin Observatory Operations (ROO) plan as presented is for steady state with a minimal technical presence needed at the summit. We have always planned that there would be a transition period between the end of construction and steady state operations. The details of the needed technical staff during nighttime operations is becoming clearer as the construction commissioning advance. Adjustment are currently being made in the detailed transition plan for nighttime support.

With the need for Technical ical support in mind we are also training up the nighttime Observing Specialists to have some level of technical awareness and expertise to server as the front line for diagnosis and solving technical issues as they arise. This is a core part of the Observing Specialists functions and has been part of the plan from the beginning. The technical training of the Observing Specialists is part of the designed work scope during construction commissioning.

As the construction project progresses through its commissioning phase it is clearly evident that significant technical presence will be needed in the early stages of operations.  The Rubin Observatory Operations (ROO) plan as presented is for steady state with a minimal technical presence needed at the summit.  We have always planned that there would be a transition period between the end of construction and steady state operations.  The details of the needed technical staff during nighttime operations is becoming clearer as the construction commissioning advance.  Adjustment are currently being made in the detailed transition plan for nighttime support.

With the need for Technical ical support in mind we are also training up the nighttime Observing Specialists to have some level of technical awareness and expertise to server as the front line for diagnosis and solving technical issues as they arise.  This is a core part of the Observing Specialists functions and has been part of the plan from the beginning.  The technical training of the Observing Specialists is part of  the designed work scope during construction commissioning.

PREOPS-3518

OO-III.1-1

Monitor the effectiveness of the initial maintenance strategy

JOR_2023_02

2023-10-01

Not Started

Monitor the effectiveness of the initial maintenance strategy during the first 2 years of operation and adjust the strategy accordingly if needed. Reserve corresponding resources in the operations plan [by FY 2024].

As part f the Rubin Construction integration, test and commissioning effort the Rubin Observatory Operations (ROO) team its carefullym monitoring the maintenance needs of the observatory. As systems are bing integrated and tested the ROO team ias part of their construction/operations dual responsibilities are documenting technical issues and noting specific maintenance needs. These will be recorded in the Computerized Maintenance Management System (CMMS) under development by the construction project. This body if information is currently being used to update the maintenance strategy and planning with resultant resources needed on various timescales.

The ROO Strategic Maintenance Plan is a key deliverable from the FY23 early operations efforts. This document is being develop[ed now with contributions from the ROO Team and Group leads (who are also part of the Rubin construction team). Staffing and non-labor resource adjustments are expected from the Maintenance Strategic Plan update and will be part of revised operations planning for ROO in FY24.

As part f the Rubin Construction integration, test and commissioning effort the Rubin Observatory Operations (ROO) team its carefullym monitoring the maintenance needs of the observatory.  As systems are bing integrated and tested the ROO team ias part of their construction/operations dual responsibilities are documenting technical issues and noting specific maintenance needs.  These will be recorded in the Computerized Maintenance Management System (CMMS) under development by the construction project.  This body if information is currently being used to update the maintenance strategy and planning with resultant resources needed on various timescales.

The ROO Strategic Maintenance Plan is a key deliverable from the FY23 early operations efforts.  This document is being develop[ed now with contributions from the ROO Team and Group leads (who are also part of the Rubin construction team).  Staffing and non-labor resource adjustments are expected from the Maintenance Strategic Plan update and will be part of revised operations planning for ROO in FY24.

Fri Dec 1 14:28:18 UTC 2023