RTN-064
JOR recommendations#
Review recommendations extracted from Jira.
2024-09-10 19:28
Issue key |
Rec# |
Summary |
Report date |
Due Date |
Implementation Status |
Description |
Response |
Implementation Status Description |
|---|---|---|---|---|---|---|---|---|
OO-III.1-1 |
Monitor the effectiveness of the initial maintenance strategy |
JOR_2023_02 |
2023-10-01 |
Implemented |
Monitor the effectiveness of the initial maintenance strategy during the first 2 years of operation and adjust the strategy accordingly if needed. Reserve corresponding resources in the operations plan [by FY 2024]. |
As part of the Rubin Construction integration, test and commissioning effort the Rubin Observatory Operations (ROO) team its carefully monitoring the maintenance needs of the observatory. As systems are being integrated and tested the ROO team as part of their construction/operations dual responsibilities are documenting technical issues and noting specific maintenance needs. These will be recorded in the Computerized Maintenance Management System (CMMS) under development by the construction project. This body of information is currently being used to update the maintenance strategy and planning with resultant resources needed on various timescales. The ROO Strategic Maintenance Plan is a key deliverable for early operations efforts. This document is being developed now with contributions from the ROO Team and Group leads (who are also part of the Rubin construction team). Staffing and non-labor resource adjustments are expected from the Maintenance Strategic Plan update and will be part of revised operations planning for ROO in FY24. |
Monitoring of the maintenance strategy has started and findings are being recorded in the CMMS. Three active risks are being tracked, with funds held in reserve to cover a combined cost exposure of $128k. |
|
OO-III.1-2 |
Re-consider the on-site presence |
JOR_2023_02 |
2023-10-01 |
Implemented |
Re-consider the on-site presence of engineering and technical support staff during nighttime for the first years of operations, until steady-state operation with nominal unplanned downtime has been reached [by FY 2024]. |
As the construction project progresses through its commissioning phase it is clearly evident that significant technical presence will be needed in the early stages of operations. The Rubin Observatory Operations (ROO) plan as presented is for steady state with a minimal technical presence needed at the summit. We have always planned that there would be a transition period between the end of construction and steady state operations. The details of the needed technical staff during nighttime operations is becoming clearer as the construction commissioning advance. Adjustment are currently being made in the detailed transition plan for nighttime support. With the need for Technicalical support in mind we are also training up the nighttime Observing Specialists to have some level of technical awareness and expertise to serve as the front line for diagnosis and solving technical issues as they arise. This is a core part of the Observing Specialists functions and has been part of the plan from the beginning. The technical training of the Observing Specialists is part of the designed work scope during construction commissioning. |
This is complete as far as the original rec to reconsider plans is concerned. We have initiated a discussion with NOIRLab and SOAR+Gemini to also add a shared nighttime technical resource on Pachón. |
|
OO-III.1-3 |
Plan for SLAC maintenance |
JOR_2023_02 |
2023-10-01 |
Implemented |
Ensure that the plan for SLAC maintenance of the LSSTcam is fully described for future reviews [by FY 2024]. |
Both the construction and operations teams (they are essentially one in the same) continue to work closely with the team at SLAC developing LSSTCam. Maintenance plans are continually being updated as specific details are learned during the final test and verification action phase for the instrument. These details are being folded into the Rubin Observatory Operations (ROO) strategic maintenance plan and include reassessment of both technical and personnel resource needed to maintain LSSTCam through the 10-year LSST program. The ROO Strategic Maintenance Plan does mention this as a a key deliverable for FY23 and will be a featured discussion point in subsequent reviews. |
As a deliverable from the LSSTCam construction project a Rubin Technical Document (RTD-1041) has been created to index all existing camera procedures (including maintenance). It lists 432 released camera procedures and lift plans. ~800 maintenance use cases are also included with notes on which camera and summit procedures are needed. Camera procedures to be executed on the summit are re-released with summit specific updates (“Authorized for Summit Use” stamped). Finally, Diane Hascall, system engineer from the camera team is integrated in the Rubin operations maintenance planning team. |
|
DM-III.2-4 |
Decision to split software developement |
JOR_2023_02 |
2024-02-01 |
Implemented |
Re-evaluate the decision to split software developer management between different departments [by the next operations review]. |
We will convene a discussion among the ADs to see how to best move forward. |
The decision for software organization in Operations has been evaluated. There is consensus to have a overarching role at the Operations directorate level to manage SW within Rubin. The ultimate organization will be set before full survey operations. The current scope of how this might work and what the roles and responsibilities are outlined in the last section of [rtn-069|http://ls.st/rtn-069]. |
|
DM-III.2-5 |
Agreement with SLAC |
JOR_2023_02 |
2023-06-01 |
Implemented |
Re-evaluate the agreement with SLAC offering new personnel software staff positions and consider instead continuing existing software staff that could transition from the NSF construction project [by June 2023]. |
The FY24 Scrub allowed us to explore this topic and make some advances. SLAC has been working with us to retain more staff from construction. It not obvious we can re-valuate the agreement with SLAC, DOE need to fund half the project. We will keep pressing our case. |
The FY24 Scrub allowed us to explore this topic and make some advances. SLAC has been working with us to retain more staff from construction. Directors office is satisfied with the progress made and approves this recommendation for closure. |
|
DM-III.2-6 |
Concise and complete set of non-science performance metrics |
JOR_2023_02 |
2024-06-28 |
Implemented |
Develop and agree on a concise and complete set of performance metrics that are tracked by the team and reported to operations management [by Data Preview 1]. |
We do need to document a set of (non science science) performance metrics - it has not been done yet. |
RTN-077 captures a set of non science metrics - ensuring completeness is perhaps difficult. |
|
DM-III.2-7 |
User Support Model – Metric of success |
JOR_2023_02 |
2024-12-31 |
In Progress |
Define a metric of success that enables the RDM team to evaluate their chosen user support model, and the efficacy of the community support [by Data Preview 1]. |
One or more metrics of success will be defined to enable evaluation of the chosen user support model, and the efficacy of the community support |
CST has started developing and measuring user support metrics as part of our new annual reviews. Confluence page describing the CST annual reports and listing all the things we’d like to track and measure: [https://rubinobs.atlassian.net/wiki/display/LSSTOps/Annual+Reports|https://rubinobs.atlassian.net/wiki/display/LSSTOps/Annual+Reports] Confluence page with the first CST 2023 Annual Report: [https://rubinobs.atlassian.net/wiki/display/LSSTOps/FY+2023+CST+Report|https://rubinobs.atlassian.net/wiki/display/LSSTOps/FY+2023+CST+Report] And the relevant work tickets that produced this annual report: [https://jira.lsstcorp.org/browse/PREOPS-3748|https://jira.lsstcorp.org/browse/PREOPS-3748] |
|
SP-III.3-8 |
Implementation of the Operations CCB |
JOR_2023_02 |
2023-12-31 |
In Progress |
Complete the implementation of the RPF CCB and RB [by the end of 2023]. |
The Operations CCB will be implemented by the RPF Systems Engineering team. The Systems Engineering team has few resources to dedicate to pre-operations work so consulting help will be needed |
The CCB is in place. Change requests are tracked in the [CCB|https://rubinobs.atlassian.net/browse/CCB/] Jira Project. An open slack channel, #ops-ccb, is in place for discussion and communication of change requests and is integrated with the Jira project. Currently only changes to documents are being tracked managed but as we move into Operations, the scope will expand. Regular meetings are not yet needed and will be started when there is need. The RRB is also in place with meetings occurring monthly. Risks are managed in the NOIRLab risk tool and cannot be easily integrated with slack, Jira. Candidate risks are being proposed and studied. |
|
SP-III.3-9 |
Mitigate system engineering work overload |
JOR_2023_02 |
2023-12-31 |
In Progress |
Mitigate the upcoming system engineering work overload in the handoff between Rubin construction and RPF operations. One possible mitigation is the use of external contractors [by the end of 2023]. |
Consulting help will be hired to mitigate system engineering work overload |
Austin Roberts, the Lead Systems Engineer resigned before this was complete. Holger Drass has stepped into the role of Lead Systems Engineer and a new DOE staff member onboarded. This is on hold for now and will be reviewed in FY25 |
|
SP-III.3-10 |
Add a formal review and sign off following pilot processing runs |
JOR_2023_02 |
2024-04-09 |
In Progress |
Add a formal review and sign off for the transition to data release processing. The signoff should occur between the end of the pilot run processing and the start of production processing [before the next review]. |
A formal review and sign off following pilot processing runs will be added |
Formal review and sign-off following pilot processing runs and before starting production processing is the job of the Data Release Board. The details of this process will be described in a document – Charge to the Data Release Board – [https://rtn-052.lsst.io/|https://rtn-052.lsst.io/|smart-link] has been started. It will be developed as we move through commissioning and prepare the Data Previews and early Alert Production. |
|
SP-III.3-11 |
Visibility of change procedures for the survey cadence, scheduling and strategy |
JOR_2023_02 |
2024-02-29 |
In Progress |
Increase the visibility of the change procedures of the survey cadence, scheduling, and strategy to inform the community and agencies regarding ongoing compliance with the SRD [by September 2023]. |
A process will be developed to make the change procedures of the survey cadence, scheduling, and strategy more visible and inform the community and agencies regarding ongoing compliance with the SRD. |
The website [http://survey-strategy.lsst.io|http://survey-strategy.lsst.io|smart-link] now contains all information currently available about the survey. This site will continue to be updated as the strategy evolves and is executed when we enter operations. |
|
SP-III.3-12 |
Track RSP usage |
JOR_2023_02 |
2024-12-31 |
Implemented |
Continue to find ways to track how broadly the Rubin data and RSP are being used by the community. In particular, assess the impact of the LSST survey data reaching historically underrepresented groups [before Data Preview 1]. |
The RPF Community Science team will investigate ways to track RSP usage |
DP0 continues to draw in new users for the Rubin Science Platform (see plot at lower right). We ran a PCW session on supporting science at small and underserved US institutes (SUIs), and continue to reach out to establish new contacts and support scientists with one-on-one meetings. As of February 2024 there were 984 RSP accounts, and 64% (632) of them used an institutional identity provider or email address, 55% (350) of which were in the USA. Of them, 13% (47) are non-R1/R2 institutes. That we have nearly 50 participants from teaching-focused institutes is really great, and is a direct result of our outreach to SUIs. This is deemed sufficient to mark this recommendation implemented, though we plan to continue monitoring use going forward. |
|
CS-III.4-13 |
Consolidated report Plan vs Actual |
JOR_2023_02 |
2023-08-31 |
Implemented |
Develop a consolidated report showing plan versus actuals by WBS by month for the Rubin Operations team and agencies [by summer 2023]. |
We will make this report. The NSF side is essentially already done in collaboration with Program Operations and NOIRLab Management Services. |
Here is the working epic https://jira.lsstcorp.org/browse/PREOPS-4152 |
|
CS-III.4-14 |
Training on new tools |
JOR_2023_02 |
2024-01-31 |
Implemented |
Provide sufficient training on new tools and budget and the budget planning process for ADs and team leads [prior to the launch of next annual planning cycle]. |
We will train the ADs on all the tools necessary to do their part of planning and tracking. |
The Rubin Directorate held a briefing on the FY23 scrub outcome and reviewed the staffing tools as they are deployed during the scrub by the Team leads. The briefing was attended by team leads for ROO, RDM, RPF, REO and RDO. Briefing slides are here. Notes and feedback/Q&A were captured in the private Confluence pages of the RDO OPS Exec. This training completes the JOR23 recommendation, but training will continue through each scrub cycle. |
|
ESH-III.5-15 |
Implementation of Road Safety Study |
JOR_2023_02 |
2023-07-01 |
Implemented |
Deliver a timeline for implementing the recommendations from the Road Safety Study (Estudio de Seguridad Vial by Ambitrans Ingenieros Consultadores, SpA) report [by July 2023]. |
Minimun Commuting Time Test , March 2023 , done , Submit new Minimum Timing Table to AURA-O and RSLT (done) GPS monitoring Submitted to AURA-O for approval (May 2023) Distribution of new Minumum time comutting , June 2023 |
||
ESH-III.5-16 |
Plan to execute recommendations |
JOR_2023_02 |
2023-09-01 |
Implemented |
Create a plan for executing the recommendations from the ES&H review of the Rubin Observatory which took place on 10/6/22. Focus on electrical safety, confined spaces, ladder safety/working at heights and earthquake bracing [by September 2023]. |
Reinforce of : Electrical Safety ; LOTO training update Confined Sapce : Develop of a Confine Space Training Working on height and Fall protecction : review of current traning and add of ladder safety |
[https://docushare.lsst.org/docushare/dsweb/G]et/Document-40659/Safety%20Orientation%202023eng.pdf |
|
ESH-III.5-17 |
Document lessons learned |
JOR_2023_02 |
2023-06-01 |
Implemented |
Document the lessons learned recovering from a major setback such as resuming work after 2 months of shutdown following the incident that took place at SLAC in December 2022 and share with the Rubin team [by June 2023]. |
Safety Lesson Learned has been distributed |
||
ESH-III.5-18 |
Lessons learned folded into ops |
JOR_2023_02 |
2023-06-01 |
Implemented |
Ensure that the lessons learned and recommendations from construction are folded into operations. • Complete the Winter Operations Plan and Response to Weather Emergency Situations plan well in advance of the 2023 winter season to allow time for equipment procurement, dry run exercises, testing of equipment [by June 2023]. • Develop a Fatigue Mitigation Plan. Suggest including fatigue observation checklists and assessments [by June 2023]. • Perform an annually quantitative measure of the impact of implementing recommendations from the external traffic safety company evaluation. [Complete the first one prior to the next operations review.] |
Winter plan 2023 on place Fatigue Mitigation program on progress Safety Road Enginnerering Study (done) Corrective Measures in progress |
Winter plan 2023 on place Fatigue Mitigation program on progress Safety Road Enginnerering Study (done) Corrective Measures in progress |
|
M-III.6-19 |
Policy to allow employees to work from any state |
JOR_2023_02 |
2024-02-01 |
Implemented |
Make a policy change at AURA allowing for the hiring of employees to work in any state, without consideration for AURA’s existing presence in that state [by the next review]. |
Rubin will work with NOIRLab leadership to enable this change with AURA HQ. Rubin has already worked with NOIRLab to put in a formal request to add Oregon, NY, and North Carolina. This request is in AURAs hands. |
A process has been defined by AURA to request states expansion. Rubin has worked closely with NOIRLab leadership to expand in states where we have an immediate need and/or we see potential for hiring, particularly in software engineering and development. SLAC can hire in any state via Stanford and we have used this ability in several recent cases. |
|
M-III.6-20 |
Develop salary equity process for AURA & SLAC |
JOR_2023_02 |
2024-03-01 |
Implemented |
Develop a process to ensure AURA and SLAC review salary and benefits at least annually to ensure similar compensation for similar positions and experience across the organizations [by March 2024]. |
This will depend on AURA and SLAC HR. |
Working with the Program Operations team, we have developed a process to compare salary ranges for SLAC and NOIRLab staff broken out by broad job category and geographic location. The latter allows us to correct for cost of living differences. The process avoids individual salary comparison as SLAC and AURA HR do not allow outside the institution sharing of salary data for named individuals. Instead we use general costing data aggregated by job type and location. This comparison is built in to our tools and so is automatically updated as salaries change. Importantly, the comparison shows scientist and management salaries compare well (corrected for cost of living in different locations). Indeed, our recent experience in recruitment suggests salary disparity is not an issue. We had an ongoing hire in NOIRLab that we could not fulfill for a remote software role. The role was switched to SLAC and the candidate negotiated a salary consistent with what NOIRLab would have provided. Also, a recent postdoc finished a job at NOIRLab in Tucson, and was offered a new job for Rubin in Menlo Park (a competitive recruitment for KAVLI). The salary expectations fit comfortably within the Stanford/KAVLI range and were not out of line with the immediately prior NOIRLab salary. Technician salaries do not compare well on value, but this is due to somewhat different skill sets and cost of living between primarily Menlo Park and Chile. |
|
M-III.6-21 |
Formalize advisory committee processes |
JOR_2023_02 |
2024-03-01 |
Implemented |
Formalize the make-up and meeting cadence of the advisory committees, particularly for the case of the science collaborations, to ensure the science collaborations are appropriately represented and that the process for acknowledging and responding to their advice is clear. Additionally, create a single unified organization chart that shows all advisory committees and the communication lines to management [by March 2024]. |
We will do this. Most of it already is documented in RDO-018. |
Committee information is detailed in [RDO-018, section 12.2|https://docs.google.com/document/d/1kIZUl2SoIg3-Lfqtjihshh5aEZxl8pXBiIRHxRELF-g/edit#bookmark=id.u27ihajmoeqn]. A number of key committees will (do) aid the leadership team in maximizing science from Rubin Observatory. Rubin ensures representation of the Science Collaborations on these committees as laid out in the [+federation agreement with the Science Collaborations+|https://drive.google.com/file/d/1YkYQuNe-DnorNH0_FFHAyz4vVeOmu0um/view?usp=sharing] and the committee charge documents noted above. The [+science advisory committee+|https://project.lsst.org/groups/sac/welcome] membership explicitly includes representation from all science collaborations. The _main_ communication and feedback channels for the committees and Rubin and the Science Collaborations are shown in the figure below. !ph4dESiPAO5ssRZ9ALEvny7FaUaDF6TmTJScMRH-GQMK8-OhiDy29nkiEx-QhP7iRfeCjxsyEapbTvlGMHy-74vuLdSJOm1Wou8tLBB6KdFCsiWmhjHk4RBzRxIhqiOkGs0wgw_WcuUdlke2OwPbn1U|width=657,height=495! Figure - Rubin operational diagram with key communication and feedback channels for existing committees convened by Rubin. DM Liaisons work directly with science collaborations. Websites are available for all the key committees (see [RDO-018 sec 12.2|https://docs.google.com/document/d/1kIZUl2SoIg3-Lfqtjihshh5aEZxl8pXBiIRHxRELF-g/edit#bookmark=id.u27ihajmoeqn]), listing their membership and including information on meetings and other aspects. The SAC and UC, both central to the science community and its connection to Rubin, meet annually at the Rubin Community Workshop (where open sessions are included). |
|
M-III.6-22 |
Best hiring practices for all Rubin hiring committees |
JOR_2023_02 |
2024-03-01 |
In Progress |
Complete and implement a set of best hiring practices for all Rubin hiring committees, regardless of parent organization for the hire, that includes within it best practices to ensure diversity, equity, and inclusion [by March 2024]. |
We will do this. |
A Rubin Hiring Toolkit that addresses this recommendation is under development, with a “Hiring Recipes” best practices document, RTN-070, distributed to current Rubin hiring managers as “beta testers”. The Hiring Recipes will be launched to all Rubin staff and the LSST Science Community in September. The full toolkit will be built out during the FY25 hiring season. |
|
M-III.6-23 |
Change management plan and resourcing |
JOR_2023_02 |
2024-03-01 |
Implemented |
Ensure that sufficient resourcing is allocated during the pre-operations phase to allow all future operations staff to become engaged, regardless of their ongoing construction activities [by March 2024]. This includes but is not limited to: • Develop a plan to transition the existing internal project Science Team to operations, including detailing the processes for this group within the operations plan. • Continue work to solidify the processes that allow delegating responsibility to lower-level managers and enabling them to make appropriate decisions. • Deliver the change management plan that is understood and accepted by all staff with high priority. |
We will do the first two bullets. It is not clear we need a change management plan so much as a transition activities that includes regular communication down to the individual team member, clarity and sign off on individual roles and responsibilities, etc. |
Transition the Project Science Team. The LSST Project Science Team (PST) serves as an operational unit, within the Project, that carries out specific scientific performance investigations as prioritized by the Director, the Project Manager, and the Project Scientist. Its membership includes key scientists on the Project who provide specific necessary expertise. The Project Science Team provides required scientific input on critical technical decisions as the project construction proceeds. The current PST already is focused on Operations related issues, for example, satellite constellations and science impact based on agency mandated data security protocols. The PST also communicates with the Science Collaboration chairs on a regular basis to get input and communicate important science relevant issues or advances (e.g. throughput enhancements due to Ag vs Al mirror coatings). In Operations, we will transition the team as is to continue to support the LSST and overall performance of the survey. The Head of LSST will lead this group and be supported by the scientists from around the Operations Program as well as the broader+engaged scientists that currently serve. In year one of the survey, we will review the membership, offer those who wish a chance to roll off and add new expertise as appropriate/needed. The current Project Scientist and Deputy Scientist will be invited to continue as members of the OPS Science Team (OST). Delegation of responsibility and authority has been advanced in rtn-005. This includes budget/account authority pushed to levels below Department Assoc. Dirs as well as the planning process that includes input, priority setting, and ownership of plans at the team level of teh WBS (level 3). See also [https://rubinobs.atlassian.net/browse/PREOPS-3541|https://rubinobs.atlassian.net/browse/PREOPS-3541|smart-link]. Change Management. The following actions are being taken to manage change and transition into Operations:
|
|
M-III.6-24 |
Develop, document, and socialize detailed decision-making processes |
JOR_2023_02 |
2024-06-01 |
Implemented |
Develop, document, and socialize the detailed decision-making processes for both short-term tactical choices and longer-term strategic operations choices that impact scientific results. This should include the specific lines of communication and decision-making authority within the operations team as well as how feedback is solicited and incorporated from science collaborations and other stakeholders [by one year before operations start]. |
We will do this. A start has been made in rtn-005. |
Current practice for both Strategic and Tactical decision making is embedded within the Rubin Director’s Operations Executive Council (Ops Exec). This council meets weekly and discusses all current issues within Operations that need near term decisions. Planning is led from the Ops Exec and Department level tasks and deliverables are delegated to the Department leaders. Implementation and performance are tracked by the Director’s Office and regularly reported to NSF, DOE, SLAC, and NOIRLab. The Ops Exec periodically meets with key Team Leads for specific planning tasks, like the annual budget scrub. These aspects are covered in [rtn-011|http://ls.st/rtn-011]. Long term strategy is developed in coordination with stakeholders (operations partners SLAC and NOIRLab). Such development can be Rubin-partner, or Rubin and both partners. The overall direction of the Observatory is approved by the Rubin Management Board. The Board is comprised of the NOIRLab and SLAC Directors or their delegates and other members of the two partners as well as community members. The Board is fully described in [RDO-018 section 9.|http://ls.st/RDO-018] Community input and advice is also used in the development of long term strategy and advice. The two principal conduits to support decision making are the Science Advisory Committee and the Users Committee. Other key community based input comes from the Survey Cadence Optimization Committee, the Contribution Evaluation Committee, and the Resource Allocation Committee. The SAC has a role in helping the Rubin Director establish all these committees. Each is described in [RDO-018|http://ls.st/RDO-018]. The SAC and UC also have charters defining their activities as charged by the Rubin Director: [SAC charte|https://docs.google.com/document/d/1qzFH8T3pc57OW_frvUShrNYqdIk1NOdHeqtgQlu5-vQ/edit]r, [UC Charter|http://ls.st/RDO-051]. Meetings and input gathering from the community as well as reporting are defined in the charters. |
|
M-III.6-25 |
Hold a risk register scrub/workshop |
JOR_2023_02 |
2023-09-01 |
Implemented |
Hold a dedicated risk register scrub/workshop with subsystem leads and risk owners as well experts in risk management, in addition to the upcoming cost scrub to ensure focus on risk and complete the risk register. This activity should engage all levels of management that will be involved in operations [by September 2023]. |
This was completed in May 2023. The scrub is done and the RDO is updating budgets and Risks. |
A pre-scrub risk workshop was held in April 2023, and served to homogenize and finalize the cost and schedule impact analysis of the risk register, as well as surface some new candidate risks. This workshop was attended by the Risk Board, which includes several internal experts in risk management. |
Tue Sep 10 19:28:46 UTC 2024