SNS Accelerator Advisory Committee
2016
Recommendations & Resolutions
Recommendations 1, 20, 22, & 27: Begin Beam Testing of the RFQ as quickly as possible. Then following completion of the Beam Authorization Basis, quickly proceed to RFQ commissioning.
Resolution: The Beam Authorization Basis documentation was completed and an Accelerator Readiness Review was successfully conducted on August 2, 2016. The final committee report was issued on August 8, 2016. Pre-Start Action Items were completed and authorization to operate was granted on September 6, 2016. The first beam through the RFQ to the beam stop for Commissioning occurred on September 9, 2016.
Recommendation 2: Carefully review the plan and schedule for all beam tests to make sure there is sufficient detail to prevent delays once approvals are received and all items needed for go/no-go decision are included in the plan. Issue the RFQ commissioning and testing plan.
Resolution: The Beam Test Facility (BTF) Low-Power Test Measurements and High Power Commissioning, Spallation Neutron Source (SNS) Document number SNS RAD-BI-PN-001-R00, plan was approved and issued on August 30, 2016. The plan includes all necessary tests for go/no-go decision. Namely, they are the RFQ transmission measurement, the RFQ output energy measurement, the transverse RFQ output emittance measurement and the full power 24-hour endurance run. The plan has a detailed list of step-by-step tasks with realistic duration estimates.
Recommendation 3: Once the new RFQ is installed, appoint a manager for the BTF who will be responsible for making it operational again with the old RFQ, and then for operation of the facility for the planned future research.
Resolution: Sasha Aleksandrov was appointed as the BTF Manager on May 3, 2016.
Recommendation 4: Continue to develop and refine the PyORBIT code.
Resolution: Work on developing and refining the PyORBIT code is continuing. It is now capable of routine simulations of both the Linac and the ring. Additional diagnostics modules are in progress.
Recommendation 5: Consider adding the redesign of the electron catcher to the project prioritization planning.
Resolution: Redesign of the electron catcher has been added to the project prioritization under Task MSO-18, Chicane 02 Vacuum Vessel and e-Catcher Emergency Spare.
Recommendation 6: Consider an approximately 1 month lagging start to IRP replacement during the 4 month shutdown (or delay the start of the long shutdown).
Resolution: The scheduling of the IRP Replacement has been delayed by events at the manufacturer. The IRP Replacement outage is currently scheduled to begin December 22, 2017.
Recommendation 7: Perform risk analysis to compare relative consequences of failure of the hot cell servo-manipulator among the potential AIPs under consideration.
Resolution: An AIP prioritization methodology was developed which includes Research Accelerator Division (RAD) Performance Criteria and Risk. All current and proposed AIPs were evaluated using this methodology. The Servo-Manipulator was one of the proposed AIPs that was evaluated.
Recommendations 8, 10, & 29: Place more emphasis on an effective/stable gas bubble injection system, setting requirements/scope to significantly mitigate pressure wave. Incorporate bubble injection system with jet-flow design in the next design/build cycle. Incorporate target vessel instrumentation. Consider reprioritization as listed below.
8.1 Redirect the design of three in-fabrication jet flow targets for retrofit of gas injection hardware.
Resolution: A focused emphasis on gas injection and diagnostics is necessary. SNS-specific bubbler technologies are being developed using the Target Test Facility (TTF) and the water-loop target facility with collaboration from Japanese Spallation Neutron Source (JSNS) staff. Plans for high-gas flow rate implementation into the mercury loop are also proceeding. Potential safety consequences for gas injection into the mercury process loop, which was not originally designed for gas injection, must first be mitigated. In the meantime, an expedited gas-injection approach is being pursued that should be moderately effective at mitigating target strains created by the pulse pressure wave.
The design of three in-fabrication jet-flow targets have been re-directed for retrofit of gas injection hardware. Two of those targets were received by the end of FY16, and a third will be received before the end of FY17. Already two targets have been operated with strain instrumentation in place. A gas injection and handling system is being designed.
A Design Change Notice and Thermal Hydraulic Analysis were filed as evidence of closure.
8.2 Design a gas injection and handling system.
Resolution: Not closed - A gas injection system is being designed.
Recommendation 9: Use fatigue and stress analyses to set a quantitative minimum reduction in strain required to achieve satisfactory target vessel performance.
Resolution: Accepted industrial codes, such as the ASME Boiler and Pressure Vessel Code, are not comprehensively applicable for the SNS target geometry, loads and environment, so that existing general design criteria are not reliable for quantitatively evaluating the margin to fatigue failure. However, as strain data for multiple targets become available from in-beam measurements at SNS, acceptable strain values can be estimated from operational experience. In theory, as these same strain data also reveal the relational dependency of stress/strain upon beam power, the required mitigation of strain at an anticipated higher beam power can be calculated by extrapolation. However, the mitigation by gas injection will not be uniform across the target vessel, and the accuracy of the extrapolation will also vary with the non-uniformity.
The required reduction of strain will be determined by using the fatigue and stress analysis to extrapolate measured and calculated results for successful target performance to anticipated higher beam powers. The quantitative requirement will be available when gas is first injected to a mercury target.
Recommendation 11: Keep a focus on Cyber Security. Keep current with evolving NIST standards. Try to maintain a risk-based approach to mitigating measures. Unfortunately, some impediments to ease of use are almost inevitable. Educating staff on the importance of cyber-security is a good thing.
Resolution: The SNS infrastructure team remains vigilant in their efforts to keep the Integrated Control System (ICS) network secure from cyber attacks. The system is validated twice a year using the highly regarded ICS-Cert tools provided by the Department of Homeland (DHS) Security. The Intrusion Detection System (IDS) added last year provides an additional level of monitoring supported by ORNL IT and has not detected any attempts to access the system by unauthorized individuals. To stay abreast of the latest challenges in this area, the system administrator and protection system team leader participated in the DHS Industrial Control Systems Joint Working Group Fall Meeting. Controls Group staff members use dedicated laptops for access to Programmable Logic Controllers (PLCs) on the ICS network and are trained by their supervisors on procedures to maintain these laptops with appropriate security patches. E-mail, a common threat vector, is not available on ICS workstations. All employees take Cyber Security Awareness Training annually.
Recommendation 12: Make sure your backups are reliable, including those of user data. Among the many other good reasons for reliable backups, it is a key mitigation for addressing ransom-ware attacks.
Resolution: Backups of accelerator software have traditionally been executed daily using a set of 24 rotating tapes. Tapes are stored in the front end building, geographically separate from the central equipment room which houses our server infrastructure. Backups are tested periodically and user requests for restoration of missing files have been successfully performed. The log file associated with a recent (11/30/2016) backup/restore test can be located in the closure evidence file. Additionally, we have augmented the tape backup system with daily disk-to-disk file copies, capable of storing approximately one year of backup files on-line for quick recovery.
Recommendation 13: Consider adopting the same code management tools for other PLC systems at the facility, including, but not limited to those for which Controls Group has responsibility. Importantly this should include the Personnel Protection System (PPS) PLC systems. This is an important tool for ensuring and validating that correct versions of code are deployed.
Resolution: All Process Control Team PLC and PanelView code for accelerator and instruments uses AssetCentre to maintain version control. The Software Team RF PLC and PanelView code was migrated into AssetCentre with the exception of HV Convertor Modulator's code as replacement with Labview is well underway. The Hardware Team MPS PLC code will be transferred into AssetCentre by the end of February 2017. Now that the Controls Group has experience with AssetCentre, the Protection System Team plans to migrate their PLC code to this repository in FY18. Before this can take place, the team must augment the requirements and procedures for use, verification, cyber security, and isolation of safety logic from other facility PLC code, hence the longer timeline.
Recommendation 14: Continue to address Control System Studio - Best OPI Yet (CSS-BOY) scaling problems. This is key if CSS-BOY (or its evolved version) is to replace Extensible Display Manager (EDM).
Resolution: The Display Builder is a comprehensive update of the CS-Studio BOY panel editor and runtime engine. Its modular design separates the model, i.e., the information about widgets and their properties, from the graphical representation and the runtime engine. The model is fully multithreaded. The representation has been demonstrated in both Standard Widget Toolkit (SWT) and Java Effects (JavaFX), for now intending to concentrate on the latter. The runtime, based on the thread-safe model, avoids user thread delays when opening new displays, handling value updates received from the control system, and improves overall performance for complex widgets like images as well as scripts and rules. The CS-Studio Display Builder is now operational on BL11B (Mandi), where it handles most of the existing BOY displays that are shared with other beam lines without modifications. In addition, it offers plots with error bars, and allows integration of the point-to-center camera-based sample centering application into the control system user interface. Finally, it is our first tool capable of displaying the full-resolution detector histogram (4500 x 1800 pixels) with interactive zooming and panning. Development continues. Deployment and evaluation will continue on additional beamlines prior to more demanding use in the accelerator complex.
Recommendation 15: Many staff have been with the project since construction and have a lot of corporate knowledge. This knowledge will be critical for success of the Proton Power Upgrade (PPU) and Second Target Station (STS) upgrades. These staff may not be around when the projects are most active. Take the opportunity of this ‘gap’ to think about succession planning for key staff.
Resolution: The Neutron Sciences Directorate has an established succession planning activity that is reviewed quarterly. This activity focuses on leadership succession planning (Division Director, Group Leader) as well as identification of high potential staff for development activities such as the laboratory’s Management Boot Camp, Developing Leadership Potential, and Laboratory Operations Leadership Academy (LOLA). The Research Accelerator Division (RAD) has had success in directing promising staff through these programs as needed.
The RAD Division Director, assisted by Human Resources, annually evaluates the demographics of the division’s staff and, in consultation with group leaders, develops succession planning pathways for key staff nearing retirement or areas of vulnerability where particular skills are in jeopardy. RAD management recognizes the necessity to bridge skills needed to ensure the long-term success of PPU and STS. In certain cases, cross-training through functional consolidation and integration of teams can provide the desired outcomes.
RAD has undertaken an aggressive hiring program in key areas (balanced by concomitant reductions in procurements) to ensure that key skill areas are fully staffed before the period of execution of PPU and STS. Recent activities include:
- Consolidation of the Central Utilities and Water Teams to leverage similar skill sets and provide an avenue for succession planning for a key water system engineer.
- Hiring of process controls engineers in anticipation of the retirement of a key lead engineer.
- Hiring of a safety instrumentation engineer in anticipation of a possible retirement and to support PPU/STS.
- Hiring of an accelerator mechanical engineer with a background in neutron scattering instrument design in anticipation of both PPU and STS.
- Hiring of 4 accelerator operators to mitigate turnover and restore full crew staffing.
- Elevation of an early career accelerator physicist to the role of Ring Area Physicist to permit the senior individual to focus on PPU activities.
RAD management will continue to pursue similar strategies to ensure availability of qualified and knowledgeable individuals to support both sustainable high-reliability operation of the SNS complex and the successful execution of the PPU and STS projects.
Recommendation 16: The priority for the effort to upgrade several diagnostic systems should be part of the overall priority scheme for facility improvements.
Resolution: Accelerator Improvement Project (AIP) 37 is the upgrade of SNS Beam Instrumentation. Beam Instrumentation (BI) is an important accelerator system essential for accelerator tuning and safe, reliable operation. Beam Instrumentation devices measure and monitor various parameters of the beam in the accelerator. The main components of BI devices are signal detectors or pick-ups, and data acquisition and processing electronics. The current SNS BI systems were designed in the early 2000s. The technology of the detectors and pick-ups has not changed significantly since then; therefore, the existing devices are still adequate for the current SNS accelerator parameters as well as for the STS upgrade parameters. Electronics technology has changed dramatically, which has made many existing BI electronics obsolete and difficult to keep in functional state. The purpose of this project is to upgrade the major BI system data acquisition and processing electronics to the latest technology standards, while placing an emphasis on Commercial Off the Shelf (COTS) solutions.
Recommendation 17: Keep doing what you are doing on gathering performance metrics, analyzing data and underpinning operations by tackling shorter downtimes.
Resolution: Accelerator Operations continues to track downtime each week during operations. A metrics presentation is generated each week and is discussed with system owners during our weekly metrics meeting. A Machine Health Report is also generated and discussed each week which looks at trends that may indicate a problem in the future. These items are addressed on maintenance days to avoid future downtimes.
Recommendation 18: Keep doing everything possible to tackle long downtime events, primarily focusing on understanding and alleviating target failures.
Resolution: This is addressed in the SNS Target Management Plan 106010000-PN0005, R00. This plan describes the strategy to interleave First Target Station (FTS) target design and engineering analysis, transient response measurements, manufacturing, capability enhancements, and post-irradiation examination (PIE) with the planned SNS operating schedule to best use the existing and anticipated inventory of target module hardware for neutron production at SNS. The key objective is to develop a robust target module that can operate predictably and reliably at a beam power of 1.4 MW and an accumulated energy of up to 3,500 MW-hr.
Recommendations 19 & 21: Continue developing a quantitative prioritization process addressing outage work, AIPs, other large tasks, and the two major project activities. Look into best practices elsewhere. At the next AAC meeting, present output of this process showing transparently how projects met the funding criteria or not.
Resolution: A quantitative prioritization process was developed to address outage work, AIPs and other large tasks, and the two major project activities.
Recommendations 23 & 24: Continue to make the case for an adequate AIP budget. Adopt a transparent grading system to prioritize requests for AIP funds. Prioritization criteria should be decided by SNS management and applied uniformly throughout the organization.
Resolution: A quantitative prioritization process was developed to address AIPs. It was the basis of the process used for all RAD Project prioritization.
Recommendations 25 & 26: We encourage quick dissemination of the planned outage work planning improvements to all impacted personnel. The schedule for doing this is already tight. Get the message out! We encourage improvements to outage work planning proceed immediately and that the summer 2016 outage be used as a beta test of the new process.
Resolution: Priorities for the summer 2016 Planned Maintenance Outage were disseminated on April 29, 2016, and the first published outage schedule was disseminated on May 4, 2016. Priorities for the winter 2017A Planned Maintenance Outage were disseminated on Oct. 21, 2016, and the first published schedule was sent out on Dec. 15, 2016. These documents can be located in the closure evidence file. The beta test of the new process was incorporated into summer 2016 outage planning.
Recommendation 28: Proceed with plasma processing during the 4-month outage, and take advantage of other opportunities in the schedule for plasma processing. Plan strategically to ensure that resources are available for this effort and prioritize cryomodules with the highest potential for improvement.
Resolution: Date changed to 9/30/16 to coordinate with the extended outage. 9/29/2016. Plasma processing for 5 high beta cryomodules is planned for during the IRP outage scheduled to start in December 2017.
Recommendation 30: Consider adding the conventional construction features for a radioactive work cell that could be outfitted later, if necessary.
Resolution: We are considering this option. First, a strong use case should be developed to define any hot-cell needs. Then the possibility of using the FTS hot cell and other robotic manipulator hot cell capabilities at ORNL will be examined. If there is a strong case and other cells are not available, space will be provided by STS. In any case, a review on this subject with outside experts will be held before making this important of a decision.
