FOLIO Project Update – Month of December 2020

The Metadata Management SIG conducted a review of the project Vision/Strategy and Governance document and members of the SIG provided feedback individually. Entity management continues to be a concern (there is a general desire to have development sooner than outlined in vision/strategy document). The SIG is focusing right now on MARC needs in order to inform development priorities around quickMARC, MARCcat, and possible integration with external tools such as MARCedit. They hope to have outline by the end of next week of most critical functionality, priorities for functionality by record type, and preferred/recommended solutions. They already had documentation on the agenda for their meeting and are exploring more “hackathon” type events.

The Resource Management SIG reviewed several features that needed refinement.UXPROD-2373UXPROD-1995UXPROD-1925: they are reviewing the connection between order lines and inventory, including how the “piece” that can be received via the Receiving app is connected to the item record. Inventory continues not to display any information about connections to Acquisitions, and the lack of explicit links between the apps is problematic. UXPROD-961: they are working on the workflow to allow the creation of orders from the instance record (instead of starting with the orders app). UXPROD-1647: they are working through how to maintain relationships to the order if holdings/items are moved in inventory. In general, the SIG thought the POL should be updated to the new instance information but recommended creating a spreadsheet of use cases. UXPROD-2455: they are adding a specific material type to the POL that is independent of the Material Type in the Item record in Inventory. The SIG also discussed the desire to add custom fields to the PO/POL. While that wasn’t seen as the ideal solution for this case, custom fields would give institutions needed flexibility to add local fields as needed without bloating up the PO/POL. UXPROD-2665: they are discussing the mechanisms to export records from search results, as a way of efficient in-app searching. The SIG also discussed the need for a date stamp to the interfaces section of the Organizations app, as user name/password information is often stored there and frequently needs to be updated. Within the ERM Subgroup, Cornell presented a demo of their experiences structure and workflows to date. ERM discussed functionality around searching for Agreements and licenses using both relative and specific dates, permissions and roles for Library staff using ERM apps, searching for and displaying period dates in Agreements, and how results display within an Agreement. The Acquisitions small group covered a number of features, including: updated mock-ups for fund balances to allow for understanding of changes to balances over time, export and display of PO data, and changing locations and quantities on open orders. The acquisitions Small Group also considered performance and what would be acceptable in terms of systems response time.

The User Management SIG reports that the development team in Leipzig continues to work on deletion of user records, both via the API and via the UI. They have been working out what dependencies require that interrupts a record deletion and decided that, if a certain dependency exists, the record will not be deleted. If a person deletes a user record through the UI, and the record has a dependency attached to it, a pop-up box should appear. If a person deletes a user record though the API, and the record has a dependency attached to it, the record should be skipped and the user data should be written to a log file that staff can look at later. The dev team is working on doing a dependency check as part of the deletion process; they expressed a desire for that check to be a procedure that can be run independently of the deletion process. They continue to discuss setting a PIN field separate from the password field.  They don’t have consensus on how PINs are initially set. There is a Discuss post here, if you wish to read/contribute. When users are deleted, “data fragments” are likely to exist in other apps.  They are still discussing whether a procedure for clearing out these data fragments should be made.  I brought up the question in the sig-conveners Slack channel, and I believe app-interaction will discuss this on Friday.

The Resource Access SIG looked at unavailable status: circ actions and the impact of recalls on loans. They reviewed the governance and vision & strategy. They also discussed proxy borrowing interaction with patron block on borrowing, revert loan when recall is cancelled, patron comments in Request reports and NFR questions related to Requests, Locations and Service points (Vasily Gancharov  – the solution architect – is collecting NFR details about FOLIO modules).

The App Interaction SIG reviewed proposed UX patterns on Export Results lists and Results list column display to follow Keyboard shortcut lists and Catastrophic Error message. The SIG discussed requirements and possible steps towards notifications. The SIG also discussed relationships between records, especially Agreements and Inventory, and in that context talked about the vision of the Entity Management App. Having the Entity Management App coordinate the relationships would solve the existing issues. The plan is to develop in individual apps towards Entity Management App which is scheduled for being developed in 3 to 5 years.  The next meeting, Call on Elastic search, was postponed until January. This week there will be a discussion on deletion of user records and “consequences” in other apps.

The Reporting SIG has an important action for all FOLIO participating institutions: please review Reporting SIG-All Report Clusters (51 issues) in JIRA and rank each report cluster for your institution (R1-R5). Explanation: For reporting, institutions only need to rank the UXPROD Report Cluster JIRA issues. All reporting requirements, which are captured in REP-XXX issues, roll up to the UXPROD Report Clusters. Report clusters cover one or more report (REP-XXX issue) requirements. How can FOLIO participating institutions help with FOLIO Reporting? They can contribute an SQL Developer to work with subject matter experts in the Reporting SIG to understand report requirements, then to develop new, open source report queries for the FOLIO community. They can also contribute a Test Data Developer to work with the Reporting SIG to assist the reporting community with curating and generating test data for our development environment so that report writers can test their queries with substantial data sets. The SIG working groups for all functional areas are collaborating on derived table query development for FOLIO Reporting. At this time, 46 derived table queries identified so far have been written, tested, indexed, and posted on the FOLIO Analytics GitHub at https://github.com/folio-org/folio-analytics/tree/master/sql/derived_tables.  There are 5 reporting subgroups working on report development for the FOLIO Analytics report repository: Resource Management, Metadata Management, Electronic Resource Management, Resource Access/User Management, and External Statistics. Report development leads are now meeting weekly with the Reporting Product Owner to review query development progress and issues. The RM Reports Working Group is working on queries for title, item, and subscription count reports as well as invoice, purchase order, and fund reports. The MM Reports Working Group is working on queries for duplicate orders, pick list, title and volume count reports. The ERM Reports Working Group is working on report model development for using data from eHoldings and agreements. The RA/UM Working Group is working with Cate on a solution for showing data on recalls in reports, refactoring existing queries to use derived tables and external statistics. The 4 main functional area subgroups will take on queries for External Statistics reports such as ACRL, ARL, etc. These reports will require running multiple queries to run, but will each be captured under one UXPROD issue. When a data element needed for one or more reports is missing, SIG reporting development groups have been working closely with FOLIO app functional area Product Owners to address the impact of missing data elements on report development. Many thanks to the POs for helping with this! LDP (Library Data Platform) software development: LDP 1.1.6 is available. A bug has been fixed that could prevent a table from being updated after an error occurred in a previous update. https://github.com/library-data-platform/ldp/releases/tag/1.1.6 (or use the 1.1-release branch)

The Consortia SIG is continuing to fill in their open source bibliographic utility vision document with contributions from the ICoLC members. Bib Utility Vision Document

The Accessibility SIG did not meet in December. They are still working on Accessibility Statement. This work and interest from implementing Libraries has highlighted the need for more testing on mobile/touch devices. From Khalilah Gambrell (PO for accessibility): Teams are addressing issues identified by Deque aXe and Mike at UC Boulder. Working on UX pattern for keyboard shortcut list and teams are addressing focus issues with Search and Sort.

The SysOps SIG issue with integration of JIRA and github is very special: The initial question was how to get the source of truth about a release. Versions of modules, schemas, versions of afka, etc. They need to find a mechanism to automatically update the “fixed version” field in JIRA. When you create a release, you should reference the JIRA tickets with the release version of the modules. The github pull request should trigger an update in the “fixed version” field. At the moment, it is being done manually. Anton is taking care of this but needs more dev ops resources to fix this. They also addressed the separation of “Reference Data” and “Custom Data”. Wayne, Tod & Stephen are working on this. Terminology will change to “default data” and “operational data”. The upgrade process will involve some form of reconciliation between the default data of the previous module, the new module and operational data created by the tenant over the course of operation. For more information see https://wiki.folio.org/display/SYSOPS/Upgrades+with+Reference+data and https://discuss.folio.org/t/reference-data-and-upgrades/2858. The group also gave feedback on the Governance Model and Objectives & Initiatives Paper.

The Implementers SIG have seen demos from in-production libraries of live workflows with Inventory and Acquisitions over the past month. These demos will continue in January. These demos and others are being collected and linked on the Implementers’ wiki pages https://wiki.folio.org/display/COHORT2019/FOLIO+Work+flow+demos and Discuss pages https://discuss.folio.org/t/reference-data-and-upgrades/2858.

The Public Library SIG was initially kept to a small group in order to get a sense of how the group might function.  The most recent meeting was focused on reviewing the gap analysis process, identifying potential additional members, and confirming meeting schedules for 2021.

The Forum Facilitators report that the last Forum went really well. Done with Forums for the year, won’t do a year in review session (because we all just want to forget this year 🙂 )

In the Product Council, Harry reported on the results of a review of the Goldenrod release that was focused on why there were so many hotfixes needed. The review group made several recommendations for future releases that should prevent this from happening again. Anton reported on the Honeysuckle bugfest. Each release sees more participation by the community, more cases resolved, and fewer bugs reported. Jason Kovari proposed forming the Entities Management subgroup with a draft charge and vision for an Entities app. He is visiting the relevant SIGs for input on the final charge. The PC bid a fond and reluctant farewell to Cate Boerema and Emma Boettcher, and Cate presented the areas of the project that will need to be covered.  PC members are urged to find people from their institutions to step up. Mike G updated the PC on changes made to the proposed governance structure based on feedback from the community, with hopes that the Community Council can be formed in March and the new PC and TC formed in June/July.

The Product Owners are in the process of conducting a search for Lead PO / PO’s. (Product Owner Responsibility Handover) They need: 1,5 FTE PO from the community. In terms of a Lead PO, there are a couple of candidates but several details need to be worked out. New POs will have a mentor, Ann-Marie Breaux will host sprint reviews, and Lucy will help plan PO meetings. They really need someone for Requests.

In the last month, the Technical Council discussed the new Governance Model. Key observations: this doesn’t seem to address what many feel is the largest issue for sustainability – where will the development resources come from as we build and maintain more and more functionality; it may have a negative effect on attracting developers as it diminishes the role of Technology (TC “reports” to CC). It reinforces the concept that FOLIO is more about community and features than technology. They discussed the progress of the group working on Definition of Reference/Default Data Types. The team is converging – key notion of separating “Reference data” into 2 groups: Operational Data and Default Data. Should have a proposal soon. They reviewed a proposal by Craig McNally on Personal Data Disclosure: Concept is to have a template in Github that provides checkboxes for potential personal data. Require teams to maintain an updated list. Will be part of the Definition of Done… TBD if a mechanism will be created to programmatically verify a review was done. Tech Council approves. The discussed the status of Tech Debt top items including: DEBT-1 (optimistic locking): the platform team has pulled RMB-727 into the current sprint. It’s an implementation ticket for platform support for optimistic locking via a “version” field stored in the entity (record). This support is enabled per table/endpoint.  We expect that for R1 the functionality will be available to experiment with but the actual “prevention” of conflicting updates will not be enabled for any APIs. For selected APIs, however, we are considering enabling “detection” — conflicts will not be prevented but there will be a warning produced in the system log for suspected conflicting updates. QUESTION – how does the “enabled per table/endpoint” manifest itself in API changes. ANSWER – for R1 no changes to APIs – essentially allowing us to detect/log when and how often”collisions” happen. We think (Jakub not here to confirm) that R2 will allow the backend to reject requests, unsure if UI modules will be able to deal with this at this time. Because of the impact of this change it’s being rolled out very slowly and deliberately. DEBT-3: (batch APIs) the platform team has pulled RMB-389 into the current sprint, it’s an implementation ticket for the “delete by query” functionality that is a requirement by the batch delete API. QUESTION – has this moved into “in progress”? There was a discussion recently in Data Import group relating to frustration that this doesn’t exist. They questioned how will this work be prioritized if a PO isn’t there to lobby for it, as it needs to be implemented per module. They think it’s a Capacity Planning Group action and/or a Tech Debt/Tech Council push. (There isn’t capacity in R1 so may be R2 at best). DEBT-5: accessibility work continues. They are trying to get accessibility tests baked into Regression tests. There was a question about how this moves from In Progress to “Done”? Zak will think about this and come back to TC on a recommendation. DEBT-6: The Performance Task Force has been active and is now at a point where they recommend all teams conduct performance testing. Transferring Performance Testing to FOLIO Teams.  The cost of Team’s adopting/implementing this approach has not been estimated. There was a question whether it is still useful for institutions to contribute data sets (e.g. orders, etc.). DEBT-11: Work there continues. A framework exists but some legacy apps haven’t been migrated. DEBT-18:  Elastic Search is underway; Falcon is the team that is working this for R1. They also discussed Transferring Performance Testing to FOLIO Teams determined that they need to review doc again and invite Martin to the next TC meeting.