FOLIO Project Update – Month of November 2020
Project Updates / 12 Nov 2020
Reporting requirements, Accessibility Statement and search implementation in this month's project report.
The Metadata Management SIG recently discussed results display improvements in Inventory, specifically some additional elements to display, the order of the display, and the desired sorting behavior. For example, they would prefer to have the “resource title” display as title, but for the sort to be on the index title (title without initial articles). Out of this discussion came a recommendation to add a new element to the administrative data accordion in the Instance record: sort date. This will be a single, four-digit date (year) mapped from the MARC fixed field (008 positions 7-10). More information here. They made progress with assessing the pros and cons, including development challenges, for the different models proposed to handle representation of bound-with materials. Based on consultations with the Resource Access SIG, they selected a model in which a single item record, representing the circulating unit, may be associated with more than one holdings record. This requires a change to the data model, which currently enforces a one-one relationship from items to holdings. More information here. They spent time considering the desired behavior of the MARC 005 field (date/time stamp of last update), particularly when changes are made via Data Import or quickMARC. The FOLIO model, in which the Inventory Instance and the underlying source data are distinct, requires us to be deliberate about this, particularly if we rely on this data for reporting or for “optimistic locking” in extract/transform/reload transactions. Inspired, by the ERM SIG, they have begun actively collecting topics to inform future demos from implementers. The working group on productivity statistics is making progress, and will have recommendations soon for long-term and short-term solutions. Today’s meeting will be spent focusing on the recent Vision & strategy and Governance model documents specifically from the SIG’s perspective. They anticipate a lot of interest and involvement in the forthcoming developments around Elastic Search.
The MARCcat SIG looked at how FOLIO will attract libraries to the community if there is no internal working MARC cataloging tool and what no full MARC cataloging tool in FOLIO does to the adoptability of FOLIO. Next week, they will talk about what current community libraries are doing in terms of MARC cataloging. We know that some are editing in consortial systems and moving data in/out of FOLIO, and not storing MARC in FOLIO; some are editing in OCLC and importing/exporting, with QM for small updates. To date, only 4 libraries are requiring MARC Holdings, and resources haven’t been allocated to that development work yet. What resources does the community have to help this along? If more resources need to be put into a MARC cataloging tool in FOLIO, what areas get set aside or delayed due to resource constraints?
The Resource Management SIG spent time discussing budget structures, and ways to track allocations and expenditures against a single budget. A new mock-up shows how FOLIO can differentiate between initial allocations, adjustments, encumbered and expended funds, and expended funds. The SIG also enjoyed some lively discussions around structuring the budget, and how to use expense classes and grouping to bring together funds in different ways (format and topical subject area, or groupings by different campuses in a single-tenant system), which grew in part from discussions in Acquisitions Small Group and how information around the budget should be displayed. The main SIG also discussed the connection between POLs and inventory, and how to display the connection back to the POL from Inventory, including what happens if a POL needs to be moved to a different instance. Currently this connection is being made at the Instance level only. The SIG approved more standardization for the display of tags. The Acquisitions Small Group reviewed how to validate funds by Acquisitions Unit, so the wrong fund can’t be applied to a POL, if that fund isn’t supposed to be used by the Acquisitions Unit. The group also discussed error handling during fiscal year roll-over, mock-ups for initial encumbrance, how to display and track invoices that were approved, but then canceled, and exporting results out of acquisitions. The ERM SIG provided several demos: a demo of the usage app by Annika Schröer and a demo of Cornell’s implementation of ERM, by Emma Raub. The group also discussed the triangular connection to the POL and then to Inventory, and recommended bringing in a direct link to the instance and adjusted displays for multi-column lists in the search screens in Agreements.
The Resource Access SIG clarified some JIRA-Issues with the User Management SIG (Fielded Search, Boolean Query Search). The got an update by Patti from the User Management SIG on Field Restrictions. They got an update on Bound-with materials from Laura and Charlotte from the Metadata Management SIG, heard about Item states – Lost item application and the using of statistical codes from Michelle Suranofsky, and got an update on Shortcut keys by Khalilah. Ongoing conversations continued on claimed returned reports, claimed returned: add search, patron note on Request, item statuses: confirm circ behavior for new statuses, fees/fines export spreadsheet, unavailable status in CDL/ETAS environment, notices with multiple fee/fine charges and actions, recalls: notifying patrons when their loaned item has a queue of recalls, and patron block override handling.
In the User Management SIG, Patty brought up that development resources are being allocated, but that UM JIRA tickets have low rankings. She has brought several for discussion, and possibly updating our rankings. The highlights: UXPROD-2148 Users App must support multiple external system IDs. People are going live without it, so it’s being de-prioritized. UXPROD-2489 Make phone number repeatable with type labels. The SIG was concerned about hiding contact info behind a permission. It will be discussed with RA-SIG. They also need to discuss Patron Block actions. University of Leipzig has taken on several development projects: fielded search, e.g., search by barcode, last/first name, username, actions buttons in User record, e.g., Edit, Create request, patron expiration date set by the patron group they’re assigned to, user deletion in API and in UI (still being worked on), UXPROD-1811 implement PIN as a field in the User record separate from the Password field (this has been taken on, but not developed yet). Maura shared the governance model document with members of the SIG.
In the App Interaction SIG, they discussed lack of permissions in other apps. They looked at what is displayed when a user lacks permissions in other apps (use case: A user is on a record he/she has permissions to view. The user clicks on a link that leads to a linked record in another app. In the other app the user has no permissions to view.) The SIG collected use cases and discussed possibilities and decided that they need patterns for different solutions to then look at them with the SIGs. They will start with the following 3 options: display some information after following the link, display an error message after clicking on the link, and hide the accordion in the app the user is starting from. In terms of the ERM navigation/dashboard, ERM users are required to get an overview of the current state of e.g. agreements and anything that might need action or investigation. They discussed requirements and needs across FOLIO, like summary information about upcoming renewals, upcoming expiry dates, notifications. The next meeting will review “results export lists” mockups. Still on the agenda/planned for upcoming meetings: talk with SysOps about requirements around permissions management in a single tenant, discuss display of empty & unmapped fields, orders – Inventory – agreements relationships, and cross-app searching. The Item status subgroup decided on naming dummy items as “intellectual items. There was a discussion in the Resource Access SIG on how to handle items for controlled digital lending, on whether an existing status could be used or whether another status is needed – this group is waiting for decision. The remote storage integration subgroup has a Vision document and is currently discussing configuration settings in the document.
The Reporting SIG has an important note for all FOLIO participating institutions: please review Reporting SIG-All Report Clusters (50 issues) in JIRA and RANK each report cluster for your institution (R1-R5). For reporting, institutions only need to rank the UXPROD Report Cluster JIRA issues. All reporting requirements, which are captured in REP-XXX issues, roll up to the UXPROD Report Clusters. Report clusters cover one or more report (REP-XXX issue) requirements. How can FOLIO participating institutions help with FOLIO Reporting? In the Resource Access functional area, contribute a SME (Subject Matter Expert) to help SQL developers understand report requirements. Their highest need is currently for a Resource Access SME to join our RA/UM Reports Working Group. Contribute an SQL Developer to work with subject matter experts in the Reporting SIG to understand report requirements, then to develop new, open source report queries for the FOLIO community. Contribute a Test Data Developer to work with the Reporting SIG to assist the reporting community with curating and generating test data for our development environment so that report writers can test their queries with substantial data sets. The Reporting SIG working groups for all functional areas are collaborating on derived table query development for FOLIO Reporting. At this time, 40 derived table queries identified so far have been written, tested, indexed, and posted on the FOLIO Analytics GitHub at https://github.com/folio-org/folio-analytics/tree/master/sql/derived_tables. There are 4 reporting subgroups working on report development for the FOLIO Analytics report repository: Resource Management, Metadata Management, Electronic Resource Management, and Resource Access/User Management. Report development leads are now meeting weekly with the Reporting Product Owner to review query development progress and issues. Here is the current focus for each report development group: RM Reports Working Group – title, holdings, and item count reports and invoice, PO, and fund reports; MM Reports Working Group – updating derived tables for naming consistency; ERM Reports Working Group – report model development for using data from eHoldings and agreements; RA/UM Working Group – fees/fines reports. When a data element needed for one or more reports is missing, the Reporting SIG reporting development groups have been working closely with FOLIO app functional area Product Owners to address the impact of missing data elements on report development. Many thanks to the POs for helping with this! Regarding LDP (Library Data Platform) software development, LDP 1.1.3 is available. A new table configuration_entries has been added for interface /configurations/entries provided by module mod-configuration. https://github.com/library-data-platform/ldp/releases/tag/1.1.3 (or use the 1.1-release branch)
The Consortia SIG, has had two joint meetings with members of the International Coalition of Library Consortia. The ICOLC has been having discussions with OCLC about many of the same issues that have led the Consortia SIG to start talking about an open source bib utility. ICOLC members have come to the same place we are in wanting to find a way to develop an alternative. ReShare members have also been participating in the meetings, and they are interested in the project. They are working on developing the user stories to describe what we would want such a system to do.
In the Accessibility SIG, CU Boulder testing is on schedule. They have an Accessibility Statement draft. The first version was reviewed by the Technical Council and they are working on edits and focusing on documenting how FOLIO makes progress with regards to accessibility. Further guidance regarding VPATs is still on the Product Council’s to-do list.
The SysOps SIG has 2020 Q2 Goldenrod upgrade & release notes: Jason Root has posted extensive notes on these. It is important that these are incorporated into the Q3 release notes. Release notes are not GitHub artifacts. It would be nice to have some sort of auto-generated release notes pulled and collated from modules. It would also be nice to have the release notes also on github. It is desired to have integration of github and JIRA – they are supposed to be integrated, and they need to look into this. FOLIO behaves differently in an on-premise deployment vs. in a hosting environment. The Data Migration Group looked at requirements for bulk APIs for migration. Ian Walls has created the JIRA issues and they were ranked by the Product Owners. The SysOps SIG is looking for a new representative for the Support SIG.
The Support SIG continues identification and triage of bugs affecting implemented libraries (Support Dashboard). They would like to work with SIGs to proactively identify bugs and ease implementation for libraries that will be going live. They are recommending that SIGs review existing functionality and document available workflows. They are working with the Implementers on bugs vs features and are working with Product Owners to establish workflow for non-specified P1, and P2 bugs that are affecting implemented libraries. Non-specified bugs are those that reflect feature implementation that was assumed but that was not specified, and in Production does not work as intended. These bugs are not the ‘fault’ of the developers, but do need to be resolved in the current release.
The Implementers Group discussed issues that in-production libraries are facing with search. Several key fields are not indexed and trying to search on them crashes the system. Trying to load too many records at once also crashes it. Cate met with the group about bugs vs. features. They looked at features and how the app/process is supposed to work, as defined by the user stories submitted by the SIGs. The bugs feature doesn’t work as specified in the feature; production libraries are finding things that don’t work as they expect and submit bug reports, but often the problem is that the feature wasn’t defined correctly or completely and does not qualify as a bug for developers. Production libraries didn’t like the idea that these problems would become a new feature that has to start from scratch and be voted on. The end result was that the Product Owner will make the call on whether it’s a bug or feature, and use established tags or coding in JIRA for tracking these. At the latest meeting, Missouri State University and Simmons College gave demos of MARC record editing and described the workarounds they are using while waiting for new or improved features promised in Honeysuckle and Iris.
The Privacy SIG now has five volunteers (could use more!), and the SIG will be restarting.
The Public Library SIG did not meet in October. The next meeting is November 18 at 1pm ET.
The Community Outreach SIG is working on a draft press release for the Honeysuckle release and will share with the PC. They plan to release it the first week of December. They are working on two wiki pages: consolidating all open roles in the community to one page and a jobs page. Giving Tuesday is December 1 and we are going to help promote a fundraising campaign for the Open Library Foundation. Funds raised go to the sustainability of the Foundation and back to the projects. Donors can choose which project to support.
The Forum Facilitators report that a forum being planned for December 2 on circulation rules. It will be followed by a Zoom call for more questions/answers. Planning is starting for a FOLIO end-of-the-year review.
The Product Council recently heard an update on the FOLIO developments with the Shanghai libraries as well as a report from their FOLIO exhibit and conference. Marcia Borensztajn updated the group on her work with documentation. A draft version is available at https://docs.folio.org/ and content is being added. She is refining the site, building the infrastructure and establishing guidelines, working on version control. There is still a lot of work to be done, especially with context-sensitive documentation, localization, and the need for content by the community. The PC spent a great deal of time discussing two major documents: PC vision & strategy: FOLIO Strategic Objectives and Initiatives – October 2020 and a proposed FOLIO governance model: New FOLIO Governance Model – October 2020.
Over the last month the Technical Council met 3 times and discussed the draft Accessibility Statement for FOLIO LSP and provided feedback to that team. They also discussed the Plan on Search implementation. The Plan has three potential phases: Phase 1: Basic functionality and integration with FOLIO, Phase 2: Use ES for log aggregation, Phase 3: ‘advanced’ search functionality for FOLIO. Phase 1 vs Phase 3 – TBD how many search use cases can/should be handled by Phase 1 capabilities. It’s expected that work would include accurate hit counts. They are only concerned with the Phase 1 proposal at this time. There are 3 deliverables in Phase 1: deliverable 1: Connect ElasticSearch with Inventory and SRS so that indexes can be populated; deliverable 2: Allow the ElasticSearch engine to be searched from a FOLIO UI; deliverable 3: Support ElasticSearch as part of CI/CD pipeline and reference environments. There is an open question on whether it’s more efficient to send IDs as Kafka messages with Indexer calling back to Inventory/SRS to get actual records vs sending the data on the Kafka message queue. They will evaluate the approach as work takes place. There is a need to recognize that changing search behavior and capabilities while welcomed, need to be coordinated with SIGs and POs to make sure that expectations continue to be met. It would be very beneficial to have a Product Owner for search to navigate all of the requirements, etc. If Iris is the target release for Phase 1, the resource recommendations are roughly 3 developers, (depending on specific requirements). A possible way to implement Phase 1 without getting mired in all the various requirements is to expose ES as a new “Type” of search in Inventory – a peer to Query Search – e.g. “Full Text Search”. They are looking at whether to consider a non-ElasticSearch option aside from PostgreSQL Full Text Search (which has already been evaluated). Some have experience with Solar Cloud which has its own challenges. Much of this has already been discussed and we feel ES is the best option to explore. In conclusion: the TC are OK with the proposed Phase 1 and its deliverables – noting there is still some concern with the way Kafka is planned to be used, which will be addressed/considered in these first phase. They also discussed the “Spring Way” presentation delivered at the Tech Leads meeting earlier today. Spring Way is a FOLIO-specific term meant to imply a group of technologies, frameworks, tools, etc. that relate to using the Spring Framework, Liquibase, OpenAPI, etc… in pursuit of developing FOLIO modules. It’s clear that this approach has many benefits and is attractive. The main question from Tech Leads meeting was “what does this mean for me/existing modules”. A mixed system (Spring Way* and RMB) will make things more complicated and cause some extra work, some lack of consistency and complexity. One example is documentation that today comes “with” RMB. Release coordination also becomes more complicated. Perhaps the bigger change than the framework is the change from RAML to OpenAPI. The outcomes were: they agree and accept that Spring Way is an acceptable alternative to RMB; the Tech Council recommends that new modules use the Spring Way Framework and OpenAPI instead of RMB and RAML but we should understand that there will be gaps in our processes (documentation, CI/CD) for the first (few?) modules that head down this path and the project will have to work to close those gaps; the Tech Council recommends an opportunistic approach for existing modules: when work needs to be done a migration to the Spring Way will be considered. A more detailed discussion around a planned migration can be had in the future. They reviewed the Proposed FOLIO Governance Model. The organizational structure presented may have a negative effect on attracting developers as it diminishes the role of Technology (TC “reports” to CC). It reinforces the concept that FOLIO is more about community and features than technology. Terms of 1,2 years may impede continuity – they want to consider longer terms. It is unclear how the Tech Council and the Product Council should communicate – do decisions need to go through and/or be approved by CC? Consider the case where TC recommends the project put resources on a Tech Debt Item. Does PC decide where resources are applied or does CC, or does CC have to ‘approve’ PC’s decisions? The decision processes should be fleshed out. This doesn’t seem to address what many feel is the largest issue for sustainability – where will the development resources come from as we build and maintain more and more functionality. The doc should state that all meetings will be open (it’s implied but better to be specific). They would like to discuss this again next week.