We are excited to see this work continue to grow!
Selecting software for any large organization is an involved process. So, when it came to selecting an Electronic Lab Notebook (ELN) for HMS research community, I knew that the wide array of use-cases and the large number of ELNs on the market that would have to be evaluated made for a real challenge. Could we find a single ELN that we could support for the whole community? Would it be better to find 2 or 3 products? Or would we find that no ELN products could adequately support the workflows of some HMS research?
HMS labs have been reaching out to Research Computing (RC) for advice on which ELN products to use for a few years now. Starting in September 2019, a Blavatnik funded Discovery project, co-sponsored by Bill Barnett and Caroline Shamu, enabled us to reach out to many other HMS community members, and take the necessary time to do a full evaluation. We were able to dig deeper into both HMS requirements and the most promising ELN products on the market.
We had some technical requirements from IT, but understanding the needs of researchers was critical to the success of this discovery process. Through individual interviews, pizza lunch discussions, and other networking, we gathered opinions, ideas, and requirements from a broad spectrum of the HMS community. We learned that many labs were already using some kind of electronic tool as an ELN with varying degrees of success. Some were using general purpose note taking tools such as OneNote or Evernote, while others were using specialist tools such as Benchling that provided some ELN functionality as part of their Molecular Biology toolkit. Still others had invested the time and effort to build their own ELNs to facilitate their specific workflows. As we expected, there was a diverse array of use-cases at HMS, but we were also hearing a core set of very similar needs. One surprising take-away was that many labs and cores wanted an integrated inventory management system as part of the ELN.
Next came arguably the most important part of the discovery process. We needed to find some research partners to work with. One key partnership was with the Corey lab, and particularly Domenica Karavitaki. Domenica and I first met at a Countway Library ELN seminar run by Julie Goldman (I recommend that you check out one of these excellent seminars if you would like to learn more about ELNs). I learned that Domenica was interested in finding an ELN to serve the diverse nature of data collected in the lab. A group of us from RC (Mason Miranda, Kathleen Keating, and I), along with members of 4 different Neurobiology labs (Born lab, Corey lab, Kaeser lab, and the Goodrich lab) crammed into the Neurobiology department’s lunchroom for a fruitful discussion. It emerged that the Corey lab’s requirements were fairly representative of other Neurobiology HMS labs, and some of the other labs that we had met from other departments. They were also keen to try to integrate inventories with protocols so that they could streamline their workflows. Moreover, David Corey saw the value of ELNs and agreed to support the time and effort spent by his lab members on this effort.
We set about finding the 6 most promising products. Concerned mostly with the Corey lab’s requirements, but also, IT’s technical requirements, and some other key requirements that we had gathered along the way. Using the ELN matrix (a great resource developed by RC and the RDMWG in 2018) as a starting point, along with some more recent intelligence, we agreed on a list, and spent the next 2 months attending product demos, collating notes, and updating our requirements matrix, on the way to arriving a shorter list of products that we would investigate further.
The process we used for the demos was particularly successful, and well worth emulating in the future. In addition to the Corey lab, invitations went out to various interested researchers from other labs, faculty members, the RC project team, and members of the RDMWG ELN subgroup. We setup a weekly cadence where we had demos every Tuesday, the project team spent the rest of the week collating our notes, then the following Monday the we met with Domenica and Andy Ward (also from the Corey lab) to go through those notes and discuss any important findings. Over time, the regular demo attendees became better informed and were able to better understand the value that each product brought to the table. Importantly, knowledge gathered during and after each demo allowed us to probe into the more difficult questions, and recognize gaps in each product. I’ll mention Dan Wainstock here, from the RDMWG ELN subgroup, who had a particular knack of asking those difficult questions. This became a key part of the evaluation process as the gaps had to be measured against the needs that the ELN would cover. Once this was accomplished it was easy to further narrow down to the products of interest.
From the original list of 6 products, we landed on 2 that we wanted to investigate further, RSpace, and eLabJournal. During a 90-day Proof of Concept (PoC) users from the Corey lab, in addition to other groups from BCMP (Blacklow lab and Adelman lab), and the Wyss Institute (Immuno-Materials Platform), were given access to trial versions of these products. Users could create protocols (or templates), and document experiments including attached data files, analysis, annotated images and more. We asked users to test each of the products using old experiments. Everything needed to be securely saved, with every change tracked for compliance purposes, and to prove provenance. It was also important to all users that they could organize their data, collaborate with other researchers, and search through experiments. Andy Ward from the Corey lab and Emily Stoler from the Wyss Institute was among the users particularly interested in the inventory functionality. By integrating protocols and experiments with inventories of reagents and samples, researchers could work more efficiently, keeping track of samples, and knowing when to make orders. The inventory system also allows users to keep track of equipment, or even mice!
“We have been searching for an ELN to manage the diverse storage and security requirements of data collected in the lab and our extensive inventory of equipment and reagents, and to aid the efficient management of multi-collaborator projects in our group. After attending the ELN seminar run by Julie Goldman and meeting with Joanna Loveluck, it was evident that the time was right to proceed. The valuable support from Joanna and the Research Computing (RC) team turned what initially seemed like a daunting task, within the limited time allowed from research requirements, to an efficient evaluation platform for a suitable product. With the constant exchange of ideas between our lab, especially my colleague Andy Ward, and the RC team, we successfully arrived to the selection of the ELN that will best serve our lab's needs and, we hope, the needs of many HMS groups.” — Domenica Karavitaki, Research Associate, Department of Neurobiology, Corey Lab
While researchers were testing the products’ functionality and evaluating usability, James Weatherell led the RC IT team (Safet Berberi and Miles Macchiaroli) in a technical evaluation, looking at the overall system architecture, security and authentication, vendor lock-in protection, and much more. We also reached out to Joe Zurba for a confidential vendor security analysis.
At the end of the PoC, we collected feedback from users through a Qualtrics survey developed by Kathleen Keating. Both RSpace and eLabJournal rated well, but eLabJournal rated higher on most questions, particularly inventory support, protocols support, and vendor supplied support and training. It should be noted that document organization was split, with some users preferring the less structured document management found in RSpace. From the technical viewpoint, while there were positives to note for both products, eLabJournal was also preferred.
Now that we have our preferred product, the next step is to Pilot eLabJournal. Users will be documenting real experiments with the knowledge that their data with be secure. This is very exciting for all of us who have travelled the long journey to get to this point. Many members of the HMS research community have been involved in the discovery process, that has allowed us to discover a versatile ELN product backed by a very supportive vendor (Bio-ITech/Eppendorf AG). We have also gained knowledgeable about ELNs, and will be better able to help researchers that need a different solution. Most importantly, in the Corey lab and others, we have found some very committed labs to partner with, who will work with us to define the emerging HMS ELN service.
Written by Joanna Loveluck, Project Manager, Research Computing, Harvard Medical School Information Technology