The advent of the COVID19 changing the landscape of digital learning is presenting challenges for all of us, especially in the digital learning world. I was no exception and wanted to share with you a recent experience where I was able to use xAPI to provide a solution to a problem that threw some unique challenges at me.
Like a lot of organisations, learning is delivered in many forms and the learning culture is generally very embedded. So, when it comes to change, or in this case a major disruption like COVID19, understandably things can get thrown into a little bit of chaos.
The challenge that was presented to me was to capture what a student was answering in an online assessment, as well as the result. The assessment was part of a face-to-face course that had always been delivered in this manner. The outline was:
- Take the face-to-face assessment, develop and deliver the assessment with scenarios online
- All answers by the students must be captured, if right or wrong
- All results must be captured
- The course must be delivered via the Corporate LMS
- The learner can have 2 attempts based on their score (ie < 90% redo all questions >90% and <100% redo only the questions they got wrong)
- An assessor must be able to view the learner’s results easily
- An assessor must be able export the results in Excel to be able to have a competency conversation in the event the learner does not pass
Now, this may not sound like much of a challenge, but here is what I was up against:
- LMS only supports SCORM
- LMS is very dated
- LMS is unable to apply a second attempt based on score
There are quite number of challenges that need to be overcome here. Notwithstanding the LMS is very dated, it only supports SCORM. As we know, SCORM is very limited on what it can report. It will tell us who did the course, when they did the course and did the learner pass or fail. The LMS also doesn’t allow multiple attempts based on the learners score. These are quite restrictive.
I needed to think well outside the box given the restrictions I had to work with. I coined the project making a Soufflé out of Mud.
I digress. Back to the problem in hand.
I used triggers to also look at the score that the learner obtained and set an appropriate layer based on their score. Here I ran into an issue with Storyline that was unexpected. I added a button with the built-in feature to Reset the Results. With this built-in feature us the ability to reset only the ones that are wrong. That was perfect! Not so fast. What I discovered was If you add two buttons that use the built-in feature of Reset the Results, it will set the SAME parameter on the other button. Solution was to replicate the quiz in another scene with another results page. Because the brief was that the learner must complete the entire assessment again if the score is <= 89%, having another scene and results page would work. If the learner gets >= 90%, they re-try the initial assessment but only answer the questions they got wrong. Phew!
Now I have the data, what’s next?
Whilst initially I thought this was a quick solution, it turned out to be a little more complex than expected.
Initially, as we had not setup a structure for the LRS, I looked at having one Store and simply filter the results by Activity. This to me sounded logical understanding some database structuring.
As logical as it sounds, the wrapper did not allow me to just return activities to be able to filter. What I ended up doing was pulling ALL data back, adding to an array and then filtering. This was not ideal, but at that point I couldn’t think of any other way. Enter the next issue, an LRS will only bring a max of 100 records at a time. Now I had to check the results for a ‘more’ record and run that query. This was starting to get messy, and not to mention slow.
After doing a little more research, I stumbled across a post on Learning Locker a couple of years old that highlighted the issues with the LRS querying and the speed. The solution was to use the MongoDB Aggregation. I’d never used MongoDB before and a bit of trial and error, I was able to get A LOT more control over the data and the speed was phenomenal.
The end product (at this point in time) pulls back data of all the users and allows the Assessor to export the data to Excel. The data can be filtered by Verb and the xAPI statement can also be viewed for the more advanced users.
This is a work in progress and will grow. An idea I got from Devlin Peck during the Elearning Success Summit was that he was capturing comments in a course. Understanding our learner cohort, I knew that filling out an evaluation form at the end of the course was not going to happen, and if it did, it was probably never going to be used. I added a star based ranking layer at the end of each module where a leaner can rate the module and then provide any comments. This was all captured in the LRS using xAPI. I could then use this data to provide an average rating and also capture feedback in the moment, not when they finish.
I’ve learnt a lot out of this challenge. xAPI is very powerful and very flexible, but needs to be presented in a meaningful way for those that are not technical. My future LRS’s will be structured differently. Where large data is to be captured, a separate store per course or even organisations will be setup. I learnt MongoDB and just how fast it is.
It also proved that people don’t know what they want until you give it to them. I’ve been trying to introduce xAPI for 12 months, but it wasn’t until something was built with relevance that xAPI was accepted as a viable solution.
Below is are a few screen shots of the application.
I can’t share the current iteration of the product, but please reach out to me if you’d like to know more.