Tech Training Workshops in a Pandemic World

January 27, 2022

Training in a Hybrid Teaching Environment at mnNOG3

During 2021 NSRC participated in dozens of network training and Internet development activities with a couple thousand network engineers from all regions of the world. All of these activities have been done remotely by NSRC staff and contractors, but in some cases these events have involved a mix of local on-the-ground participation with remote assistance.

From Monday through Friday of October 25-29, 2021 a team of instructors and assistants located in time zones around the globe and in Ulaanbaatar, Mongolia taught a workshop on virtualization technology as part of the 2021 mnNOG3 event. During the week a group of 20 students and 4 local assistants located at the Information Technology Park in Ulaanbaatar worked with a group of NSRC staff and contractors based in the United States, the United Kingdom, Denmark, Uganda, Bangladesh, and Australia to deliver hands-on training about how to build an open source-based virtualization platform with redundant data store capabilities.

You can read much more about the details of this event at

https://nsrc.org/workshops/2021/mnnog3/

Gonchig Altansukh, who was part of the local instructor team noted:

We held a half-day online conference last year 2020 because of covid-19, and this year's goal was to have a workshop anyway, at a time when the impact of the coronavirus in Mongolia has weakened considerably. We conducted a survey among engineers, and most of the engineers were very interested in the topic of virtualization. That's why we decided to boldly choose the topic of virtualization and organize training.

Remote Instructors and local participants all joined in the mnNOG3 Closing Ceremony activities

What do you mean by Hybrid Teaching Environment?

In the past, NSRC staff, contractors and local network engineers worked together, in-person at physical venues to teach and train on technical topics. During the Covid-19 Pandemic the NSRC enhanced our Virtual Training Platform (VTP) with updated software, topologies, and a user interface to make utilizing our training kit simpler. The VTP now can run in the cloud (for instance, using Amazon Web Services or Google Cloud Platform facilities) or via rack-based machines we deploy in several strategic locations. We have taught many completely remote workshops since the Pandemic started and we have taught several workshops in which participants are able to go to a physical location in their country, work with local engineers while we lecture and assist with technical lab exercises remotely. The latter situation is a hybrid environment.

During the mnNOG3 workshop we did not use our virtual training platform, but rather had the local instructor team build the entire virtualization environment by hand before the workshop began. This involved numerous hours of coordinated Zoom calls between NSRC contractors and local instructors to finalize the build of the environment. The topic of virtualization can be done using our virtualized environment, but it is already so abstract that building a virtual environment on top of a virtual environment, we feel, makes teaching the topic more difficult. By having physical machines that create virtual compute clusters with shared file systems the participants can understand more viscerally what is being done and what the advantages are to virtualizing servers and file systems vs. using a single physical machine per server.

How hard was this?

Tugsorshikh Badarch, one of the local organizers for the mnNOG3 event had this to say about preparing for the virtualization workshop:

Although we were aware of the huge, day-with-night, time difference, mnNOG PC team asked NSRC to conduct hybrid mode training for the best benefit of the local participants, to let them learn from NSRC teachers' expertise and to maintain the quality standard of the mnNOG workshops.

The hardship/sacrifice was far greater than we expected due to the complex lab setup for the Virtualization Workshop. NSRC team and local instructors worked together throughout the whole process of designing, setting up and verifying the lab and also, assisting the participants during the lab sessions. This required NSRC instructors being present remotely around the clock for weeks. Frankly, I was amazed with their commitment for local development and patience and also found out that my initial thought/estimation of the workload on them was nothing compared to reality.

After completion of the very successful workshop, mnNOG3 PC and Coordination teams were so grateful to and proud of all our Workshop instructors. Due to the restrictions and cautions we registered only 20 participants in the Virtualization workshop and many more wanted to attend, but had to wait for the next round of the workshop.

Gonchig Altansukh noted:

We had a hard time finding what we needed for the lab, and after everyone worked hard and asked from many places, we found physical devices & its resources. Before the training, we had to prepare the lab environment according to the training requirements, and we had to understand it very well. We didn't have enough time to try to build the lab two or three times as our instructors suggested. We only set up a lab once and had a lot of problems when we were preparing the lab. For example, I worked with Hervey and Hans together all night to find a problem, but it didn't work out. After a day I came back and replaced the router with a new one. It solved the problem. Devices collected from many different organizations were also a problem.

What worked?

Quite a bit, really – but it required a lot of coordination and, at least, one person to be available for almost all the sessions. NSRC staff and contractors worked very closely with network engineers in Ulaanbaatar from System Center LLC, the National Civil Aviation Center and Fibo Cloud to build the initial, physical training platform.

The local instructor team and the local organizers prepared the workshop classroom with excellent video conferencing capabilities. One of the most important items that was added was a panning camera that allowed remote instructors to see the entire classroom. This is and was invaluable as it let us know what was happening in the room while teaching and, more importantly, during lab sessions.

Assisting participants remotely

Instructors and student view (upper-left and right) during the mnNOG3 Virtualization Workshop

One of the hardest challenges teaching remotely is to be able to assist with the hands-on labs. We have experimented with numerous platforms, but what has worked the best is when local, technical personnel are available at a venue and can assist participants. If a problem is encountered that needs troubleshooting working with the local person, the participant and our team via desktop sharing, chat, video, and voice can be quite effective, but we often find that the critical piece is someone local to facilitate all these modalities of assistance.

For other remote workshops we have begun to use Discord. We’ll talk about this a bit more in a moment.

What was different and cool?

For this workshop the local team had set up a system whereby we (the remote instructors) could see the entire classroom. A set of cameras was used that allowed us to view students as they came and went, when they asked questions, and to view the class as we were presenting. This was extremely useful. When presenting to a wall of named tiles it can be hard to maintain enthusiasm and concentration at times but being able to see people as we presented was incredibly useful.

During the course we had a team of local instructors who were all versed in the topic that was being taught and who were admins of both Linux machines and networks at their own organizations. As so much of the content taught required an understanding of these topics, having local instructors who could guide the students and answer their questions about both class topics and more generally about how to use Linux, how to use the platform, and so forth was invaluable. The local instructor team had practiced ahead of time by following the workshop presentations and labs and installing and configuring servers and software before the course had even begun. All of this made the course go smoothly.

mnNOG3 Virtualization Workshop class photo

For our class photo (see above) the local team put up the instructor team on a large display, then a picture of the rest of the class was taken with everyone around the display. This was ingenious and really made us (the remote team) feel included.

During the closing ceremony the local instructor team and another team teaching another workshop were included on a large monitor and cameras were put in place so that we could see ourselves on the screen as well as see the entire group of participants for the entire event in a single room. There were three camera angles and during the closing ceremonies the local speakers addressed the remote instructor team directly on several occasions and each of us had the opportunity to share closing remarks. As the ceremony wound down, the local participants had a small party and the instructor team was able to stay present while groups of participants came up to the cameras and wished us well – this was incredibly touching and really made us feel like a part of the event, even though it was all happening far away from us.

What was harder?

Assisting students remotely is still difficult. The best interactions took place when a local instructor would drive a session with a participant and share their screen with us while we helped to diagnose issues. But, for most of the workshop the local instructor team would tell us what needed to be looked at via a separate chat channel we had configured, or occasionally by talking to us directly on video, we would review the issue on our own and report back to the local team. In some cases, we assisted the local team remotely by verifying if exercises were completed and if there were issues with specific machines or groups and what needed to be done. This turned out to be very effective, mainly because the remote team was both able to quickly follow-up on what we found and assist participants as needed, and were adept at communicating with us via chat or video.

Even though we had cameras that let us see the classroom we would suggest placing even more cameras for viewing the participants. Placing a camera in the back of the classroom would be very useful for the remote team to be able to see students and their work as we normally do when teaching in person. Basically it reminds us of being able to look over someone’s shoulder as they work on a problem.

Some of the participants complained that trying to ask questions or speak with the instructor team via the cameras and microphones mounted on the larger monitors was difficult and having to do this with masks on was even harder. A few said it was like yelling all the time and they found this unpleasant. Providing either a roving microphone or two, or, perhaps, using local machines for voice might improve the situation.

And Gonchig Altansukh notes:

The first day of training was a nightmare 😅. I was sweating and didn't know what to do when our internal network got packet loss even though we just set up the router a day ago. After the first day of training, we also troubleshooted the lab environment with NSRC instructors, and we stabilized the network by finding a problematic device from the lab devices. We set up the two large displays to separate networks, assuming that the remote instructors should be uninterrupted if something happens in the lab environment. Since then, there have been no device issues at the lab.

Students in class during the mnNOG3 Virtualization Workshop

What might we do differently next time?

For this class the environment worked as well as we have seen. Now that we have experience with an environment that is properly set up, as previously mentioned, we would suggest adding a camera, or more, to the back of the classroom, improving microphone placement or ability to ask questions via voice and consider an alternate platform than Zoom for actual desktop troubleshooting during the lab sessions. We have already been testing several options and, so far, we have found Discord to work quite well when it comes to interactive sessions. The environment was designed for gaming, and it shows. The ability to quickly form groups where video, voice, screens, and chat can all be shared is powerful. We have found that it can be chaotic, but many of our participants seem to understand how to navigate the interface quickly and after two completely remote workshops where we used Discord, we polled our participants and found a significant majority indicated they preferred using it over Zoom for lab environments.

Our local instructor, Boldbayar Jantsan, had several ideas about what could be done differently and whether, in the end, the workshop was a success:

The whole hybrid session was great overall.

The most challenging part was to build a lab environment. Here are my thoughts when I was working as a local instructor.

We should have a more sophisticated microphone in the classroom. Sometimes remote instructors hardly hear attendees' questions. In addition, chatting Q&A solution should be used with voice Q&A solution simultaneously for those people who can not speak English well, but can listen and understand presentation very well. :D

Secondly, we should have a silent classroom. When we open the windows, outside noise easily interferes with the remote instructor's voice. There was no AC in the classroom, closed windows, and the room temperature made attendees sleepy. :))

Local and remote instructors should have synced their information as soon as possible. Sometimes labs and presentations links on the workshop's page were changed in the morning before the classroom starts.

Anyways, the workshop was going so well. Contents and topics were so good. Attendees were very satisfied due to feedback. I am very happy to be one part of this great workshop.

What is the single most important thing that could be done to improve remote trainings?

Preparation. For those who are on-the-ground it can be hard to fully appreciate how much time it can take to build or understand a training environment. In addition, running through all the exercises in a workshop before class starts is mission critical to ensuring that surprises do not happen while teaching. The local team in Mongolia had done this and it really helped to make the event run smoothly. Software and software configurations are constantly changing. Different hardware behaves differently. No matter how much time you prepare trying to make sure that all labs will work flawlessly, something will be different or unexpected. By installing and configuring ahead of time, and then actually doing all the workshop labs before the first day of the event things run much more smoothly. When we teach a completely remote workshop on our own virtual training platform that we control 100% we still do this before each workshop. It’s one of the ways we refine our materials, but it, also, helps to keep the surprises to a minimum during class!

Is this sustainable over time?

The honest answer is yes and no. Providing remote training in both hybrid and fully remote environments is definitely possible, but there is no way around the fact that it’s harder to do vs. being there in person. The hands-on lab components and working through as much material as we would normally do is not possible in the same amount of time in a remote or a hybrid environment. In a hybrid environment we found that it was possible to get through most of our materials, but the cost to the remote instructor team is very high. Cognitively teaching via video is more draining than teaching in person. Often the remote instructor is in a different time zone so they may be teaching very late at night or very early in the day.

For this hybrid event we (NSRC) worked with seven members of our team over the course of four days. This was largely due to time zone differences. For an in-person class we would normally teach this course using two or, at most, three instructors and we would have taught over five days instead of four. This event required a much larger amount of coordination and planning to make sure the workshop ran smoothly. In addition, one NSRC team member stayed online for the entire four days of the workshop to ensure continuity between the different instructors and to deal with any unforeseen issues that came up.

Gonchig Altansukh states:

Our training went very well except the first day :), but I don't know how it felt for the remote teachers and just knew it's too late for the instructors. I thought the training went smoothly, including all of covid's recommendations and we followed the rules. If we could have found more resources there would have been more participants who could have participated.

mnNOG3 Virtualization Workshop local instructor team

Our improved virtual training platform that can be accessed via the web has made remote teaching much simpler. We continue to refine this platform and we are already working on additional enhancements for 2022.

We are quite excited by the idea that at some point we will be teaching and learning in technical workshops in-person once again. We miss the ability to connect with our class and to be able to talk through solutions and labs in more detail, but one of the clear wins for our organization and other groups like us is our new-found experience teaching remotely. While it may not be ideal, it is clearly a powerful tool and we now have the hard-won experience about what works, what does not and how we can leverage our abilities by using these tools where appropriate. It is clear that training remotely on specific topics can be done and (in many cases) can be done at a lower cost, with less preparation and more quickly. We look forward to training again in person, but hybrid events like mnNOG3 and fully remote events have shown us how we can be more effective going forward with the resources that we have while still building and maintaining a healthy community with Internet network operators.

And, as Gonchig Altansukh told us:

A guy asked me, why are you doing this job without getting paid? I said if you know something whenever you share your knowledge with others it always makes you even more knowledgeable. We believe that our beliefs are united on a voluntary basis, so we think we have been able to organize the training very successfully.

Closing Ceremonies for mnNOG3

National Technology Park, Ulaanbaatar, Mongolia (Image courtesy Tugsorshikh Badarch)

mnNOG 3 event details at https://2021.nog.mn/