September 9, 2020
Starting in March of this year the NSRC started planning for a change in the way we provide technical, hands-on training due to the ongoing Covid-19 Pandemic. It was clear that in-person travel to teach hands-on workshops and to provide Direct Engineering Support was not going to be possible for awhile.
To this end NSRC updated our Virtual Training Platform (VTP) to run both in the cloud as well as making it simpler to access remotely for participants of our training programs. Our VTP uses a virtualized network and server setup on a single machine with access via SSH and the Web to allow students to work in a simulated network environment and to configure and instrument the environment using hands-on labs. NSRC has created multiple variations of the platform including architectures for hands-on trainings on:
Network Monitoring and Management
Campus Network Design and Operations
Peering and IXPs
Routing Security (RPKI)
In addition an ELK stack architecture and a larger, virtual Network Operations Center server can be created and attached to any of these topologies.
Our first use of our updated platform took place on July 7th and 8th as NSRC and the Latin American and Caribbean Association of ccTLDs (LACTLD) organized and taught a workshop in Spanish on Network Monitoring and Management to a group of 22 ccTLD operators from Argentina, Bolivia, Chile, Colombia, Costa Rica, the Dominican Republic, Ecuador, Honduras, Nicaragua, Panama, Paraguay and Uruguay. Details for this workshop are available at:
The workshop was supported by Public Internet Registry (PIR), ICANN, LACTLD and NSRC. LACTLD organized the event coordinating participants from the region and providing the online videoconferencing system we used. To support a multi-lingual experience Zoom was used with multiple instances with interpreter services provided by LACTLD during the entire event.
As we planned for this workshop we realized that a different approach was going to be needed. After having organized, taught and participated in hundreds of workshops and events in over 140 countries since 1992 NSRC has gained considerable experience about what works and does not work for practical, technical trainings. Some of the core tenets we have learned include:
- There is nothing better than doing to learn how something works
- Long presentations are hard on participants and retention will quickly drop after 20 or 30 minutes
- Clean and simple slides are easier to understand and retain
- Imagery is worth many words and can make a strong impression
When moving to an online environment all of these points are true, but with some interesting twists, including:
- Mental fatigue sets in more quickly when training via video
- Feedback is reduced, which is hard on both the presenter and the participant
- Working in groups or with partners is challenging
- Assisting participants is slower and can be difficult
Many NSRC workshops are 40 hours in length and taught over a 5-day period. This is no longer feasible online. Generally speaking NSRC attempts to provide approximately 60% of allotted workshop time to hands-on labs. This means in a 40 hour workshop participants can expect around 15 hours of presentations in a week and 25 hours of labs. Trying to do this online is difficult. In response to this NSRC has been experimenting with several formats for teaching. After this event and others, we have solicited feedback from participants about what does and does not work. Here is the schedule we used for this event.
LACTLD Online Event Schedule
|Session 1:||50 minutes|
|Session 2:||60 minutes|
|Session 3:||50 minutes|
|Session 4:||60 minutes|
One of the issues with an online event is what timezone to use? If the event is regional, such as this one, then finding a time slot that works across multiple timezones is necessary. If an event is for a single location, then this is much simpler, but a secondary issue is what timezone the instructors live in. For instance, for this workshop we had instructors who were living in Bolivia, Canada and Chile while LACTLD staff participating were in Uruguay. With this in mind our schedule looked like this:
Workshop Schedule by Timezone
|UTC-3:||12:00 to 14:00 & 15:00 to 17:00|
|UTC-4:||11:00 to 13:00 & 14:00 to 16:00|
|UTC-5:||10:00 to 12:00 & 13:00 to 15:00|
|UTC-6:||09:00 to 11:00 & 12:00 to 14:00|
During the sessions we included presentations, question and answer sessions, surveys and about 50% of the time involved breaking out in to individual Zoom breakout rooms where participants could do hands-on labs using our online virtual training platform. In the Zoom breakout rooms we had instructors roam and make themselves available to participants if they had questions or needed some hands-on help. In a few instances instructors were able to share control of participant desktops using Zoom to assist with labs or any other questions where this type of intervention was useful.
Since this workshop in July we have participated in additional events and will be participating in several more over the next few months. For full week events a different format is being used. This involves a 2 to 3 hour session on the mornings of (for instance) Monday, Wednesday and Friday. These sessions include breaks, question and answer, surveys, etc. Then, between the sessions students are assigned hands-on labs that they can do at their leisure.
Instructors are available via alternative communication channels, such as WhatsApp. Students can ask questions or, even, ask for a screen sharing session (via Zoom, Jitsi or other tools) to obtain help if they are having problems completing an assigned lab.
At the start of the next session a review of the labs is done. In addition, NSRC has written back-end tools to see how students are progressing and can proactively reach out to participants who look like they may need more help.
An additional twist on this is to provide some pre-determined times when an instructor or instructors are available via Zoom or other video conferencing software for students who may have questions and prefer a fixed schedule when assistance is available.
In order to make our virtual training platform work seamlessly in the cloud additional effort was required by NSRC personnel. For instance, during the LACTLD event in July the NSRC Network Monitoring and Management version of the VTP platform was built running on an Amazon cloud server in a data center in Sao Paulo.
Students were provided with their own individual Linux instance (Ubuntu 18.04) using LXD containers. For some exercises students worked in groups to configure heavier weight software packages on a larger Linux VM instance.
The platform that we built included:
- 1 x larger KVM virtual machine for our Network Operations Center (NOC)
- 6 x KVM virtual machines for heavier weight software packages
- 36 x LXD containers spread across the 6 KVM servers
- 2 x transit routers
- 6 x border routers
- 6 x core L2/L3 devices
- 12 x building L2/L3 devices
For a total of 26 network devices, 7 VMs and 36 containers
Generally speaking this platform requires a quad-core intel i7 or better processor, 32-64G of RAM and around 200G of disk space. The platform runs using the GNS3 virtual network platform, KVM with LXD containers and Linux bridge utilities configured appropriately. We provide direct console access to the student machines and to each network device via a web interface. Some students who are used to using SSH with jump hosts opt to connect to a jump host we provide on our platform as well. Otherwise we use a tool called shellinabox to provide web-based ssh access to the resources (servers and network devices) running on our virtual training platform that runs on a cloud-based server.
During the LACTLD event we provided a fully configured central Network Operations Center server with the network monitoring and management tools taught in the workshop pre-configured to monitor the entire virtual training platform. This NOC instance connects to our virtualized network environment using available Linux network bridging tools.
NOC Connecting to Virtual Network Environment
A workshop web site is provided under our https://nsrc.org/workshops/ hierarchy so that all materials, diagrams, labs, images and more are maintained indefinitely for participants and the general public to access. We use a tool called Syncthing to allow instructors to update the workshop web site on local laptops and have changes propagate within a short timeframe to anyone working on the workshop web site.
We run a local copy of the workshop web site during workshop on our virtual training platform. This can improve access time for participants nearer to our virtual platform.
In order to speed up the use of our training materials we have created a materials development platform that allows us to create a customized workshop agenda (or use ones we already have) with links pre-populated with presentations and labs. You can see some of our pre-populated workshop agendas with descriptions by going to https://nsrc.org/activities/outlines/. These materials are automatically updated anytime a member of our team updates presentations, labs, images, etc. in our centralized Git repositories where we coordinate materials development.
In order for our virtual training platform to be effective we wanted to run it on a server that was physically near where our participants were located. The NSRC servers are located in Eugene, Oregon at the University of Oregon on the United States West Coast. For the LACTLD Network Monitoring and Management workshop our participants were from South and Central America and the Caribbean. We did testing from numerous locations in countries in the region and determined that using hardware in Sao Paulo data centers made more sense. Latency times were generally 50 to 100 milliseconds to Sao Paulo vs. 100-200+ milliseconds to the West Coast of the United States.
NSRC personnel tested cloud solutions from Amazon, Google and Microsoft. In the end we chose to use a physical server instance run by Amazon (AWS) in their Sao Paulo data center. We needed physical access as hardware acceleration for running nested virtualization (KVM inside of KVM) for our virtual solution was not available to us using Ubuntu 18.04 on any of the platforms. The smallest Intel-based physical platform available to us from Amazon was:
This was more hardware than we needed, but to be honest everyone enjoyed how seamlessly the platform ran on a server of this size. Normally we utilize close to 100% CPU in our workshops, but in this case we never saw more than around 5-10% total CPU utilization.
The AWS environment worked very well for us and our participants reported no issues with latency or performance.
While building our platform we were able to do proof-of-concept tests using nested virtualization (KVM running inside of KVM) on the following AWS hardware:
|c5.4xlarge||16||32||Up to 10|
On these instances performance was so slow that even ping would fail for network devices as soon as we put the environment under any load. Still, this allowed us to prove that our build processes worked, verify connectivity and access to the platform at a much lower cost.
Going through this process has taught us a lot about the various cloud solutions available and how we can use them, where physical machines are available for future workshops (we have already taught a DNSSEC workshop using AWS hardware in South Africa) and how much these solutions will cost. The good news - even the large metal servers are not all that expensive to run for a week or two.
After the LACTLD workshop was completed and the participants filled out surveys one of the more telling comments we saw from multiple participants was a request that we provide time to let the students introduce themselves to each other and the instructors. This is something we have always done when teaching in person, but we had skipped due to time constraints with just the instructors introducing themselves. Future workshops will include this as part of the schedule. While technical, hands-on participation is critical to learning how to build networks, NSRC has found that the human relationships built during events are just as important, if not more so, than the technical knowledge gained.
Working virtually challenges our ability to teach as much material as we have done previously and to build the social bonds that in-person events provide. Learning how to do this with remote tools and new paradigms has become part of our mission in 2020. While we know that in-person events will return the knowledge we have gained and will continue to gain about how to use remote tools to teach and build social and physical networks will be invaluable and enhance what we and others can do going forward.