Wiki: Advanced ccTLD Workshop Planning: Planning Notes

Notes are organized last-first.

ccTLD Advanced workshop model suggested by Jaap Akkerhuis, 30.5.2008

Advanced ccTLD workshop

Jaaps Model - Draft - 1 June 2008

Introduction

Problems with the Current Outline: It is a nice bag of subjects but it doesn't give broad picture. What we really need is to give it a some body.

Starting with crypto stuff and only tell in day three what how you build out an will likely confuse people. It is too easy to loose one self (as audience and teacher) in details. (And admitted, these are interesting on its own).

Therefore I propose to start with an Introductory 1/2 day which will aim to

	(1) Define the role of a registry in the framework of the whole
	    Internet;

	(2) Give some definition of what a registry is.

Both these two points will hopefully lead to

	(3) Get everybody at the same footing (most of the people did an
	    earlier course but I assume that not everyone got the same
	    out of it.);

	(4) Give the base for some running story line for the next
	    days and give frame work we can refer to. It probably might
	    make it easier to present country presentations

Referring to this framework the following days we can hopefully keep some coherency and also give an insight in why we present such a plethora of subjects and how they all come together.

All by all, I will first expand how such material might look like. I use the pictures from Daniels slides he used for Nairobi.

Figure 1 "Functions of a Registry" tells how the data streams goes in and out the registry and how the whole DNS is hierarchical. A registry is just part in the chain (see also Figure 2, "Second Level Domains"). Apart from the technical parts, it is also political. The ICANN Accountability Framework is also based on this picture(*) <http://www.icann.org/announcements/announcement-12feb06.htm>, so this might serve as an illustration how a registry relates to ICANN. So this is a hook to get policy discussions and tutorials by/over ICANN & IANA etc. into the program.

As usual, In Figure 3, "Variation on the Model", one can also talk about things like thin & thick registries, relations between registries above and below (and registrars as well).

But the real bulk about what things will change when a registry grows is to be seen in Figure 4, "Functions inside a Registry".

The model will still hold, but in a small registry most of the functions will often be shared by the whole staff. With the expansion there will be often some specialization take place. Also, one can hardly prevent extra functions to come into existing. Things like that might be a financial, human resources and similar departments. I will leave these out this discussion but one should be considered. Dealing with ICANN, local politics and government, local Internet community might warrant its own box. When the organization grows, not everybody will have knowledge or be interested in all these (often non-technical) details so it is wise make this an explicit fiction as well. So this is where the policy arrows from the earlier figures are going to and from.

The other boxes itself might become further divided.

| Box "DNS Service"

To be split in just the "publication" service but also "quality control/monitoring" of DNS.

	DNS Publication:

		Tools related to this
			DNS servers (BIND, NSD, PowerDNS)
			AXFR, rsync etc.

	Quality Control:
		DNSSEC
		Pre-publication zone file testing
			(scripting tools, Perl NET::DNS etc.)

	Monitoring:
		Discussion about what to monitor
			(system level; name server level; name service
				level. As outlined in the NLnet Labs
				report about SE and NL)

		Tools how to monitor stuff:
			System resources:
				SNMP
					system load
					disk usage etc.
				NAGIOS
				Syslog
				MRTG/RDDTool
					munin
			Name server
				Smokeping 
					(Don't use ping but look at DNS
					data)
				MRTG
				Syslog + scripting
				Stats from name server
				Network use (wireshark ...)
				Name server checkers

			Name service "The global picture"
				DNSMON
				Various for hire services
					(autonomica and others, need to
					make an inventory of what is
					available)

With this "quality control/monitoring" discussed, it is easier then to talk about outsourcing DNS service and defining SLA's.

| Box "Boxes"

I also call this box the "substrate". What is here meant originally is the shared wisdom of the registry to maintain the system, write scripts in various languages, program, using a solder iron and similar stuff. It might actually grow into a separate department which is actually split off in to workstation maintenance, database administration, network engineering, group for billing support (COBOL hackers :-)), development group etc. etc. In short, the IT department.

How such a department is organized is out of scope for this course. It might be a full blown IT- organization which provides people and resources for other groups inside the registry to other just a project manager which hires in outside help or completely out source development of registry applications. In the latter case a real on-hands in depth experience of the tools we will talk about is not required but some is useful. In the first case, in is of course a necessity.

| "Registry Data Store"

The registry data store will likely grow out to a full blown data base. It might be based on classical databases (Oracle, MySql, LDAP etc.) and could have multiple functions: Storing registry data, data from a ticket system, billing information. It might be beneficial to store various data in one database or have a couple of dedicated databases which are interconnected so they can exchange information. Various strategies have pros and con's.

Also the function of a database might play a role (from simple data store on a spread sheet helping the application to were application is actually build in the database (I used to do a talk about that).

The way various registry systems (Registro, CoCCA etc) interfaces with the data store(s) plays a role in the choice.

With a growing registry, this part is likely to require staff with more knowledge on database administration then on DNS.

| Box "Whois Service"

There actually won't change a lot. When the registry does i18n, this will have effect with on this.

| Box "Registration Process"

When the registry grows, this might actually become more then just a registration process. A better is probably "Customer Care" of something similar like that.

The different functions in this block one can define:

	Registration process:
		Use of EPP or similar tools
		Part of the process might be a name server delegation checker
		Written with the aid of:
			Perl, Ruby, Python etc.
				(Note: There are a lot of bad
				implementations around, and some good
				ones. Also, there are third parties
				doing this and most are really bad).

	Customer billing:
		Various technologies are available; usefulness depends
		on the process defined

	Complaint department/Help desk; This can be as big as one
	wants, but think about:

		Delegation problems:
			Especially when delegation checks are done

		Conflicts about names:
			UDRP in-house or outsourced;
			Protests against registration rules

		Technical Problems;
			Complaints about the DNS (non-) functioning;
			Complaints about the weather :-)

	Delegation audits (potential things is will handle):
		Lame delegations;
		Key expire detection for DNSSEC etc.

In general, DNSSEC implementation will cause more interaction with the registrars and/of registrants.

Also note that the registry in it self is a registrant, notably by the parent--the root for a TLD. That role should be define somewhere, not only the politics (see above) but also the interaction for technical stuff (TLD NS server maintenance, TLD DS roll over). This is a point to tell about IANA processes.

_________________________________________________________________ (*) yeah, this whole frame work was started by Bart Boswinkel and me to overcome the stalemate over the "Touton" frame work which one can describe as "One contract fits all".

http://ws.edu.isoc.org/images/notes1.jpg

http://ws.edu.isoc.org/images/notes2.jpg

ccTLD Advanced workshop meeting, IETF 68, 20.3.2007

Joe, John, Lucy, Jaap, frederico, Steve Conte, mir

Define the goal a bit better:

open up doors for them to advance their operations
provide them with skill sets and a set of tools that can be used for various things.
first identify a set of known students and find out what the gaps are

at the moment, there is way too much material for 4 days. It was decided to focus on: monitoring tools, security, ticket systems, scripting (shell scripts, Pearl?), databases

agreed not to start with an intro to UNIX and DNS on the first day, but to require that as background knoowledge and to practice during the first exercise (at least in the first round of the workshop suggested to hand-pick the participants again like we did for the first worksop in Amsterdam).

agreed to take out DNSSEC or maybe only as an introductory presentation, no DNSSEC hands-on (also talked to Olaf and Peter Koch later and they confirmed that this would not be useful as part of this workshop and that it would be better to concentrate on normal network security issues)

Jaap has material on network monitoring that was prepared NL and SE he will get it and make it available

Joe Abley knows someone who is expert in Postgress Databases

TZ, MZ, KE is using the BR code now but BR is doing a lot of support is not really a generic tool

As soon as we have agreed on the outline, we will collect existing material. John will include an amount in the next ICANN budget round (June) for meterial development.

We could then have a first workshop beginning of 2008 (or end of 2007). NANOG will take place in Dom. Republic in Feb. 2008. Possibly organise an advanced ccTLD workshop in conjunction to that.

June 8, 2007

* Initial questionnaire * Operating System and DNS hands-on practice [????] - What OS. We need to choose. * Revision Control/Change management (RCS/CVS/SVN?) [Phil] ("Why is it that as soon as people move from paper to electrons, their processes "fall apart"?) - Concepts, zones, databases, etc. - We need to pick what we are using. SVN seemed to be a choice on the mailing list discussion. - Paper trail of everything you do. - application code must be versioned - network and OS upgrades must be managed - change history to zone data, WHOIS/contact information must be maintained (easy with flat text zones, but requires good practices such as append-only structures, rollback possibilities when transitioning to databases) - fulfilling auditing requirements * Introduction to Databases and Design [Phil/Hervey] * Scripting [Phil] * Cryptography Intro [Hervey] * Discussion of DNSSec (customized primers) [????] (DNSsec: why you don't need it know, but here is what you should know to be prepared the day your boss asks you to implement it (technical, policy reasons, motivations). - Discuss cryptography behind it. - What it implies and needs. - Conceptual overview. - Do some queries Use stuff from the WRC * Scaling up your Registry [Phil] (Scalability being not how much it can grow, but how easy it is to do so.) - Scalability (how easy to scale and use) - Splitting out services - Databases * Build a NOC to support the previous topics: [Phil] - Nagios, DNSMON, MRTG, SmokePing, Snort, syslog-ng, backup, RT (HelpDesk/Ticket systems), Trac, SLA, etc... - Talk with some registries. What are they using for ticketing systems. - More advanced topics. Escalation. How to prevent errors. - We need to pick a subset of these. - Error detection: + Business "end-to-end" process monitoring: how do we know that the zone we are about to publish is actually the right one? How do we detect it before it goes wrong? When do we deactivate a delegation? * Adding the secure bits (SSH, SSL, PGP) [Hervey] * SSL Certificate process [Hervey?] * Policy? [John] * Final exam? * Provisioning and Billing systems [Phil] - How do you do it? - Working with databases - Policies, payments (suspension, auto-renew, etc.) - Need to be usable in batch mode (mass suspend, renew, etc) - Buying an existing system (or being forced to use one by the beancounters) and "gluing" the two systems together. + Or, writing your own billing system New/Other --------- * Going from single to multiple registrars - two-tier vs. three-tier - Policies - EPP * Business model (Might tie this in with provisioning and billing) - Business monitoring system - Record changes - Same process every N number of hours - Business model (data) * Who we are inviting. - Technical people. - Larger registries * Security - what we mean. - Good business practices and processes - Recovery from security incidents - Well administered systems - Proper base security in place

Original Summary of Ideas

Basic OS and Security (Get folks on track) * Basic DNS (get folks on the same page) * DNSSEC * Creating a NOC * Monitoring your network * Helpdesk and ticketing requests * Mitigating damage from attacks, involves: - Building a NOC - Building admin servers - Several DNS boxes - Splitting class in to two networks - Attacking the setup * Scripting and automation * Debugging tools * Programmatic interfaces to DNS: - whois - registry tools - revision control - databases * How to make it all scale (can be a theme throughout the workshop) * Disaster recovery: - Logs - Backups - Archiving queries

Mirjam's Notes

Meeting with Lucy, john, mir on advanced ccTLD workshop development 4 - 5 days - security and DNSSEC - monitoring tools - how to set up a NOC Lucy: after the Thailand workshop, we will know more about how much time it takes to do hands-on DNSSEC training for a group of ccTLDs Day 1: ------ - build machine with everything we did in the basic workshop (BIND, OS, tools etc.) mir: not sure that is really necessary. might be better to provide that? Day 2: ------ - security (Hervey's slides) - monitoring tools, open source, processes - MRTG - ticketing system - debugging tools (general and DNS specific) Day 3: ------ - databases - scripting and other automation - how to make things scalable Day 4: ------ DNSSEC 1. half of 2007: material development 2. half of 2007: 1 workshop (possibly at AIT?) - maybe ask Alain if he wants to work on DNSSEC material or parts of it (1 or 2 days?) - also involve Peter Koch and Olaf Kolkman - ask Hervey to work on the other parts (security and monitoring tools), maybe together with Alain and Phil - ask Shane (ISC) if he wants to work on Database material (he might have some)

Hervey's Notes from Guyana

Our Raw Notes from the "Advanced ccTLD Workshop Planning Session" Georgetown, Guyana Brandsville Apartments - Terrace Office February 16, 2007 1830-1930 * Advanced DNS topics - DNSSEC - Higher availability * Management Operations - Monitoring - DNSMON - Network Monitoring - Logging - Secondaries and load (why?) - Capacity planning - Equipment lifetime - Scalability * Security and Availability - Attacks - DDOS - Mitigation - Resiliency - Disaster Recovery - Offsite archive of logs and data - Keep query log as TLD - Sign it, archive, stats * Incidents and Request Management - Help Desk - Tickets - Operations - Customer inquiries The idea is to build something. Build a NOC, two networks, several DNS servers, and administrative boxes for the network. Split this in to two teams, and they work on these together. And, we'll probably build a demo on our own. First 2 days we're building the system, talk them through theories, monitoring systems, then attack the setup. DNS Application Infrastructure ------------------------------ * Programmatic Interfaces - databases, - whois server - revision control - registry tools How to build, protect, monitor your registry infrastructure. How to build and maintain a solid network. Additional idea - Keynote things to discuss (1/2 hour) on cool new technologies as we go forward, such as IPv6, IDN, DNSMON, DLV, Bind (new version)...