Instance shapes - nested KVM

Oracle Cloud does support nested KVM, but only on the Intel-based (X7) instance shapes.

The AMD-based shapes (E2 and E3) are cheaper, but E2 doesn't support nested KVM: the kvm-amd module refuses to load. (E3 not tested).

Performance-wise, nested virtualization has a high penalty. 2 OCPUs are only barely enough to run one campus of NMM, along with one transit router and the NOC - most probably due to the high background CPU usage of IOSv.

The likely instance shapes required for a full campus are:

  • VM.Standard2.16 (16 OCPU, 240GB RAM) - about $25/day; or
  • VM.Standard2.24 (24 OCPU, 320GB RAM) - about $37/day

These have not yet been tested with a full topology, however we expect CNDO will almost certainly require 24 OCPU. NMM might just about run with 16 OCPU but will not leave much headroom for running monitoring tools on the srv1 instances.

Instance shapes - bare metal

Much better performance is likely to be achieved from bare metal instances:

  • BM.Standard2.52 (Intel, 52 OCPU, 768GB) - about $80 per day
  • BM.Standard.E2.64 (AMD, 64 OCPU, 512GB) - about $46 per day

The AMD option is especially interesting, as it is not much more expensive than a 24-OCPU Intel virtual machine.

However, we have not yet tested it to confirm that it supports KVM (AMD-v).

Pricing

Compute prices shown are as of December 2020.

Block storage is extra, but only about $8.50 per month for a 200GB block volume with "balanced" performance (2000 performance units).

Note that unlike AWS, Oracle Cloud pricing does not vary between regions, and also includes free 10TB/month of egress traffic.

Signing up

You will almost certainly need to start with a paid account. A free trial account limits you to:

  • 2 Intel OCPUs per availability domain (data center)
  • 6 AMD E2 OCPUs and zero AMD E3 OCPUs per availability domain
  • A single region

This means you cannot create anything larger than VM.Standard2.2 (2 OCPU) to use with nested virtualization, and you cannot create any bare metal instances.

You can request a limit increase, but when we did this it was rejected - after a 12 day delay.

A paid account is also required to create instances in regions other than your home region.

Note

When signing up, we found that mails from the Oracle cloud platform were blocked by spam filters. Therefore, use an E-mail account where you can manually review and release quarantined messages, and/or add oracle.com and oraclecloud.com to your list of trusted sender domains.

Note

To convert a free trial account into a paid account, go to ☰ > Account Management > Payment Method and select "Pay As You Go".

Increasing instance limits

To check your limits and/or request an increase:

  • Go to ☰ > Governance > Limits, Quotas and Usage
  • Service: "Compute"
  • Scope: select the Availability Domain where you will run your instance (typically xxx-AD-1)
  • Resource:
    • Intel: "Cores for Standard2 based VM and BM instances" (standard2-core-count); or
    • AMD: "Cores for Standard.E2 based VM and BM Instances" (standard-e2-core-count)

This will show your current limit. Then click on "request a service limit increase" at the top of the page, which opens a new page.

Again select:

  • Service Category: "Compute"
  • Resource: as above
  • Under the xxx-AD-1 limit, enter the increased value
    • Intel: "24" or "52" (the latter for bare metal BM.Standard2.52)
    • AMD: "64" to be able to use bare metal BM.Standard.E2.64
  • Add the reason for your request, e.g. describe the training

You can create a single request with multiple instance limits in it if you wish.

Note

It is supposed to take around 1 business day for this to be actioned, but can take longer, so make sure you have this done well in advance of the start of your training.

If you've not had any response after a few days, use the Chat function to talk to an agent. You will need the CAM reference number which is sent to you via E-mail when you create the request - the web console doesn't have the ability to track outstanding requests.

Instance creation

  • Image: select Canonical Ubuntu 18.04
  • Shapes: see above.
  • Assign public IPv4 addresss: Yes
  • Select "Paste public keys" and paste in your SSH public key
  • Check "Specify a custom boot volume size" and select a larger size. (Your account allows a single 100GB volume for free)
  • After it boots, ssh to ubuntu@<public-IP>

Once you have logged in, double-check that KVM is available:

$ sudo apt-get install cpu-checker
$ sudo kvm-ok

If not, you will see:

INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used

and you should not proceed with this instance.

Firewalling

There are two things you need to change to enable inbound web traffic.

Firstly, although the instance itself has no security group by default, the subnet has a security group which only permits port 22 (SSH).

  • Go to ☰ > Networking > Virtual cloud networks
  • Click on your vcn
  • Click on your subnet
  • Click on the security list ("Default Security List for vcn-xxx-xxx")
  • Click "Add Ingress Rules"
    • Source CIDR: 0.0.0.0/0
    • IP Protocol: TCP
    • Destination Port Range: 80,443
    • Description: Allow web
    • Click "Add Ingress Rules" to confirm
  • Optionally add another rule to allow inbound pings:
    • Source CIDR: 0.0.0.0/0
    • IP Protocol: ICMP
    • Type: 8
    • Code: 0

Secondly, the way Oracle configures Ubuntu, they install their own iptables rules which block inbound traffic apart from ping and ssh. This also prevents srv1 instances from reaching apt-cacher-ng.

Check the existing ruleset:

$ sudo iptables -L INPUT -n -v --line-numbers
...
10    7053 2717K REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
$ 

Make a note of the number of the last (REJECT) rule - in this case it was 10. Use this number when inserting the new rules, so they go before it:

$ sudo iptables -I INPUT 10 -m state --state NEW -p tcp --dport 80 -j ACCEPT -m comment --comment "Allow web"
$ sudo iptables -I INPUT 10 -m state --state NEW -p tcp --dport 443 -j ACCEPT -m comment --comment "Allow web"
$ sudo iptables -I INPUT 10 -m state --state NEW -p tcp --dport 3142 -s 100.64.0.0/10 -j ACCEPT -m comment --comment "Allow apt-cacher-ng"
$ sudo iptables-save | sudo tee /etc/iptables/rules.v4

Shutting down

In testing, halt -p within an instance was not enough to stop it completely - the console said it was still "Running". It was necessary to issue a Stop from the console as well. This is important to prevent ongoing charging.

Reserved IP addresses

An instance can be shutdown for a while and it will often retain the same IP address when restarted - that is, in practice it is not quickly recycled, even after several days.

However, if you want to guarantee that it can be brought up again on the same IP public address, then you should use a Reserved Public IP (similar to what AWS calls "Elastic IP").

Details are in the documentation under managing public IPs.

IPv6

IPv6 was originally only available for US government users, but since 7 July 2020 is in limited availability for everyone else.

If you want to use it, you need to ask Oracle to enable it for your tenancy, although we've been unable to test it so far.

According to a response via Chat:

Please note that it is strongly recommended to contact My Oracle Support Team (MOS) for assistance regarding this topic; please click on: https://support.oracle.com using your Customer Support Identifier (while logged into your console, please go to the right top corner and select Profile > Tenancy > CSI Number). You may also contact the Oracle Hub Team by telephone. Call 1.800.223.1711 (US only) and/or check here the telephone number for a different Country or Region: http://www.oracle.com/us/support/contact/index.html

Support

Your cloud admin login is not an Oracle account for https://support.oracle.com, so you either need to register for a new Oracle account, or login with an existing one, which you should then be able to link to the support identifier (CSI number) from your cloud account.

Once you've requested this, your Oracle account will show the CSI in state "pending". You can then back into your cloud account, go to Profile > (username), and select More Actions > Link Support Account.

Note

If the E-mail used for the cloud login is different to the E-mail used for the Oracle account, it may take some time for the link to be manually approved. Better to make them the same.