Resubmit-based user loadable programs

[rcgoodfellow]

This issue describes how we might use the packet resubmit flow to realize user loadable P4 programs.

Our effort to make P4 switches user-programmable involves a system program which is always running and a set of dynamically loadable user programs that the system program can chose to execute. This issue is about exploring resubmit as a mechanism for doing that.

This is illustrated by extending the hub.p4 program to support dynamically loaded user programs.

User program control flow

We start with adding a user_program field to softnpu's ingress metadata. This field is used to track whether a packet should be processed by the system program or by a particular user program with a 16 bit identifier.

struct ingress_metadata_t {
    bit<16> port;
    bool nat;
    bit<16> nat_id;
    bool drop;
    bit<16> user_program; // <------ new field
}

As noted above, whether or not a packet is to be processed is a decision that is made by the system program. In this example, we'll modify hub program linked above to be a system program capable of executing user programs. For this modified hub program, let's say that the control plane for the hub program will provide an interface that lets user programs run for a range of source mac addresses.

The hub program currently has an empty egress controller. We can add the necessary P4 code to establish this source mac to user program association.

control egress(
    inout headers_t hdr,
    inout ingress_metadata_t ingress,
    inout egress_metadata_t egress,
) {

    // Set the user program in the ingress metadata to the program id in table
    // action entry.
    action set_user_program(bit<16> program_id) {
        ingress.user_program = program_id;
    }

    // This table associates a range of source mac addresses with a user program
    // id.
    table tbl {
        key = {
            hdr.ethernet.src_addr: range;
        }
        actions = {
            set_user_program;
        }
        default_action = NoAction;
    }

    apply {
        tbl.apply()
        // If we have a non-zero user program id, that means the user program
        // should be executed. Do a resubmit which will pick up the user program
        // id in the parser and jump into it.
        if(ingress.user_program != 0w16) {
            resubmit()
        }
    }

}

Then we check this field early in the parser and jump to it.

parser parse(
    packet_in pkt,
    out headers_t headers,
    inout ingress_metadata_t ingress,
){
    state start {
        // Check if a user program has been specified, if so, jump to it.
        if (ingress.user_program != 0w16) {
            // The following transitions execution to the user program. This
            // program executes no further, the user program is now in complete
            // control of the fate of this packet.
            exec(ingress.user_program);
        }
        pkt.extract(headers.ethernet);
        transition finish;
    }

    state finish {
        transition accept;
    }
}

Note that resubmit() and exec() are new softnpu mechanisms. And they should probably be added as P4 externs, they are just shown as simple functions for brevity here.

As I write this out, it also occurs to me that we may not even need the new metadata field or the trip back into the parser of the main program and could write the apply function of the egress controller like this

    apply {
        tbl.apply()
        // If we have a non-zero user program id, that means the user program
        // should be executed. Resubmit into that program from here.
        if(ingress.user_program != 0w16) {
            resubmit_exec(ingress.user_program)
        }
    }

Dynamically loading user programs

Softnpu already has machinery for dynamically loading x4c compiled programs. See the dynamically loading test for an example of how this works. Right now softnpu uses dynamic loading to load the p4 program. However, we'll need to extend it's capabilities to load user programs with an associated 16 bit identifier. A good starting place for this work is probably to make the softnpu test harness capable of dynamically loading user programs. As we'll ultimately be implementing this in the softnpu implementations that carry real network packets, it's worth taking a look at the standalone softnpu program that runs on illumos and carries packets between illumos data links and the softnpu implementation in the propolis hypervisor that acts as a virtual switch ASIC inside a virtual machine. The latter has explicit support for runtime program loading of the system program which may be a good model to use for user loadable programs.