feat: add e2e tests

[cychiang]

Description of your changes

Add missing e2e tests for the provider-template. It includes:

1. Update build to latest main
2. Add e2e tests for example resources

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable to ensure this PR is ready for review.
• Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

1. Run make e2e
2. Create another terminal to observe resources in default namespace.
3. The status of resource should reflect what test cases trying to verify.
4. Confirm behavior as same as each test case.

[jbw976]

this will be nice to have an e2e test pattern in place for folks starting from the template repo, thanks for doing this!

i haven't kept super up to date with these patterns myself though, does the implementation in this PR come from a particular example/docs/etc.? It'd be good to align on that first and then dive into the specifics on this implementation. thank you!

[cychiang]

this will be nice to have an e2e test pattern in place for folks starting from the template repo, thanks for doing this!

i haven't kept super up to date with these patterns myself though, does the implementation in this PR come from a particular example/docs/etc.? It'd be good to align on that first and then dive into the specifics on this implementation. thank you!

@jbw976

Hey, until now, I still have not much idea around how uptest suppose to work. Some learnings from build/makelib/uptest.mk to figure out the pattern to invoke e2e from exists build tool. For example here

So I need to have a setup.sh to bootstrap the local testing environment, and have test files ready. Then look into this uptest to figure out how annotation works.

Do you have any specific pattern you would like to see when running e2e test?

[jbw976]

cool @cychiang!! following an uptest pattern is totally reasonable, I just personally haven't had much experience with it.

Do you still have more to research into here or are you feeling comfortable with the current implementation?

Can you comment about how you've verified that this is working OK?

[cychiang]

cool @cychiang!! following an uptest pattern is totally reasonable, I just personally haven't had much experience with it.

Do you still have more to research into here or are you feeling comfortable with the current implementation?

Can you comment about how you've verified that this is working OK?

@jbw976 I have updated test cases, and adopt to chainsaw as uptest is using it for e2e test. To verify it, try to:

1. Run make e2e
2. Once kind cluster is up, observe default namespace, it's a place to host the managed resource for testing
3. Observe the managed resource by using --watch
4. The state of managed resource should reflect the purpose from each test case that will apply from chainsaw.

[jbw976]

awesome, thanks for continuing to drive this and establish a pattern here! everything seemed to run OK on my machine in my manual test run, so most of my questions are to make sure we are following and/or setting the right patterns in place for folks that use this template as a starter for their providers in the future. Thank you!

[jbw976]

Just looking quickly at other providers in crossplane/crossplane-contrib, it looks like they have a pattern of pointing uptest at their examples/ via UPTEST_EXAMPLE_LIST rather than a separate test manifest, with the annotations right in the examples (e.g. provider-kubernetes).

Do you think we should do the same to be consistent and to make sure the examples we publish are what are getting tested as opposed to maintaining separate e2e manifests?

I'm not married to that idea, because I can see the value of having a dedicated e2e test section/manifests also, so it's just an idea to explore here. You can make the call!

[jbw976]

make provider.prepare deletes apis/sample and internal/controller/mytype but only rewrites template, so MyType survives all through the new test/ files.

Not a blocker since the repo doesn't compile after prepare anyway, but test/ is now a fourth place to fix up and it's missing from the Developing steps. should we add a line for it there?

[jbw976]

I don't think this file can fail a run, so its exit 1s may not do anything.

i had an agent look into it more and the conclusion was that uptest drops pre-delete hooks into a chainsaw script block followed by the delete commands, and there's no set -e, so only the last command's exit code counts. Post-assert hooks are fine, those get rendered as a command entrypoint. the agent checked both on a kind cluster too.

do you want to confirm this suspicion too?

As is in this template repo, it doesn't matter too much because the import step already asserts what this hook checks, but folks building their own provider from this template and following this pattern could end up adding a real assertion in there and never know it didn't run. Should we keep this hook still you think? 🤔

[jbw976]

I noticed the local-dev cluster was still hanging around after running make e2e so I checked in on that a bit.

controlplane.down is a prerequisite of e2e in uptest.mk, so the teardown runs before the setup rather than after it:

16:53:01 [ .. ] deleting controlplane
Deleting cluster "local-dev" ...
16:53:27 [ .. ] setting up controlplane
Creating cluster "local-dev" ...

So it looks like it cleans up any local-dev that exists at the start then leaves the one it created running when the run finishes.

Is that what we want? 🤔