`Robot` Submissions and `Bot` User

[ihsaan-ullah]

Observations

I checked the wiki for robot submissions and the instructions are straight forward but it looks like they are meant for a local deployment of codabench. The wiki page also points to an example script which shows how to make a submission.

Further, I checked the code and found the use of robot submissions here:

• Can make submission API - check participation Here the code checks if the request user is bot and is not approved participant and competition allows bot submissions then a new participant is created with approved status
• Can make submission API - check can make submission Here a function from phase is called to check if the request user can make submission. If the user is bot, then no limits are checked for the user and is allowed to make unlimited submissions BUT this works only if user calls the can_make_submission API. The submissions API does not check this.
• Participation Tab of Competition: Here is the UI that shows a checkbox checked/unchecked based on the allow_robot_submissions property. This file has 2 more occurrences of robot submissions but that are only used for displaying purposes (on the front end)

Summary

I believe that some or maybe all APIs should be redesigned because:

• Proper validation is not done
• One API depends on the response of another API and are not connected
• APIs rely on data sent from the user/front-end side without any checks
• At some places it is assumed that user will first call API A to check for permission and if permissions are ok then user will call API B. But that is an assumption and users can skip calling API A and call API B directly.
• NOTE: Some problems are not clearly visible because we assume that only the codabench frontend calls the codabench backend APIs but that is not always the case. Technical users can always setup their own front-end and call codabench APIs to do what they want and can take advantage of the issues/loopholes.

[ihsaan-ullah]

We need to review bot users and robot submissions. I think we don't need these.

https://github.com/codalab/codabench/blob/develop/documentation/docs/Developers_and_Administrators/Robot-submissions.md

[Didayolo]

Yes, I'm in favor of simplification, it does look like an useful feature.

Also, couldn't this job just be done by submitting through API?

[ihsaan-ullah]

Yes we can use api but for robot submissions there are two restrictions removed which I think should be checked

1. bot user can submiit even if it is not a participant
2. bot user has no submission limit restrictions

[Didayolo]

OK, I think that definitely seems wrong to keep this. Even if we wanted this feature it should be done in a more secure way.