TaskForge is a background job queue I built from scratch for the MLSA technical task 5.3. Instead of going the BullMQ and Redis route, I decided to build the queue system from first principles using just Node.js and in memory data structures. The idea was pretty straightforward. There is an Express API where you submit jobs, a background worker picks them up and processes them, and if a job fails it gets retried a few times with increasing delays before it is given up on and sent to a dead letter queue.
I went with the build it yourself approach because the task description mentioned that a cron plus polling method was perfectly fine, and I wanted to actually understand how job queues work under the hood rather than just wiring up BullMQ. The queue itself is a class that extends EventEmitter, with separate arrays and maps tracking pending, active, completed, and dead jobs. The worker polls every 500ms and can handle up to 3 jobs at a time.
For the second year extension, I added a dead letter queue where jobs that fail all 3 retries end up with their error history intact. I also built a status dashboard that shows live counts for pending, running, completed, and dead jobs. The dashboard updates in real time using Socket.IO so there is no need to refresh. There is also a Force Fail button that submits a job designed to always fail, so you can watch it retry and land in the DLQ without having to use command line tools.
You need Node 18 or higher. No Redis, no Docker, nothing else to set up.
npm install
npm start
Server starts on http://localhost:3002. Open that in your browser and you will see the dashboard. You can also hit the API directly.
You can submit a job using a POST request to http://localhost:3002/api/jobs with a JSON body containing the type and payload. Check stats using a GET request to http://localhost:3002/api/stats. Retry a dead job using a POST request to http://localhost:3002/api/jobs/JOB_ID/retry.
Job types you can use include email, report, image processing which fails randomly 40 percent of the time which is good for seeing retries, and always fail.
The src/index.js file contains the Express server, API routes, and Socket.IO setup. The src/queue.js file is the queue engine itself handling enqueue, dequeue, retry logic, and DLQ. The src/worker.js file runs the background worker that polls and processes jobs. The src/handlers.js file holds the actual job handlers for simulated tasks. The dashboard folder contains the status page written in HTML, CSS, and JS with a Socket.IO client.
The biggest limitation is that everything is in memory. If the server restarts, all jobs are gone. In a real system I would persist jobs to a database, even SQLite would work, so nothing gets lost. I would also add proper logging instead of just console.log, and maybe a way to cancel running jobs. If I had more time I would probably add job priority levels and a proper scheduler instead of simple polling.