A task queue in the database you already run. Reliable background jobs for Go on top of PostgreSQL, MySQL or SQLite. No Kafka, RabbitMQ or Redis.
- Why goque
- Installation
- Quick Start
- Managing tasks
- Example service
- Databases
- Configuration Options
- Task Status Lifecycle
- Delivery Guarantees
- Built-in Features
- Development
- Project Structure
Emails, webhooks, polling external systems, reports: background work usually gets its own broker, which has to be deployed, monitored and upgraded. The broker is also outside your database transaction, so an order can be saved while the task for it is lost.
goque keeps the queue in a table of the database you already have:
- Transactional outbox in one line: enqueue a task in the same transaction as your domain write, so both are committed or neither is (example).
- One table, the same code on three databases:
goque_taskwith three indexes on PostgreSQL, MySQL or SQLite; SQL is type-safe through go-jet. - Safe with many instances: the fetch uses
FOR UPDATE SKIP LOCKED, so two instances never take the same task, and a task version stops a worker that lost its task from overwriting the new owner's result. - Many task types, one manager: each type gets its own worker pool and settings.
- Retries, timeouts and a healer: retries with your own backoff, a per-task
timeout through
ctx, stuck tasks returned to work, panics recovered, graceful shutdown. - Less code around each task: typed payloads,
RescheduleTaskfor "not ready yet" without spending an attempt, duplicate protection by external ID, cron jobs, before/after hooks. - Settings change without a restart: workers, batch size, intervals and
timeouts take a
Value[T]that can follow your config or feature flags. - Observable: Prometheus metrics with a ready Grafana dashboard, structured logs (zap or slog), OpenTelemetry tracing.
go get github.com/ruko1202/goqueRequirements: Go 1.25+ and one of PostgreSQL 12+, MySQL 8+ (with binlog_format
ROW or MIXED, see MySQL caveats) or SQLite 3+ for
development and tests.
Apply the migrations from migrations/ with
goose:
go install github.com/pressly/goose/v3/cmd/goose@latest
goose -dir migrations/pg postgres "your-dsn" up # PostgreSQL
goose -dir migrations/mysql mysql "your-dsn" up # MySQL
goose -dir migrations/sqlite sqlite3 "your-dsn" up # SQLiteTo run the databases locally with Docker, see Development.
Implement the TaskProcessor interface to define how your tasks should be processed:
type EmailProcessor struct{}
func (p *EmailProcessor) ProcessTask(ctx context.Context, task *goque.Task) error {
// Your task processing logic here
return sendEmail(task.Payload)
}Use a typed processor when you want Goque to decode JSON payloads before your processing logic runs:
type EmailPayload struct {
To string `json:"to"`
Subject string `json:"subject"`
}
processor := goque.NewTypedTaskProcessor(
goque.TypedTaskProcessorFunc[EmailPayload](func(ctx context.Context, task *goque.TypedTask[EmailPayload]) error {
return sendEmail(task.Payload.To, task.Payload.Subject)
}),
goque.WithCancelTaskWhenPayloadDecodeError[EmailPayload](),
)If payload decoding fails, the typed processor is not called. The decode error is returned through the normal task processing flow, recorded in goque_payload_decode_errors_total, written to task.Errors, and passed to WithHooksAfterProcessing. Add WithCancelTaskWhenPayloadDecodeError to the typed processor to cancel decode failures instead of retrying them:
goq.RegisterProcessor(
"send_email",
processor,
goque.WithHooksAfterProcessing(func(ctx context.Context, task *goque.Task, err error) {
if errors.Is(err, goque.ErrPayloadUnmarshal) {
// alert or inspect invalid payload
}
}),
)Return goque.RescheduleTask(after) (optionally wrapped) when the task is not finished yet,
e.g. while polling an external status. The task goes back to new and runs again after after;
attempts and errors stay unchanged. Rescheduling is unbounded, so the processor should
enforce its own deadline (e.g. based on task.CreatedAt).
package main
import (
"context"
"log"
"os"
"os/signal"
"syscall"
"time"
"github.com/ruko1202/goque"
"github.com/ruko1202/goque/pkg/goquestorage"
"github.com/ruko1202/xlog"
"go.uber.org/zap"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
ctx := context.Background()
// Structured logging with xlog (zap, slog or a custom adapter)
logger := xlog.NewZapAdapter(zap.Must(zap.NewProduction()))
xlog.ReplaceGlobalLogger(logger)
ctx = xlog.ContextWithLogger(ctx, logger)
// Optional: OpenTelemetry tracing, set before creating Goque
// goque.SetTracerProvider(tracerProvider)
// Optional: service name for metrics labels
goque.SetMetricsServiceName("my-service")
db, err := goquestorage.NewDBConn(ctx, &goquestorage.Config{
DSN: "postgres://user:pass@localhost:5432/goque?sslmode=disable",
Driver: goquestorage.PgxDriver,
MaxOpenConn: 20,
MaxIdleConn: 20, // at least the workers count, see "Databases"
})
if err != nil {
return err
}
// Task storage for whichever database the connection points to
taskStorage, err := goque.NewStorage(db)
if err != nil {
return err
}
// The Goque manager also runs the built-in healer and cleaner
goq := goque.NewGoque(taskStorage)
goq.RegisterProcessor(
"send_email",
&EmailProcessor{},
goque.WithWorkersCount(goque.NewStaticValue(10)),
goque.WithTaskProcessingMaxAttempts(goque.NewStaticValue[int32](3)),
goque.WithTaskProcessingTimeout(goque.NewStaticValue(30*time.Second)),
)
if err := goq.Run(ctx); err != nil {
return err
}
// Graceful shutdown: in-flight tasks finish before Stop returns
defer goq.Stop()
// Serve until SIGINT or SIGTERM
sigCtx, stop := signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)
defer stop()
<-sigCtx.Done()
return nil
}taskQueueManager := goque.NewTaskQueueManager(taskStorage)
// A task with a raw JSON payload
task := goque.NewTask("send_email", `{"to": "user@example.com", "subject": "Hello"}`)
err := taskQueueManager.AddTaskToQueue(ctx, task)
// With an external ID: enqueuing the same ID again returns goque.ErrDuplicateTask
dedupTask := goque.NewTaskWithExternalID("send_email", payload, "external-order-123")
err = taskQueueManager.AddTaskToQueue(ctx, dedupTask)
// With a typed payload marshaled to JSON
typedTask, err := goque.NewTaskWithPayload("send_email", EmailPayload{
To: "user@example.com",
Subject: "Hello",
})
if err == nil {
err = taskQueueManager.AddTaskToQueue(ctx, typedTask)
}Enqueue a task atomically with your own domain writes by passing a
*sqlx.Tx through the context. If the tx rolls back, the enqueue is
discarded with it; if it commits, the task becomes visible to workers
in the same instant the domain rows do.
tx, err := db.BeginTxx(ctx, nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }() // no-op after a successful Commit
// 1) Your domain write inside the tx
if _, err := tx.ExecContext(ctx, "INSERT INTO orders ..."); err != nil {
return err
}
// 2) Enqueue using the same tx β goque.WithTx wires it into ctx
task := goque.NewTask("send_order_confirmation", payload)
if err := taskQueueManager.AddTaskToQueue(goque.WithTx(ctx, tx), task); err != nil {
return err
}
// 3) Commit β domain row and queued task become durable together
return tx.Commit()Calls without goque.WithTx keep the existing behavior and write
directly to the storage's *sqlx.DB.
Scope of
WithTx
AddTaskToQueueis the method that matters for the outbox pattern, and it participates in your tx on every backend. Other tx-aware methods:GetTask,GetTasks,UpdateTask,CancelTask, and (PostgreSQL only)DeleteTasks/CureTasksβ they also honor an attached tx.
GetTasksForProcessingandResetAttemptsalways run in their own internally-managed tx (the worker fetch usesFOR UPDATE SKIP LOCKEDand must not be entangled with a caller's tx). On MySQL/SQLite,DeleteTasksandCureTasksare in this group too because they do batched read+write internally.
AsyncAddTaskToQueueis not outbox-safe. Its goroutine outlives yourCommit/Rollback. If a tx is detected on the async path it is stripped automatically and logged at WARN β but the enqueue then runs against*sqlx.DBand is no longer atomic with your domain write. Always use the synchronousAddTaskToQueuefor outbox.
TaskQueueManager (from goque.NewTaskQueueManager) covers the rest of a task's life:
| Method | What it does |
|---|---|
AddTaskToQueue(ctx, task) |
Enqueues synchronously; joins a tx attached with goque.WithTx |
AsyncAddTaskToQueue(ctx, task) |
Enqueues in a background goroutine; errors are logged, not returned; never joins a tx |
GetTask(ctx, id) |
Returns one task |
GetTasks(ctx, filter, limit) |
Returns tasks matching a goque.TaskFilter (IDs, type, statuses, age) |
CancelTask(ctx, id) |
Moves a non-terminal task to canceled; a run already in flight is not stopped, its result is dropped |
ResetAttempts(ctx, id) |
Sets the task back to new with zero attempts |
WaitAsyncEnqueues() |
Waits for AsyncAddTaskToQueue goroutines; Goque.Stop calls it for you |
CancelTask and ResetAttempts return goque.ErrTaskVersionConflict if the task
changed between their read and write; see Delivery Guarantees.
A complete example service lives in examples/service:
cd examples/service
make docker-up
# localhost:8080 dashboard and API
# localhost:3000 Grafana
# outbox on orders: POST /api/ordersIt runs PostgreSQL, Prometheus, Loki, Grafana, Jaeger and an OpenTelemetry collector, five task types and a load generator. See examples/service/README.md for the API and dashboards.
| Database | Enqueue | Processing, 1 β 50 workers | Best For | Production Ready |
|---|---|---|---|---|
| PostgreSQL | 0.44 ms | 1,040 β 4,810 tasks/s | High-throughput production systems | β Recommended |
| MySQL | 1.09 ms | 400 β 2,050 tasks/s | Teams already running MySQL | β Yes, see MySQL caveats |
| SQLite | 0.06 ms | 9,840 β 8,170 tasks/s | Local development, testing, CI/CD |
Measured with make bench against Docker databases on one laptop; see
PERFORMANCE_ANALYSIS.md for the setup and what each
number means. Throughput is bound by database round trips: each processed task
costs two single-row writes plus a share of one fetch transaction. SQLite looks
fastest only because it has no network hop; it serializes all writes on one
connection and one process.
Size the connection pool for your workers. Keep MaxIdleConns at least at
the total number of workers plus a few for the fetcher, healer and cleaner.
Below that, database/sql closes and reopens connections on every burst; with
50 workers on PostgreSQL that cut throughput by almost 3Γ. Across processes,
keep instances Γ MaxOpenConns below the server's max_connections (151 on
stock MySQL).
SQLite serializes writes at the file level. Two practical consequences for goque on SQLite:
- Concurrent writers deadlock under the default
BEGIN DEFERREDtransaction mode. Operations likeCureTasksandGetTasksForProcessingopen their own transactions viadbtx.WithinTx; when several workers call them in parallel, one SELECT takes a SHARED lock and the next UPDATE can't upgrade to RESERVED β SQLite returnsSQLITE_BUSYimmediately and the_busy_timeoutparameter does not apply to lock upgrades. Mitigation: pass?_txlock=immediate&_journal_mode=WAL&_busy_timeout=5000in your DSN so all transactions start asBEGIN IMMEDIATEand multiple readers can coexist with the single writer. - One process at a time. Even with the DSN tuning above, a single
SQLite database file should be opened by a single goque process.
Spinning up two binaries that both write to
goque.sqlite.dbwill seedatabase is lockederrors no matter what.
If you need real horizontal scaling, run PostgreSQL or MySQL β they have row-level locks and don't have this class of problem.
goque runs its own MySQL transactions (task fetch, healer, cleaner,
ResetAttempts) at READ COMMITTED, whatever the server default is.
Under InnoDB's default REPEATABLE READ the fetch's locking range scan
also locks the index gaps around it, and those gaps belong to tasks that
workers are moving to another status at the same moment; the two
UPDATEs deadlock (Error 1213). READ COMMITTED locks only the rows
the scan returns.
This needs binlog_format ROW (the default) or MIXED. With
STATEMENT, MySQL rejects writes in a READ COMMITTED transaction with
Error 1665.
Goque installs a single table named goque_task plus three indexes
(goque_task_type_external_id_idx, goque_task_type_status_next_attempt_at_idx,
goque_task_type_status_updated_at_idx). The full DDL lives in
migrations/ β apply it with make db-up.
The GoqueProcessor supports various configuration options. Scalar options take a
goque.Value[T], so they can be constant or change at runtime (see Dynamic Configuration):
WithWorkersCount(v Value[int])- Number of concurrent workers (default: 10, max: 10000)WithWorkersPanicHandler(handler func(context.Context) func(any))- Set a custom worker panic handlerWithTaskProcessingMaxAttempts(v Value[int32])- Maximum retry attempts (default: 5)WithTaskProcessingTimeout(v Value[time.Duration])- Per-task timeout (default: 30s)WithTaskProcessingNextAttemptAtFunc(f)- Custom retry backoff strategyWithTaskFetcherMaxTasks(v Value[int64])- Maximum tasks to fetch per cycle (default: 100)WithTaskFetcherTick(v Value[time.Duration])- Fetch interval (default: 30s)WithTaskFetcherTimeout(v Value[time.Duration])- Timeout for fetching tasks from storage (default: 30s)WithHooksBeforeProcessing(hooks ...HookBeforeProcessing)- Add pre-processing hooksWithHooksAfterProcessing(hooks ...HookAfterProcessing)- Add post-processing hooksWithCleanerPeriod(v Value[time.Duration])- Cleaner run interval (default: 5m)WithCleanerUpdatedAtTimeAgo(v Value[time.Duration])- Completed-task age threshold for cleanup (default: 3h)WithCleanerTimeout(v Value[time.Duration])- Cleaner operation timeout (default: 30s)WithHealerPeriod(v Value[time.Duration])- Healer run interval (default: 5m)WithHealerUpdatedAtTimeAgo(v Value[time.Duration])- Stuck-task age threshold for healing (default: 1h)WithHealerTimeout(v Value[time.Duration])- Healer operation timeout (default: 30s)
All scalar values must be positive. An invalid (<= 0) value is never applied: the last valid value
(or the default) is used and a warning is logged.
Every scalar option is read on each processor cycle, so it can be changed without a restart.
goque.NewStaticValue(v)- a constant value.goque.NewDynamicValue(ctx, default, fetch, interval)- pollsfetcheveryintervalin the background and caches the result. The first fetch is synchronous; on errordefaultis used, and later errors keep the last successful value.fetchgets a context with a timeout ofintervaland must honor it. Polling stops whenctxis done. Invalid arguments (nilctx/fetch,interval <= 0) are logged and the value stays atdefault.- Any type implementing
Get() T(non-blocking, must not panic, safe for concurrent use) works as agoque.Value[T].
// Read workers count from your config system / feature flags every 10 seconds.
goq.RegisterProcessor(
"send_email",
&EmailProcessor{},
goque.WithWorkersCount(goque.NewDynamicValue(ctx, 10, func(ctx context.Context) (int, error) {
return configClient.GetInt(ctx, "queue.send_email.workers")
}, 10*time.Second)),
goque.WithTaskFetcherMaxTasks(goque.NewStaticValue[int64](50)),
)When changes are applied:
- fetch
maxTasks/timeout, processingtimeout,maxAttempts, healer/cleanertimeoutandupdatedAtTimeAgo- on their next use (next fetch, task or run); - workers count - on the next fetch tick or before the next task submit; the worker pool is resized in place and in-flight tasks are not interrupted. When the pool is full, growth applies once a running task finishes; after a shrink, extra workers retire as they finish their current task;
- fetch tick and healer/cleaner period - after the current tick/period elapses.
Periodic jobs run in separate scheduler processors. On each schedule tick, Goque calls the job factory and inserts the returned task into the queue through TaskQueueManager. The inserted task is a normal one-shot task: successful tasks still become done, failed tasks still use the normal retry flow, and cancellation is handled by the queue manager.
Use NewCronJob for a standard 5-field cron expression:
periodicJob, err := goque.NewCronJob(
"daily_report_schedule",
"0 3 * * *",
time.UTC,
func(ctx context.Context) (*goque.Task, error) {
return goque.NewTask(
"daily_report",
`{"report":"sales"}`,
), nil
},
goque.WithPeriodicJobRunOnStart(),
)
if err != nil {
return err
}
goq := goque.NewGoque(taskStorage)
goq.RegisterPeriodicJob(periodicJob)WithPeriodicJobRunOnStart() is optional. When enabled, the periodic job enqueues one task immediately when Goque starts, then continues with its normal schedule.
Register a normal processor for the task type produced by the periodic factory:
goq.RegisterProcessor(
"daily_report",
&DailyReportProcessor{},
)For non-cron schedules, use NewPeriodicJob with PeriodicSchedulerFunc or any type that implements PeriodicSchedule:
periodicJob, err := goque.NewPeriodicJob(
"quarter_hour_report_schedule",
goque.PeriodicSchedulerFunc(func(t time.Time) time.Time {
return t.Add(15 * time.Minute)
}),
func(ctx context.Context) (*goque.Task, error) {
return goque.NewTask(
"quarter_hour_report",
`{"report":"sales"}`,
), nil
},
)
if err != nil {
return err
}Use external_id when the periodic job must be idempotent for a schedule slot:
slot := time.Now().UTC().Truncate(15 * time.Minute)
externalID := fmt.Sprintf("quarter_hour_report:%s", slot.Format(time.RFC3339))
task := goque.NewTaskWithExternalID("quarter_hour_report", payload, externalID)Tasks flow through the following states:
new βββ pending βββ processing βββ¬βββ done
β² β² ββββ canceled
β β ββββ attempts_left
β βββββββ error βββββββββ€ failed, attempts left (fetched again when due)
βββββββββββββββββββββββββββββββββ RescheduleTask, or returned on graceful shutdown
Healer: stuck pending or processing βββ error
CancelTask: any non-terminal status βββ canceled
ResetAttempts: any status βββ new (attempts reset to 0)
- new - Created and waiting until
next_attempt_atis due - pending - Taken by a fetcher, waiting for a free worker
- processing - Being processed by a worker
- done - Completed successfully β (terminal)
- error - Failed with attempts left; fetched again once
next_attempt_at(set by the backoff) is due - attempts_left - Failed with no attempts left β (terminal)
- canceled - Canceled by
CancelTaskor by the processor (goque.ErrTaskCancel) β (terminal)
| Current Status | Next Status | Trigger |
|---|---|---|
new, error |
pending |
A fetcher takes a due task (next_attempt_at has passed) |
pending |
processing |
A worker starts the task |
processing |
done |
Successful processing |
processing |
error |
Failed processing with attempts left; next_attempt_at set by the backoff |
processing |
attempts_left |
Failed processing with no attempts left, or its attempts were already used up (cures from processing count as attempts), in which case it is not run |
processing |
canceled |
The processor returned an error wrapping goque.ErrTaskCancel |
processing |
new |
The processor returned RescheduleTask(after) (attempts unchanged), or graceful shutdown returned the task to the queue |
pending, processing |
error |
Healer marks a stuck task (cure); a cure from processing consumes one attempt, a cure from pending does not |
| any non-terminal | canceled |
CancelTask |
| any | new |
ResetAttempts (attempts set to zero) |
Tasks in these states will not be processed again:
- done - Successfully completed
- canceled - Canceled by
CancelTaskor the processor - attempts_left - Failed with no remaining retry attempts
Goque delivers tasks at least once. A task can run more than once, so processors must be idempotent.
Every task row has a version (Task.Version). Fetching, healing, canceling and
resetting a task bump it, and every write made by a worker is applied only if the
stored version still matches the one the worker holds. A write made with a stale
version is dropped with goque.ErrTaskVersionConflict (passed to after-hooks, logged
and counted in goque_task_version_conflicts_total). So a worker that lost its task can never
overwrite the result of whoever owns the task now. The version protects goque's own
row only: it cannot undo side effects the stale worker already caused.
- The healer cures a task that is still running, e.g. the processor ignores
ctxand outlives its timeout, pod clocks drift apart, or the process stalls. The healer never uses a processing threshold below2 Γ processing timeout + 10s(a lower configured value is raised and logged), but that is best effort; the version keeps the row correct when it is not enough. CancelTaskon a pending or processing task wins the row but does not stop an in-flight run: its side effects may still happen and its result is dropped.canceledmeans "the result will not be recorded", not "did not execute".ResetAttemptson a pending or processing task takes it away from the current run; the task may be fetched and run again while the first run is still going.
- Use
task.ID(or your own business key) as the idempotency key for external calls. - Make domain state changes conditional (
UPDATE payment SET status = 'captured' WHERE id = $1 AND status = 'authorized'), not blind writes. - Honor
ctx: stop work when it is canceled or its deadline passes.
- A healer cure from
processingcounts as an attempt (and appendsattempt N: task was stucktoerrors): a task that keeps overrunning ends inattempts_leftinstead of running forever. A cure frompendingdoes not: the task never started, so itsattemptsanderrorsstay as they are. LoweringmaxAttemptsat runtime also moves fetched tasks at or above the new limit toattempts_leftwithout another run. - Accepted trade-off: a task that keeps getting stuck between fetch and start (healer
threshold shorter than the worker backlog, pods dying right after fetch, the start
write failing with a database error) is cured and fetched again indefinitely, without
ever reaching
attempts_left. None of these causes is the task's fault; fix the configuration or the infrastructure. Each cycle is one row write per healer tick and the row does not grow. - Hooks always run in pairs for every fetched task. If moving the task to
processingfails (for example withgoque.ErrTaskVersionConflict), the task is not processed and after-hooks get that error with the task stillpending. - A panic while processing a task (in the processor or a hook) is recovered, logged and
fails the task like any error (it uses an attempt); after-hooks get
goque.ErrTaskPanic. - After-hooks may receive
goque.ErrTaskAttemptsExhausted(task moved toattempts_leftwithout running). If writing the result failed,erralso wraps that error (for examplegoque.ErrTaskVersionConflict) and the task holds the result that was not persisted.
If you call UpdateTask inside your own transaction (WithTx) and roll it back,
re-read the task: its in-memory version is ahead of the database.
Goque includes a built-in healer processor that automatically monitors and fixes stuck tasks. Tasks that remain in the "pending" or "processing" status for too long are automatically marked as errored, allowing them to be retried; a cure from processing consumes one attempt, a cure from pending does not. The healer is automatically registered when you call goque.NewGoque().
Tasks are cured after WithHealerUpdatedAtTimeAgo, but never earlier than
2 Γ processing timeout + 10s: a smaller configured value would cure processing tasks a
live worker still holds, so it is raised and a WARN is logged on every healer run until
the configuration is fixed. See Delivery Guarantees.
You can configure the healer behavior:
goq.RegisterProcessor(
"send_email",
&EmailProcessor{},
goque.WithWorkersCount(goque.NewStaticValue(10)),
goque.WithTaskProcessingMaxAttempts(goque.NewStaticValue[int32](3)),
goque.WithTaskProcessingTimeout(goque.NewStaticValue(30 * time.Second)),
goque.WithHealerPeriod(goque.NewStaticValue(10*time.Minute)),
goque.WithHealerUpdatedAtTimeAgo(goque.NewStaticValue(time.Hour)),
goque.WithHealerTimeout(goque.NewStaticValue(30*time.Second)),
)Goque includes built-in Prometheus metrics for comprehensive monitoring of your task queue. Metrics are automatically collected during task processing operations.
π Ready-to-import Grafana dashboard lives in
grafana/β a Prometheus metrics dashboard with import instructions. Seegrafana/README.md.
| Metric Name | Type | Labels | Description |
|---|---|---|---|
goque_processed_tasks_total |
Counter | task_type, status |
Tasks by type and status: counted on enqueue (new), when a worker starts a task and when it finishes (final status) |
goque_task_processing_duration_seconds |
Histogram | task_type, operation |
Duration of task processing and of the internal operations |
goque_operations_total |
Counter | task_type, operation |
Tasks handled by the fetcher (fetch), cleaner (cleanup) and healer (health) |
goque_task_retry_attempts |
Histogram | task_type |
Attempts a task had used when it finished |
goque_processors_workers_count |
Gauge | task_type |
Current number of workers per task type |
goque_task_payload_size_bytes |
Histogram | task_type |
Task payload size distribution in bytes |
goque_payload_decode_errors_total |
Counter | task_type |
Typed task payload JSON decode errors by task type |
goque_task_version_conflicts_total |
Counter | task_type, stage |
Task writes dropped because another writer changed the task (stage: start, finish, return). Expected β 0; finish > 0 means a result was dropped |
import (
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/ruko1202/goque"
"net/http"
)
// Optional: Set service name for metrics labels
goque.SetMetricsServiceName("my-service")
// Expose metrics endpoint
http.Handle("/metrics", promhttp.Handler())
go http.ListenAndServe(":9090", nil)The operation label takes these values:
fetch- Taking tasks from the databaseprocessing- Running a task (duration only)cleanup- Removing old finished taskshealth- Curing stuck tasks
# Task processing rate by type and status
sum by (task_type, status) (rate(goque_processed_tasks_total[5m]))
# Failure rate: tasks finishing in error or attempts_left
sum by (task_type) (rate(goque_processed_tasks_total{status=~"error|attempts_left"}[5m]))
# Typed payload decode errors by task type
sum by (task_type) (rate(goque_payload_decode_errors_total[5m]))
# Average processing duration
rate(goque_task_processing_duration_seconds_sum{operation="processing"}[5m])
/ rate(goque_task_processing_duration_seconds_count{operation="processing"}[5m])
# 95th percentile processing time
histogram_quantile(0.95, sum by (le, task_type) (rate(goque_task_processing_duration_seconds_bucket{operation="processing"}[5m])))
# Tasks fetched per second
sum by (task_type) (rate(goque_operations_total{operation="fetch"}[5m]))
For a complete example with metrics integration, see examples/service/.
Goque uses xlog for structured logging with support for multiple backends (Zap, Slog, custom adapters). See Run Goque for configuration.
What gets logged:
- Task lifecycle events (creation, processing, completion)
- Error events and retry attempts
- Database operations and transaction management
- Worker pool events and task distribution
Goque supports OpenTelemetry for distributed tracing. By default it uses a no-op
tracer. To enable tracing, call goque.SetTracerProvider(tracerProvider) as in
Run Goque.
Traced operations:
- Task processing loop and fetching
- Hook execution (before/after)
- Task queue operations (add, get)
goque.SetTracerProvider() before creating Goque instances.
- Go 1.25 or higher
- Docker (for the local PostgreSQL and MySQL), or SQLite 3+
- Make
# Install all dependencies and tools
make all
# Install only binary dependencies
make bin-deps
# Download Go modules
make deps# Start PostgreSQL and MySQL with Docker Compose
make docker-up
# Configure your database connection in .env.local (one "KEY: value" per line)
# For PostgreSQL (recommended):
echo 'DB_DRIVER: postgres' > .env.local
echo 'DB_DSN: postgres://postgres:postgres@localhost:5432/goque?sslmode=disable' >> .env.local
# For MySQL:
# echo 'DB_DRIVER: mysql' > .env.local
# echo 'DB_DSN: root:root@tcp(localhost:3306)/goque?parseTime=true&loc=UTC' >> .env.local
# For SQLite (dev/test only):
# echo 'DB_DRIVER: sqlite3' > .env.local
# echo 'DB_DSN: ./goque.db' >> .env.local
# Install database tools (goose)
make bin-deps-db
# Apply migrations
make db-upFor detailed database setup instructions, see DATABASE.md.
# Run all tests
make tloc
# Run tests with coverage
make test-cov# All benchmarks against all databases
make bench
# One benchmark, one database, five runs
DB_DRIVER=pgx make bench BENCH=Processing BENCH_COUNT=5Results and how to read them: PERFORMANCE_ANALYSIS.md.
# Run linter
make lint
# Format code
make fmtAll database commands work with any supported database (PostgreSQL, MySQL, SQLite).
Simply configure DB_DRIVER in .env.local and use universal commands:
# Show current database configuration
make db-info
# Create a new migration for current DB_DRIVER
make db-migrate-create name="your_migration_name"
# Check migration status
make db-status
# Apply migrations
make db-up
# Rollback last migration
make db-down
# Regenerate database models for the current DB_DRIVER
make db-models
# Regenerate database models for all databases
make all-db-models
# Docker Compose commands for local development
make docker-up # Start PostgreSQL and MySQL with Docker Compose
make docker-pg-up # Start only PostgreSQL
make docker-mysql-up # Start only MySQL
make docker-down # Stop and remove all containers
make docker-down-volumes # Stop and remove containers with volumes
make docker-logs # Show logs from all containers
make docker-ps # Show status of containersFor complete database setup guide, migrations structure, and Docker Compose configuration, see DATABASE.md.
make mocks.
βββ goque.go # Goque: processor registration, Run/Stop
βββ goque_manager.go # Public API - TaskQueueManager interface
βββ goque_*.go # Public API - tasks, processors, options, values, periodic jobs, tx, metrics, tracer
βββ pkg/
β βββ goquestorage/ # Database connection helper (NewDBConn, driver constants)
βββ internal/
β βββ entity/ # Domain entities and errors
β βββ queuemanager/ # Task queue manager implementation
β βββ metrics/ # Prometheus metrics
β βββ processors/ # Task processing components
β β βββ queueprocessor/ # Main queue processor (plain and typed)
β β βββ periodicprocessor/ # Periodic job scheduler
β β βββ internalprocessors/ # Built-in processors (healer, cleaner)
β βββ storages/ # Data access layer (multi-database support)
β β βββ pg/task/ # PostgreSQL storage (go-jet)
β β βββ mysql/task/ # MySQL storage (go-jet)
β β βββ sqlite/ # SQLite storage (go-jet)
β β βββ dbtx/ # ctx-aware tx executor (WithTx, WithinTx, Executor)
β β βββ dbentity/ # Cross-backend filter/query builders
β β βββ dbutils/ # JSON validation + WHERE builders
β βββ utils/ # Internal helpers (dynamic values, tracing, time, pools)
β βββ pkg/
β βββ generated/ # Generated code (models, mocks)
βββ migrations/ # Database migrations
β βββ pg/ # PostgreSQL migrations
β βββ mysql/ # MySQL migrations
β βββ sqlite/ # SQLite migrations
βββ grafana/ # Ready-to-import Grafana dashboard
βββ examples/service/ # Production-ready example service
βββ DATABASE.md # Complete database setup guide
βββ test/ # Test utilities and fixtures
Contributions are welcome! Please feel free to submit a Pull Request.
This project is licensed under the MIT License - see the LICENSE file for details.