Changelog
Last updated
Was this helpful?
Last updated
Was this helpful?
job-scheduler: consider removing current job from wait, paused or prioritized () ()
delayed: add marker once when promoting delayed jobs () (python) ()
repeat: use JobPro class when creating delayed job () ()
worker: do not execute run method when no processor is defined when resuming () ()
worker: do not resume when closing () ()
job: set processedBy when moving job to active in moveToFinished () fixes ()
drain: pass delayed key for redis cluster () ()
job-scheduler: restore limit option to be saved () ()
job-scheduler: return undefined in getJobScheduler when it does not exist () fixes ()
fix return type of getNextJob ()
worker: markers use now a dedicated key in redis instead of using a special Job ID.
stalled: remove type check in stalled key
references:
priority: priority is separated in its own zset, no duplication needed
change job method name update to updateData
Change global rate limit Move jobs to wait or groups when global rate limit
Remove QueueSchedulerPro class. WorkerPro class should handle QueueSchedulerPro functionalities.
compat: The compatibility class for Bullv3 is no longer available.
Failed and stalled events are now produced by the WorkerPro class instead of by the QueueSchedulerPro.
The minimum Redis recommended version is 6.2.0.
groups: Group ids must be strings. Numbers are not allowed anymore.
Fixes https://github.com/taskforcesh/bullmq-pro-support/issues/25
groups: In order to make the group getter consistent for all statuses we are changing the SET type for groups:active to ZSET. Also we rename the ZSET to groups:max as it represent groups that have maxed the concurrency.
job: add moveToWait method for manual processing () ()
queue: support removeGlobalConcurrency method () ()
add-job: add job into wait or prioritized state when delay is provided as 0 () ()
worker: wait fetched jobs to be processed when closing () ()
job: save processedBy attribute when preparing for processing () ()
groups: support local limiter options () ()
job-scheduler: revert add delayed job and update in the same script ()
worker: evaluate if a job needs to be fetched when moving to failed () ()
retry-job: consider updating failures in job () ()
flow-producer: add support for skipWaitingForReady ()
job-scheduler: save limit option () ()
queue: add option to skip wait until connection ready ()
worker: avoid possible hazard in closing worker ()
queue-getters: add prometheus exporter ()
job-scheduler: save iteration count () ()
sandbox: add support for getChildrenValues ()
worker: add missing otel trace when extending locks () ()
worker: remove the use of multi in extend locks ()
retry-job: pass stalled key instead of limiter () ()
queue: use same telemetry logic in add method as BullMQ () ()
job-scheduler: use delayed job data when template data is not present () fixes ()
job-scheduler: add next delayed job only when prevMillis matches with producerId () ()
job-scheduler: avoid duplicates when upserting in a quick sequence () ()
dynamic-rate-limit: validate job lock cases () ()
job-scheduler: add delayed job and update scheduler in same script () ()
job-scheduler: add delayed job and scheduler in same script () ()
sandbox: fix issue where job could stay in active forever () ()
sandboxed: fix detecting special errors by sending default messages () fixes ()
scripts: make sure jobs fields are not empty before unpack ()
guarantee every repeatable jobs are slotted ()
job-scheduler: avoid duplicated delayed jobs when repeatable jobs are retried ()
job-scheduler: omit deduplication and debounce options from template options () ()
telemetry: add option to omit context propagation on jobs () ()
replace multi by lua scripts in moveToFailed () ()
queue: enhance getJobSchedulers method to include template information () ref ()
worker: catch connection error when moveToActive is called () ()
scheduler-template: remove console.log when getting template information () ()
flow: allow using removeOnFail and failParentOnFailure in parents () fixes ()
job-scheduler: upsert template when same pattern options are provided () ref ()
queue: enhance getJobScheduler method to include template information () ref ()
queue: add getJobSchedulersCount method () ()
stalled: move parent to group when needed () ()
scheduler: remove deprecation warning on immediately option () ()
telemetry: add telemetry support () ()
queue: refactor a protected addJob method allowing telemetry extensions ()
queue: change _jobScheduler from private to protected for extension () ()
scheduler: use Job class from getter for extension () ()
telemetry: do not set span on parent context if undefined ()
job-scheduler: add telemetry support to the job scheduler ()
queue: add rateLimit method () ()
queue: add removeRateLimitKey method () ()
marker: add base markers while consuming jobs to get workers busy () fixes ()
deps: use fixed version of bullmq v5.26.1 () ()
queue: fix generics to be able to properly be extended ()
queue-pro: expose jobs pro in getters ()
improve queue getters to use generic job type () ()
job-scheculer: avoid hazards when upserting job schedulers concurrently ()
connection: do not allow to set blockingConnection option () ()
repeatable: only apply immediately in the first iteration ()
scripts: set package version by default for extension () ()
worker: allow retrieving concurrency value () fixes ()
connection: set packageVersion as protected attribute for extension () ()
queue-events: add QueueEventsProducer for publishing custom events () ()
flows: add telemetry support () ()
deps: bump msgpackr to 1.1.2 to resolve ERR_BUFFER_OUT_OF_BOUNDS error () ref ()
scheduler: add getJobScheduler method () ref ()
queue: add a telemetry interface () ()
proper way to get version ()
scripts: add missing wait in isJobInList ()
redis: use version for naming loaded lua scripts ()
queue: add option to skip metas update ()
queue: add queue version support () ()
sandbox: fix serialization of error with circular references are present () fix ()
worker-pro: use JobPro as part of WorkerProListener events () ()
repeat: also consider startDate when using "every" ()
sandbox: catch exit errors () ()
repeat: deprecate immediately on job scheduler ()
job: expose priority value () ()
job: add deduplication logic () ()
queue: add new upsertJobScheduler, getJobSchedulers and removeJobSchedulers methods ()
worker-fork: allow passing fork options () ()
worker-thread: allow passing Worker options () ref ()
repeatable: avoid delayed job deletion if next job already existed () ()
connection: allow passing connection string into IORedis () ()
metrics: differentiate points in different minutes to be more accurate () (python) ()
pattern: do not save offset when immediately is provided () ()
groups: support changing priority intra groups () ()
queue: add getDebounceJobId method () ()
metrics: save zeros as much as max data points () ()
repeat: replace delayed job when updating repeat key ()
flows: throw error when queueName contains colon () fixes ()
sandboxed: properly update data on wrapped job () fixes ()
flow: remove debounce key when parent is moved to fail () ()
flow: recursive ignoreDependencyOnFailure option () ()
job: throw error if removeDependencyOnFailure and ignoreDependencyOnFailure are used together () ()
job: change moveToFinished return type to reflect jobData () ref ()
stalled: support removeDependencyOnFailure option when job is stalled () ()
fifo-queue: use linked list structure for queue () ()
worker: fix close sequence to reduce risk for open handlers () ()
flow: validate parentData before ignoreDependencyOnFailure when stalled check happens () (python) ()
job: consider passing stackTraceLimit as 0 () ref ()
queue-pro: add getGroupRateLimitTtl method () ()
worker: promote delayed jobs while queue is rate limited () ref ()
flow: get groupId from moveToWaitingChildren script () ()
queue-events: pass debounceId as a param of debounced event () ()
job: allow passing debounce as option () ()
repeatable: remove repeat hash when removing repeatable job () ()
repeatable: keep legacy repeatables if it exists instead of creating one with new structure () ()
repeatable: consider removing legacy repeatable job () fixes ()
repeatable: pass custom key as an args in addRepeatableJob to prevent CROSSSLOT issue () fixes ()
repeatable: new repeatables structure () ref fixes ()
worker: fetch next job on failure () ()
queue: support global concurrency () ()
groups: add getCountsPerPriorityForGroup method () ref ()
get-groups-count: consider other group states apart of waiting () ()
queue-getters: consider passing maxJobs when calling getRateLimitTtl () fixes ()
job: consider changing priority to 0 () ()
priority: consider paused state when calling getCountsPerPriority (python) () ()
priority: use module instead of bit.band to keep order (python) () ()
queue: add getCountsPerPriority method () ()
job: set processedBy using hmset () (python) ()
groups: promote maxed groups after decreasing active count () ()
maxed: consider passing max concurrency into repairMaxedGroup () ()
groups: allow passing limit when calling getGroupsJobsCount () ()
groups: consider removing job from prioritized group when using remove method () ()
worker: properly cancel blocking command during disconnections ()
extendlock,createbulk use pipeline no multi command ()
repeat: throw error when endDate is pointing to the past () ()
retry-job: throw error when job is not in active state () ()
sandboxed: ensure DelayedError is checked in Sandboxed processors () fixes ()
job: validate job existence when adding a log () ()
groups: remove concurrency when removing group () ()
group: add getGroupConcurrency method () ()
worker: make sure clearTimeout is always called after bzpopmin ()
worker: force timeout on bzpopmin command () ()
worker: do not call bzpopmin when blockDelay is lower or equal 0 () ref ()
worker-pro: correct typo in options argument for limiter ()
groups: support for local group concurrency using setGroupConcurrency method () ()
stalled: consider ignoreDependencyOnFailure option (python) () fixes ()
worker: return minimumBlockTimeout depending on redis version (python) () ()
stalled: consider failParentOnFailure when moving child into failed () fixes (python) ()
worker: reset delays after generating blockTimeout value () ()
queue: add getGroupActiveCount method () ()
worker: use 0.002 as minimum timeout for redis version lower than 7.0.8 () fixes ()
allow arbitrary large drainDelay ()
stalled: remove jobId from stalled after removing lock when moved from active () (python) ()
add-to-group: skip reinsertion of group () ()
deps: remove script loader from dist () ()
connection: ignore error when setting custom end status () ()
job: stack trace limit () ()
scripts: use command name in error message when moving to finished () ()
queue: use QueueOptions type in opts attribute () ()
worker: validate drainDelay must be greater than 0 () ()
getters: add getWorkersCount ()
group-limit: set rate limiting even when group is empty () ()
deps: move fast-glob and minimatch as dev-dependencies () ()
worker: set blockTimeout as 0.001 when reach the time to get delayed jobs () fixes ()
job: add removeChildDependency method () ()
deps: replaced glob by fast-glob due to security advisory ()
sandbox: extend SandboxedJob from JobJsonSandbox () fixes ()
add-job: fix parent job cannot be replaced error message () ()
worker: add support for naming workers ()
flow: remove failed children references on auto removal () ()
redis-connection: close redis connection even when initializing () fixes ()
flow: add ignoreDependencyOnFailure option () ()
worker: update processor types () ()
flow: parent job cannot be replaced (python) () ()
reprocess-job: add marker if needed () ()
rate-limit: move job to wait even if ttl is 0 () ()
stalled: consider adding marker when moving job back to wait () ()
retry-jobs: add marker when needed () ()
security: upgrade msgpackr https://github.com/advisories/GHSA-7hpj-7hhx-2fgx ()
worker: worker can be closed if Redis is down () ()
worker: throw error if connection is missing ()
stalled: add command to repair stalled key () ()
repeatable: allow saving custom key () ()
job: provide skipAttempt option when manually moving a job () ()
worker: improved markers handling () ()
marker: differentiate standard and delayed markers (python) () ()
change-delay: add delay marker when needed () ()
flow: add marker when moving parent to wait (python) () ()
move-to-active: check rate limited once () ()
connection: require connection to be passed () ()
job: revert console warn custom job ids when they represent integers () ()
expose sandboxed-job-pro interface ()
groups: add sandbox support for groups ()
groups: remove group when removing last job () ()
batches: differentiate movetoBatchFinished responses () ()
backoff: set marker after adding delayed job () ()
prioritized: get target list once in addPrioritizedJob () ()
groups: consider prioritized groups when maxSize is provided () ()
events: emit drained event only when finishing jobs () ()
flows: update constructor and methods to match queue base () ()
sandboxed: better compatibility with esbuild ()
child-processor: preserve dynamic imports in commonjs ()
flows: add meta key to queues created with flows ()
repeat-strategy: add missing Promise return type () ()
update-progress: remove old updateProgress script to prevent conflict () (python) ()
worker: get dirname by using module.filename () fixes ()
worker: should cap update progress events ()
sandbox: support URL (local files) as processor file ()
queue: add a paginated getDependencies () ()
sandboxes: use the more compatible dynamic import instead of require ()
worker: do not wait for slow jobs fixes ()
utils: use EventEmitter as a type instead of a namespace () ()
job: set delay value on current job instance when it is retried () (python) ()
worker: better handling of concurrency when fetching jobs () ()
rate-limit: continue promoting groups when there are in paused state () ()
queue: improve clean to work iteratively () () ref ()
utils: use EventEmitter as a type instead of a namespace () ()
job: set delay value on current job instance when it is retried () (python) ()
connection: better handling of attached listeners ()
connection: move redis instance check to queue base ()
update delay job property when moving to delayed set () ()
add-job: trim events when waiting-children event is published () (python) ()
worker: keep extending locks while closing workers () ()
sandbox: do not return empty object result when it is undefined () ()
events: do not publish removed event on non-existent jobs () ()
redis-connection: check redis version greater or equal than v6 only once () ()
events: trim events when removing jobs () (python) ()
events: trim events when retrying a job () ()
sandbox: update progress value on job instance () fixes ()
delayed: trim events when moving jobs to delayed (python) () ()
queue: expose addJobLog and updateJobProgress () ()
sandbox: convert wrapJob method as protected for extension () ()
queue: batched unpack now uses range () ()
worker: forward skipVersionCheck to blockingConnection () ref ()
worker: throw exception with NaN as concurrency () ()
queue: differentiate score purpose per state in clean method () fixes ()
connection: provide skipVersionCheck option for shared connections () ref ()
sandbox: emulate moveToDelayed method () () ref
remove: change error message when job is locked (python) () ()
sandbox: ignore extra params on processor () ()
worker: abort rate-limit delay when closing worker () ()
queue: throw error when name is not provided () ()
correct group rate limit in some edge cases () ()
rate-limit: get pttl only if needed () ()
queue: add getRateLimitTtl method () ()
group: add priority option into group option () (), closes
groups: do not move job to paused when promoting rate-limited group () ()
queue: add promoteJobs to promote all delayed jobs ()
job: add option for removing children in remove method (python) () ()
job: add removeDependencyOnFailure option () ()
flow: emit delayed event when parent is moved to delayed () ()
groups: support local priorities () (), closes
add missing error export ()
fix the GroupMaxSizeExceededError prototype ()
add getGroupStatus ()
add support max sized groups ()
pause-group: do not move job to wait when queue is paused () ()
group: move job into group list when paused and dynamic rate limit () ()
rate-limit: emit waiting event in rateLimitGroup () ()
remove-job: do not remove last group id () ()
job: save groupId even when the job is a parent () ()
priority: add prioritized as a new state () ()
ref
rate-limit: keep priority fifo order () fixes (python) ()
worker: set redis version always in initialization () fixes ()
worker: use timeout as integer for redis lower than v6.0.0 (python) () ()
retry-job: consider priority when moving job to wait (python) () ()
job: import right reference of QueueEvents () ()
job: use QueueEvents type for waitUntilFinished () ()
worker: better worker client naming ()
job: add changePriority method () ref ()
rate-limit: consider paused queue () ()
rate-limit: take in count priority () fixes ()
rate-limit: take groups in count in global rate limit counter () ()
job-pro: use saveStacktrace script () ()
remove-job: consider decreasing group concurrency () ()
add support for job batches ()
retry-job: consider promoting delayed jobs () ()
rate-limit: consider groups when global dynamic rate limit () ()
worker: add worker threads support ()
upgrade ioredis to 5.3.2 ()
rate-limit: consider paused queue when dynamic rate limit () ()
retry: consider when queue is paused () ()
worker: close open handles after closing () fixes ()
upstash: don't throw an error when detecting an upstash host () ref ()
flow-producer-pro: fix opts assignment () ()
groups: add repair maxed group function ()
job: avoid error when job is moved when processing () fixes ()
worker: throw error with invalid concurrency fixes ()
worker: close lock extended timer ()
worker: correct lock extender logic ()
worker: start stalled check timer ()
worker: run stalled check directly first time ()
worker: restore failed event job parameter typing () ()
worker: failed event receives an optional job parameter () fixes ()
worker: replace Promise.race with efficient an async fifo () ref ()
worker: simplify lock extension to one call independent of concurrency ()
worker: add remove on complete and fail options () ()
worker: add a public method to run the stalled checker ()
worker: add support to disable stalled checks ()
scripts: reuse keys array to avoid allocations ()
worker: improve worker memory consumption ()
move-to-active: remove deprecated limiter reference () ()
job: check jobKey when saving stacktrace () fixes ()
infinite worker process spawned for invalid JS file ()
worker: add a maximum block time ()
master: copy type declaration ()
redis: increase minimum default retry time ()
connection: apply console.warn in noeviction message ()
error: remove global prototype toJSON () fixes ()
rate-limit: update group concurrency after manual rate-limit ()
job: allow clearing job's log () ()
move-to-finished: return correct delayUntil () ()
worker: fix delayed jobs with concurrency fixes ()
move-to-active: delete marker when it is moved to active () ()
move-to-active: validate next marker and return delayUntil () ()
worker: add max concurrency from the beginning () fixes ()
circular references () ()
performance issues in remove() () ()
delayed: remove marker after being consumed () fixes ()
get-dependencies: replace slow object destructuring with single object () ()
stalled: add activeKey local reference () (), closes
connection: throw exception if using keyPrefix in ioredis ()
connection: use includes to check for upstash more reliably ()
job: fetch parent before job moves to complete () ()
sandbox: throw error when no exported function () fixes ()
queue: add getJobState method () ref ()
job-pro: fix opts type () (), closes
worker: avoid calling run on base class ()
counts: delete delayed marker when needed () ()
get-children-values: replace slow object destructuring with single object () ()
remove-job: check groupId is different than false on removed children () (), closes
exponential: respect exponential backoff delay () ()
get-jobs: filter marker () ()
worker: add ready event for blockingConnection () ()
rate-limit-group: several small fixes related to manual group rate limit. ()
worker: try catch setname call () fixes ()
do not allow move from active to wait if not owner of the job ()
floor pexpire to integer ()
get-workers: set name when ready event in connection () ()
job: console warn custom job ids when they represent integers () ()
add-job: throw error when jobId represents an integer () ()
queue-events: support duplicated event () ()
add-job: do not update job that already exist () ()
rate-limit: delete rateLimiterKey when 0 () ()
remove-job: check for nil as groupId instead of empty string () ()
add support for manually rate-limit groups ()
global-rate-limit: move job into group list () ()
groups: check rate-limit when moving job to active () ()
ttl: throw error when it's not provided as positive number () ()
upgrade bullmq to 3.2.2 () ()
stalled: use type result as table () ()
promote: consider groups () ()
flows: allow parent on root jobs in addBulk method () ref ()
job: send failed event when failParentOnFailure () fixes ()
redis: replace throw exception by console.error ()
connection: validate array of strings in Cluster () fixes ()
worker: clear stalled jobs timer when closing worker ()
getters: fix return type of getJobLogs ()
sandbox: get open port using built-in module instead of get-port () ()
job: update delay value when moving to wait () ()
connection: throw error when no noeviction policy ()
compat: remove Queue3 class () ()
delayed: promote delayed jobs instead of picking one by one ()
delayed: remove marker when promoting delayed job ()
getters: compensate for "mark" job id ()
sandbox: remove progress method ()
stalled-jobs: handle job id 0 ()
worker: do not allow stalledInterval to be less than zero ()
worker: use connection closing to determine closing status ()
redis-connection: allow providing scripts for extension () ()
flow-producer: allow parent opts in root job when adding a flow () ref ()
job-options: add failParentOnFailure option () ()
improve delayed jobs and remove QueueSchedulerPro ()
move stalled jobs check and handling to WorkerPro class from QueueSchedulerPro ()
scripts: pre-build scripts () ()
events: remove data and opts from added event ()
groups: do not parse gid when deserializing jobs fixes ()
delete-groups: consider rate-limit, max concurrency and paused () ()
add getGroupStatus ()
add version support ()
add getGroupsByStatus method to getters ()
delete-group: consider max-concurrency state () ()
delete-group: consider rate-limit state () ()
global-rate-limit: consider groups () ()
drain: consider empty active list () ()
sandbox: support update method () ()
timeout: delete unused option () ()
script-loader: use cache to read script once () ()
concurrency: consider base rate limit () ()
flow-producer-pro: use interim class () ()
add-job: handle parent split on js () ()
delete-group: consider children () ()
deps: upgrade bullmq to 1.90.0 () ()
waiting-children: consider decreasing group concurrency () ()
deps: upgrade bullmq to 1.89.1 () ()
job: remove from group () ()
groups: support flows () ()
deps: upgrade bullmq to 1.87.2 () ()
observables: guarantee store result order ()
observables: store last value as returnvalue ()
deps: upgrade bullmq to 1.87.1 () ()
move-to-finished: pass keepJobs into opts arg () ()
deps: upgrade bullmq to 1.86.10 () ()
move-to-active: use local jobId instead of global reference () ()
deps: upgrade bullmq to 1.86.9 () ()
retry-jobs: add jobs in batches when groupId is present () ()
retry-jobs: consider groups () ()
scripts: use tonumber on timestamp args () ()
deps: upgrade bullmq to 1.86.5 () ()
concurrency: consider retry backoff strategy () ()
job-pro: fix gid parse () ()
job-pro: expose gid value () ()
queue-pro: fix addBulk opts typing () ()
pause-group: return boolean for execution success () ()
groups: rename paused and resumed events in QueueEventsPro () ()
pause-group: allow pausing specific group () ref ()
deps: upgrade bullmq to 1.86.0 () ()
deps: upgrade bullmq to 1.85.4 () ()
worker: use isObservable () ()
deps: upgrade bullmq to 1.83.2 () ()
deps: upgrade bullmq to 1.83.0 () ()
flow-producer: use JobPro instances () ()
get-state: consider checking groups () ()
deps: upgrade bullmq to 1.81.4 () ()
stalled: allow easy transition for stalled changes () ()
deps: upgrade bullmq to 1.80.6 () ()
groups: improve addGroups to return all groups statuses ()
deps: upgrade bullmq to 1.80.4 () ()
worker: allow ttl per job name () ()
deps: upgrade bullmq to 1.78.2 () ()
groups: consider delay option () fixes ()
upgrade bullmq to 1.77.3 () ()
correctly handle stalled jobs when using groups () ()
upgrade bullmq to 1.76.6 () ()
add support for max concurrency per group ()
upgrade bullmq to 1.76.1 () ()
upgrade bullmq to 1.73.0 () ()
upgrade bullmq to 1.67.1 () ()
upgrade bullmq to 1.66.1 () ()
upgrade bullmq to 1.64.0 () ()
groups rate limit () ()
bullmq: use fixed version for 1.55.1 () ()
bullmq: upgrade to 1.55.1 () ()
expose missing classes () ()
point correct main files () ()
bullmq: upgrade to 1.54.5 version () ()
use es6 interpolation ()
rxjs: support observables () ()
add: send missing events in lua script () ()
bullmq: upgrade to 1.54.0 version () ()
npm: replace npm registry ()
bullmq: upgrade bullmq ()
remove nodejs 10 support ()
groups: initial implementation ()
add queue-pro, worker-pro and redis-connection-pro ()
initial commit ()