TheAfriForge/gitforge
1
0
Fork 0
mirror of https://codeberg.org/forgejo/forgejo.git synced 2026-07-18 23:47:50 +00:00
Code Issues Projects Releases Packages Wiki Activity

fix: reduce deadlocks merging PRs w/ async label stat recalcs (#9868)

Browse source 
The intent of this change is to reduce the scope of deadlock issues identified in #9785.  I've identified other deadlock issues from synthetic testing, so this is not a complete fix, but it's a partial fix.  This design was discussed in #9785 and this is the most basic implementation, with a very small scope of work converted to use it.

Introduces a new `forgejo.org/services/stats` module which allows for the queuing and routing of recalc requests for object stats; in this case, the "number of issues" that are assigned to a label, and the number of closed issues that are assigned to a label.

The reasons that these calculations are performed asynchronously through a queue are:
- User operations that are common and performance-sensitive don't have to wait for recalculations that don't need to be exactly up-to-date at all times.  For example, merging a pull request will be a faster operation; as it closes an issue, it needs to recalculate `label.num_closed_issues` for every label attached to the PR.

- Database deadlocks that can occur between concurrent operations -- for example, if you were holding a lock on an issue while recalculating a label's count of open issues -- can be broken by making the recalculation occur outside of the transaction.

## Checklist

The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org).

### Tests

- I added test coverage for Go changes...
  - [x] in their respective `*_test.go` for unit tests.
  - [ ] in the `tests/integration` directory if it involves interactions with a live Forgejo server.
- I added test coverage for JavaScript changes...
  - [ ] in `web_src/js/*.test.js` if it can be unit tested.
  - [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)).

### Documentation

- [ ] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change.
- [x] I did not document these changes and I do not expect someone else to do it.
    - Internal developer documentation is present.

### Release notes

- [ ] I do not want this change to show in the release notes.
- [ ] I want the title to show in the release notes with a link to this pull request.
- [x] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/9868
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Co-authored-by: Mathieu Fenniak <mathieu@fenniak.net>
Co-committed-by: Mathieu Fenniak <mathieu@fenniak.net>
This commit is contained in:
Mathieu Fenniak 2025-10-31 02:12:36 +01:00 • committed by Mathieu Fenniak
commit 9e07bb07be
20 changed files with 467 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

23
modules/queue/base_channel.go
View file

@ -10,6 +10,7 @@ import (
"time"
"forgejo.org/modules/container"
"forgejo.org/modules/log"
)
var errChannelClosed = errors.New("channel is closed")
@ -47,8 +48,9 @@ func (q *baseChannel) PushItem(ctx context.Context, data []byte) error {
if q.isUnique {
q.mu.Lock()
defer q.mu.Unlock()
has := q.set.Contains(string(data))
q.mu.Unlock()
if has {
return ErrAlreadyInQueue
}
@ -57,9 +59,10 @@ func (q *baseChannel) PushItem(ctx context.Context, data []byte) error {
select {
case q.c <- data:
if q.isUnique {
q.mu.Lock()
q.set.Add(string(data))
q.mu.Unlock()
added := q.set.Add(string(data))
if !added {
log.Error("synchronization failure; data could not be added to tracking set in unique queue")
}
}
return nil
case <-time.After(pushBlockTime):
@ -76,7 +79,10 @@ func (q *baseChannel) PopItem(ctx context.Context) ([]byte, error) {
return nil, errChannelClosed
}
q.mu.Lock()
q.set.Remove(string(data))
removed := q.set.Remove(string(data))
if !removed {
log.Error("synchronization failure; data could not be removed from tracking set in unique queue")
}
q.mu.Unlock()
return data, nil
case <-ctx.Done():
@ -85,18 +91,15 @@ func (q *baseChannel) PopItem(ctx context.Context) ([]byte, error) {
}
func (q *baseChannel) HasItem(ctx context.Context, data []byte) (bool, error) {
q.mu.Lock()
defer q.mu.Unlock()
if !q.isUnique {
return false, nil
}
q.mu.Lock()
defer q.mu.Unlock()
return q.set.Contains(string(data)), nil
}
func (q *baseChannel) Len(ctx context.Context) (int, error) {
q.mu.Lock()
defer q.mu.Unlock()
if q.c == nil {
return 0, errChannelClosed
}