Skip to content
This repository was archived by the owner on Aug 12, 2026. It is now read-only.
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions operator/redisfailover/factory.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,14 +90,29 @@ func NewRedisFailoverRetriever(cfg Config, cli k8s.Services) controller.Retrieve
},
WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
watcher, err := cli.WatchRedisFailovers(context.Background(), "", options)
watcher = watch.Filter(watcher, func(event watch.Event) (watch.Event, bool) {
if err != nil {
// Do not wrap a nil watcher: watch.Filter starts a goroutine
// that dereferences the source watcher's ResultChan, so passing
// the nil returned alongside an error panics the operator. The
// reflector retries the watch on a returned error instead.
return nil, err
}
return watch.Filter(watcher, func(event watch.Event) (watch.Event, bool) {
// Always propagate watch.Error events. Their Object is a
// *metav1.Status, not a *RedisFailover, so they would otherwise
// be dropped by the type assertion below. The reflector relies
// on these events to learn the watch has failed (e.g. an expired
// resource version) and to restart it promptly instead of
// waiting for a connection timeout.
if event.Type == watch.Error {
return event, true
}
rf, ok := event.Object.(*redisfailoverv1.RedisFailover)
if !ok {
return event, false
}
return event, isNamespaceSupported(*rf)
})
return watcher, err
}), nil
Comment on lines +100 to +115

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When filtering watch events, it is crucial to propagate watch.Error events. If a watch.Error event occurs (for example, when a watch resource version is too old and the watch is closed by the API server), the event object will not be of type *redisfailoverv1.RedisFailover. Under the current implementation, this event will be silently filtered out because ok is false, returning event, false.

If the reflector/informer does not receive the watch.Error event, it won't know that the watch has failed and needs to be restarted immediately, potentially causing the operator to hang or delay recovery until a connection timeout occurs.

We should explicitly allow watch.Error events to pass through the filter.

			return watch.Filter(watcher, func(event watch.Event) (watch.Event, bool) {
				if event.Type == watch.Error {
					return event, true
				}
				rf, ok := event.Object.(*redisfailoverv1.RedisFailover)
				if !ok {
					return event, false
				}
				return event, isNamespaceSupported(*rf)
			}), nil

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in 93fdbf30. The filter now passes watch.Error events through (returning event, true) before the *RedisFailover type assertion, so they're no longer dropped. This lets the reflector observe the error via apierrors.FromObject and restart the watch promptly (e.g. on an expired resource version) instead of waiting for the connection to close. I also added a regression test that pushes a watch.Error through a fake watcher and asserts it reaches the filtered channel.

},
})
}
Expand Down
113 changes: 113 additions & 0 deletions operator/redisfailover/factory_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
package redisfailover_test

import (
"context"
"errors"
"testing"
"time"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/watch"

mK8SService "github.com/spotahome/redis-operator/mocks/service/k8s"
rfOperator "github.com/spotahome/redis-operator/operator/redisfailover"
)

// TestRetrieverWatchPropagatesErrorWithoutWrapping is a regression test for the
// operator crashing with a nil pointer dereference when the RedisFailover watch
// could not be established.
//
// The retriever's WatchFunc used to call watch.Filter unconditionally on the
// result of WatchRedisFailovers. On a watch error the typed client returns a
// nil watch.Interface, and watch.Filter immediately spawns a goroutine whose
// loop dereferences the source watcher's ResultChan, panicking the whole
// operator process (SIGSEGV) instead of letting the reflector retry the watch.
//
// The fix returns (nil, err) before wrapping. This test asserts the error is
// propagated and that no filtered watcher is returned (which is what would
// otherwise carry the panicking goroutine).
func TestRetrieverWatchPropagatesErrorWithoutWrapping(t *testing.T) {
assert := assert.New(t)

watchErr := errors.New("the server could not establish the watch")
ms := &mK8SService.Services{}
ms.On("WatchRedisFailovers", mock.Anything, mock.Anything, mock.Anything).
Return(nil, watchErr)

retriever := rfOperator.NewRedisFailoverRetriever(
rfOperator.Config{SupportedNamespacesRegex: ".*"},
ms,
)

var (
w watch.Interface
err error
)
assert.NotPanics(func() {
w, err = retriever.Watch(context.Background(), metav1.ListOptions{})
})
assert.Equal(watchErr, err)
assert.Nil(w, "a nil watcher must not be wrapped by watch.Filter")
ms.AssertExpectations(t)
}

// TestRetrieverWatchWrapsWatcherOnSuccess verifies the happy path still wraps
// the underlying watcher (so namespace filtering stays in effect).
func TestRetrieverWatchWrapsWatcherOnSuccess(t *testing.T) {
assert := assert.New(t)

fake := watch.NewFake()
defer fake.Stop()
ms := &mK8SService.Services{}
ms.On("WatchRedisFailovers", mock.Anything, mock.Anything, mock.Anything).
Return(fake, nil)

retriever := rfOperator.NewRedisFailoverRetriever(
rfOperator.Config{SupportedNamespacesRegex: ".*"},
ms,
)

w, err := retriever.Watch(context.Background(), metav1.ListOptions{})
assert.NoError(err)
assert.NotNil(w)
ms.AssertExpectations(t)
Comment on lines +72 to +75

@coderabbitai coderabbitai Bot May 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Stop the returned watcher in the success-path test to avoid goroutine leakage.

retriever.Watch(...) returns the filtered watcher; stopping it explicitly makes test lifecycle deterministic and avoids lingering goroutines in larger suites.

Suggested patch
 w, err := retriever.Watch(context.Background(), metav1.ListOptions{})
 assert.NoError(err)
 assert.NotNil(w)
+if w != nil {
+	defer w.Stop()
+}
 ms.AssertExpectations(t)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
w, err := retriever.Watch(context.Background(), metav1.ListOptions{})
assert.NoError(err)
assert.NotNil(w)
ms.AssertExpectations(t)
w, err := retriever.Watch(context.Background(), metav1.ListOptions{})
assert.NoError(err)
assert.NotNil(w)
if w != nil {
defer w.Stop()
}
ms.AssertExpectations(t)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@operator/redisfailover/factory_test.go` around lines 71 - 74, The test
calling retriever.Watch(context.Background(), metav1.ListOptions{}) stores the
returned watcher in w but never stops it, causing goroutine leakage; update the
test to stop the watcher in the success path by calling w.Stop() (or defer
w.Stop() right after ensuring err==nil and w!=nil) so the watcher is closed
before asserting expectations (references: retriever.Watch and variable w).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, but I think this is redundant here. filteredWatch.Stop() just delegates to the source watcher's Stop(), and the filter goroutine ranges over the source's ResultChan(), so it exits as soon as the source closes. The test already has defer fake.Stop() (line 61), which closes that channel and terminates the goroutine — adding defer w.Stop() would only call the same fake.Stop() again. There's no goroutine left to leak, so I'll keep the test as is.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Arsolitt, that's a fair and accurate analysis — watch.Filter starts a goroutine that ranges over w.ResultChan() and exits naturally once the source channel is closed, so defer fake.Stop() on line 61 is already the correct cleanup point. Calling defer w.Stop() would just re-invoke the same underlying fake.Stop(). Acknowledged, happy to keep the test as-is.

}

// TestRetrieverWatchPropagatesWatchErrorEvents verifies that watch.Error events
// are not swallowed by the namespace filter. Their Object is a *metav1.Status
// (not a *RedisFailover), so without an explicit passthrough the filter would
// drop them and the reflector would never learn the watch must be restarted.
func TestRetrieverWatchPropagatesWatchErrorEvents(t *testing.T) {
assert := assert.New(t)

fake := watch.NewFake()
defer fake.Stop()
ms := &mK8SService.Services{}
ms.On("WatchRedisFailovers", mock.Anything, mock.Anything, mock.Anything).
Return(fake, nil)

retriever := rfOperator.NewRedisFailoverRetriever(
rfOperator.Config{SupportedNamespacesRegex: ".*"},
ms,
)

w, err := retriever.Watch(context.Background(), metav1.ListOptions{})
assert.NoError(err)
assert.NotNil(w)

errStatus := &metav1.Status{Status: metav1.StatusFailure, Reason: metav1.StatusReasonExpired}
go fake.Error(errStatus)

select {
case event, ok := <-w.ResultChan():
assert.True(ok, "result channel must stay open for watch.Error events")
assert.Equal(watch.Error, event.Type, "watch.Error events must pass through the filter")
assert.Equal(errStatus, event.Object)
case <-time.After(time.Second):
assert.Fail("timed out waiting for the watch.Error event to be propagated")
}

ms.AssertExpectations(t)
}