Skip to main content

A/B Testing for RSS Triggers

A/B testing helps you optimize notification content by testing multiple variants with a subset of subscribers before sending the winning version to everyone.

How It Works​

When an RSS trigger with A/B testing enabled detects a new item:

  1. AI Generation - The system generates multiple notification variants using AI
  2. Pending Review - Test enters a review queue for your approval
  3. Your Review - You can edit variants or discard the test
  4. Test Phase - A percentage of subscribers receive different variants
  5. Winner Selection - Best performer is selected (automatically or manually)
  6. Rollout - Remaining subscribers receive the winning variant

Setting Up A/B Testing​

Enable on RSS Trigger​

  1. Go to Triggers → RSS Feed → Configure
  2. Check Enable A/B Testing
  3. Configure settings:
SettingDescriptionRecommendation
VariantsNumber of versions (2, 3, or 4)Start with 2
Test %Subscribers for testing (10-50%)10-20%
Auto-SelectAutomatically pick winnerEnable for efficiency
Delay HoursTime before auto-select (2, 4, 6, 12, 24, 48, or 72 hours)2-4 hours

The Review Queue​

Accessing the Queue​

When A/B tests are pending review:

  1. Navigate to A/B Tests
  2. Click RSS Queue button (shows count of pending tests)
  3. View all tests awaiting your action

The RSS Review Queue showing pending automated A/B tests

Queue Actions​

For each pending test, you can:

ActionResult
Quick ApproveStart test immediately with current variants
Review & EditOpen detailed editor to modify variants
DiscardSkip this article, no notification sent

Reviewing and Editing Variants​

The Review Page​

Shows:

  • Source Article - Original RSS content
  • Generated Variants - AI-created notification versions
  • Test Settings - Configuration summary

Reviewing and editing AI-generated variants for an RSS A/B test

Editing Variants​

For each variant, you can edit:

  • Title (max 100 characters)
  • Body (max 200 characters)
  • URL (optional override)
  • Image URL (optional override)

Character counters help you stay within limits.

Regenerating Variants​

If the AI variants aren't suitable:

  1. Click Regenerate Variants
  2. Wait for new variants to be generated
  3. Review and edit as needed

Test Lifecycle​

Status Flow​

PENDING_REVIEW → DRAFT → TESTING → SELECTING_WINNER → COMPLETED
       ↓           ↓         ↓
   CANCELLED   CANCELLED  CANCELLED
StatusDescription
Pending ReviewAI variants generated, awaiting approval
DraftApproved but not yet started
TestingVariants being sent to test subscribers
Selecting WinnerTest phase complete, awaiting selection
CompletedWinner rolled out to all subscribers
CancelledTest cancelled, no notifications sent

Viewing Test Results​

The Test Details Page​

After approving, view detailed metrics:

Test Information​

  • Test name and article source
  • Configuration (percentage, variants)
  • Subscriber counts

Variant Performance​

For each variant:

MetricDescription
SubscribersNumber who received this variant
ClicksTotal clicks on this variant
Click RatePercentage who clicked
Progress BarVisual comparison

Viewing live performance metrics and click rates for RSS A/B test variants

Identifying the Winner​

The variant with the highest click rate typically wins. Consider:

  • Statistical significance (enough data?)
  • Click quality (right landing page?)
  • Total volume vs. rate

Selecting a Winner​

Automatic Selection​

If Auto-Select Winner is enabled:

  1. System waits for the configured delay
  2. Picks the variant with highest click rate
  3. Sends to remaining subscribers automatically

Manual Selection​

If auto-select is disabled or you want to choose early:

  1. Open the test details page
  2. Review variant performance
  3. Click Select Winner on your chosen variant
  4. Confirm the selection

Manually selecting the winning variant for an RSS A/B test

Managing Tests​

Start a Test​

From the A/B Tests list, click on a Draft test and click Start Test.

Cancel a Test​

You can cancel tests that are:

  • In Draft status (not yet started)
  • In Testing status (will stop, no rollout)

Cancelled tests don't send any further notifications.

Delete a Draft​

Draft tests can be deleted entirely if you don't want to run them.

Best Practices​

Variant Strategy​

Do:

  • Make meaningful differences between variants
  • Test one element at a time (title vs. different title)
  • Use clear, concise language

Don't:

  • Make variants too similar
  • Test multiple changes at once
  • Ignore the AI suggestions entirely

Test Percentage​

PercentageBest For
10%Large subscriber bases, cautious testing
20%Balanced testing and reach
30-50%Small subscriber bases, faster results

Timing​

DelayBest For
1-2 hoursTime-sensitive news
4-6 hoursStandard content
12-24 hoursEvergreen content

Analysis​

  • Wait for sufficient clicks before drawing conclusions
  • Consider time of day effects
  • Look at click rate, not just total clicks
  • Review multiple tests for patterns

Troubleshooting​

No Variants Generated​

  • Check AI service availability
  • Verify RSS content has title and description
  • Contact support if persistent

Test Not Starting​

  • Ensure you have subscribers
  • Check if Web Push channel is configured
  • Verify app has the RSS trigger active

Results Seem Wrong​

  • Verify subscribers are evenly distributed
  • Check for delivery failures
  • Ensure enough time has passed

Example Workflow​

  1. 9:00 AM - New blog post published
  2. 9:15 AM - RSS trigger detects post
  3. 9:16 AM - AI generates 3 variants
  4. 9:16 AM - Test enters Pending Review queue
  5. 9:30 AM - You review and approve
  6. 9:30 AM - Test starts, 20% of subscribers get variants
  7. 1:30 PM - Auto-select triggered (4 hours later)
  8. 1:31 PM - Winner identified (Variant B: 4.2% click rate)
  9. 1:31 PM - Remaining 80% of subscribers get Variant B

Next Steps​