Skip to main content

Troubleshooting

Common issues and solutions when using UnfoldCI.

Installation Issues​

GitHub App Installation Failed​

Problem: Error during GitHub App installation.

Solutions:

  1. Verify you have admin access to the repository or organization
  2. Check organization settings allow GitHub Apps
  3. Try installing on a personal repository first
  4. Clear browser cache and try again

Repositories Not Showing in Dashboard​

Problem: Dashboard shows no repositories after installation.

Solutions:

  1. Click the Sync Repos button in the dashboard
  2. Verify installation at GitHub Settings → Applications
  3. Ensure at least one repository is selected in the installation
  4. Wait up to 30 seconds for initial sync

Can't Access Dashboard​

Problem: Unable to login to app.unfoldci.com.

Solutions:

  1. Use "Sign in with GitHub" (the GitHub App must be installed first)
  2. If redirected to install, complete the installation process
  3. Clear cookies for app.unfoldci.com
  4. Try incognito/private browsing mode

Workflow Issues​

No Test Results Found​

Problem: Action logs show "No test result files found".

Solutions:

  1. Most frameworks work automatically! UnfoldCI auto-detects JUnit XML from 25+ common locations (Jest, pytest, Vitest, Go, Maven, Gradle, Mocha, PHPUnit, xUnit, etc.)

  2. Check the action logs—they show helpful tips for configuring your specific framework

  3. If using a non-standard location, specify it:

    with:
      results-path: 'my-custom-path/**/*.xml'

  4. Add a debug step to list files:

    - name: Debug test files
      run: |
        echo "Looking for XML files:"
        find . -name "*.xml" -type f | grep -v node_modules

  5. Ensure tests run before the UnfoldCI action

API Key Not Working​

Problem: Action logs show authentication errors.

Solutions:

  1. Verify the secret name is exactly FLAKY_AUTOPILOT_KEY
  2. Check the secret exists in repository settings
  3. Regenerate the API key from the dashboard
  4. Ensure no extra whitespace in the secret value

Tests Not Appearing in Dashboard​

Problem: Workflow succeeds but no data in dashboard.

Solutions:

  1. Verify the action completed successfully (check logs)
  2. Confirm test result files exist and contain valid XML
  3. Check the action outputs for any warnings
  4. Refresh the dashboard page

Notification Issues​

Slack Notifications Not Arriving​

Problem: Configured Slack but no notifications appear.

Solutions:

  1. Verify the webhook URL is correct and complete
  2. Test the connection using the Send button in settings
  3. Check the Slack channel still exists
  4. Verify the webhook hasn't been deleted from Slack settings
  5. Ensure notification events are enabled

Discord Webhook Invalid Error​

Problem: "Invalid Discord webhook URL format" error.

Solutions:

  1. Ensure the URL matches the format: https://discord.com/api/webhooks/ID/TOKEN
  2. Legacy discordapp.com URLs are also supported
  3. Copy the full URL without extra characters
  4. Regenerate the webhook in Discord if needed

Teams Notifications Failing​

Problem: Microsoft Teams notifications not working.

Solutions:

  1. Verify the webhook URL contains webhook.office.com
  2. Check the connector is still active in Teams
  3. Ensure the channel permissions allow incoming webhooks
  4. Test with a simpler webhook first

Duplicate Notifications​

Problem: Receiving multiple notifications for the same event.

Solutions:

  1. Check if multiple installations have the same webhook
  2. Verify only one set of credentials is configured
  3. Review enabled events for duplicates

Test Result Issues​

Tests Showing Wrong File Path​

Problem: Test file paths don't match your repository structure.

Solutions:

The action attempts to extract file paths from JUnit XML. If paths are incorrect:

  1. Configure your test framework to include file paths:

    Jest — Use --testLocationInResults flag

    pytest — Paths are automatically included

  2. Create .flaky-autopilot.json with path aliases if using custom imports

Code Hash Not Calculating​

Problem: Tests don't show code change detection.

Solutions:

  1. Ensure test files exist in the repository
  2. Check import paths are resolvable
  3. Add .flaky-autopilot.json with correct aliases:
{
  "importResolver": {
    "aliases": {
      "@": "./src"
    }
  }
}

Flake Detection Issues​

No Flaky Tests Detected​

Problem: Known flaky tests not being flagged.

Solutions:

  1. Flake detection requires 5+ test runs minimum
  2. Verify tests are actually failing intermittently (not consistently)
  3. Check test details in dashboard for run history
  4. Wait for more CI runs to gather data

PR Not Created for Flaky Test​

Problem: Flaky test detected but no fix PR generated.

Reasons:

  1. AI confidence below threshold
  2. Root cause analysis still in progress
  3. Similar PR already exists
  4. Test was recently analyzed

Solutions:

  1. Check test details page for analysis status
  2. Click "Reanalyze" to trigger fresh analysis
  3. Review existing PRs in the repository

Dashboard Issues​

Data Not Updating​

Problem: Dashboard shows stale data.

Solutions:

  1. Hard refresh the page (Ctrl+Shift+R or Cmd+Shift+R)
  2. Click "Sync Repos" button
  3. Verify recent workflow runs completed successfully
  4. Check browser console for errors

Settings Page Shows No Installation​

Problem: Settings page says "No Installation Found".

Solutions:

  1. Verify the GitHub App is installed
  2. Click "Sync Repos" on the main dashboard
  3. Sign out and sign back in
  4. Reinstall the GitHub App if necessary

CI Savings Not Showing​

Problem: CI Savings card shows zero or no data.

Solutions:

  1. Ensure you have at least 10 test runs recorded
  2. Wait for flaky tests to be detected and fixed
  3. Savings accumulate over time based on actual reruns avoided

Pattern Analysis Empty​

Problem: Pattern Analysis section shows no data.

Solutions:

  1. Verify the repository has test files
  2. Check that test files are accessible (not in .gitignore)
  3. Pattern analysis runs on first test result submission
  4. Click "Reanalyze" on the repository page

Common Error Messages​

"No test result files found"​

Problem: Action logs show no XML files were found.

Solutions:

  1. Most common: UnfoldCI auto-detects JUnit XML from 25+ common locations. If your tests output to a non-standard location, specify it:

    with:
      results-path: 'my-custom-path/**/*.xml'

  2. Verify your tests actually output JUnit XML format (see Configuration for framework-specific setup)

  3. Check the action logs—they show helpful tips for your specific framework

"Usage limit exceeded"​

Solution: Check your plan limits in the dashboard. The workflow will continue normally.

"API error"​

Solution: Transient issue. The workflow continues without blocking. Check logs for details.

"Invalid webhook URL format"​

Solution: Verify the webhook URL format matches the expected pattern for the platform:

  • Slack: https://hooks.slack.com/services/...
  • Discord: https://discord.com/api/webhooks/...
  • Teams: Contains webhook.office.com

Debugging Tips​

Enable Verbose Logging​

Add debug steps to your workflow:

- name: Debug environment
  run: |
    echo "Repository: ${{ github.repository }}"
    echo "Commit: ${{ github.sha }}"
    echo "Branch: ${{ github.ref_name }}"
    
- name: List test files
  run: find . -name "*.xml" -type f | head -20
    
- name: Show test file content
  run: head -50 test-results/junit.xml || echo "File not found"

Check Action Outputs​

- name: Analyze Flaky Tests
  id: flaky
  uses: UnfoldAI-Labs/UnfoldCI-flaky-autopilot-action@v1
  # ...

- name: Debug outputs
  if: always()
  run: |
    echo "Status: ${{ steps.flaky.outputs.status }}"
    echo "Tests: ${{ steps.flaky.outputs.tests_analyzed }}"
    echo "Flaky: ${{ steps.flaky.outputs.flakes_detected }}"

Getting Help​

If you're still experiencing issues:

  1. Check the GitHub Action logs for detailed error messages
  2. Review your workflow configuration against the Quick Start Guide
  3. Visit the dashboard to verify installation status

Information to Gather​

When seeking help, prepare:

  • Workflow file contents
  • Action log output
  • Dashboard screenshot
  • Repository name (if shareable)
  • Error messages

Repository Visibility Issues​

Can't See My Repositories​

Problem: Expected repositories not showing in dashboard.

Solutions:

  1. Check Installation Scope: Repositories only appear if the GitHub App is installed where they exist:

    • Personal repos → Install app on your personal account
    • Org repos → Install app on that organization

  2. Verify Repository Access:

  3. Click Sync Repos in the dashboard to refresh

Only See Organization Repos (Not Personal)​

Problem: Dashboard shows org repos but not personal repos.

Solution: The app is only installed on your organization, not your personal account. To see personal repos:

  1. Visit github.com/apps/unfoldci-flaky-test-autopilot
  2. Click "Install"
  3. Select your personal account
  4. Choose which personal repos to monitor

Only See Personal Repos (Not Organization)​

Problem: Dashboard shows personal repos but not org repos.

Solution: Install the app on your organization:

  1. Visit github.com/apps/unfoldci-flaky-test-autopilot
  2. Click "Install"
  3. Select your organization
  4. Choose which org repos to monitor
tip

You can have the app installed on multiple accounts (personal + multiple orgs). Each installation appears unified in your dashboard.

Authentication Issues​

Error 403 on Dashboard​

Problem: Getting "403 Forbidden" when accessing the dashboard.

Causes:

  • Your GitHub App installation was removed/reinstalled
  • Session token is linked to an old installation ID
  • Your account or installation was suspended

Solutions:

  1. Clear Session: Navigate to app.unfoldci.com/signout
  2. Force Re-login: Navigate to app.unfoldci.com/signin?force=true
  3. Sign in again with GitHub

Stuck on Sign-In Page​

Problem: Sign-in page redirects back to itself or shows errors.

Solutions:

  1. Clear browser cookies for app.unfoldci.com
  2. Navigate to app.unfoldci.com/signout first
  3. Then go to app.unfoldci.com/signin
  4. Try incognito/private browsing mode

Session Expired After Reinstalling App​

Problem: Deleted and reinstalled the GitHub App, now can't access dashboard.

Explanation: When you reinstall the app, you get a new installation_id. Your browser has an old session token.

Solution:

  1. Go to app.unfoldci.com/signout
  2. Sign in again with GitHub
  3. A new session will be created with your new installation

Can't Sign In on Mobile​

Problem: Desktop works but mobile shows 403 error.

Explanation: Each browser/device has its own session. Mobile may have stale credentials.

Solution: On your mobile browser:

  1. Clear site data for app.unfoldci.com
  2. Navigate to app.unfoldci.com/signin?force=true
  3. Sign in with GitHub

Tests Pass Locally But Fail in CI​

This is exactly the type of issue UnfoldCI helps detect and fix!

Common Causes​

CauseSymptomsSolution
Timing IssuesTests fail randomly, often in setTimeout/async codeAdd proper awaits, increase timeouts
Race ConditionsDifferent failure messages each runAdd synchronization, use locks
Environment DifferencesMissing env vars, different OS pathsMatch CI environment locally
Network FlakinessExternal API call failuresMock external services in tests
Resource Exhaustion"Out of memory" or slow testsRun tests in isolation, clean up after tests
Timezone/LocaleDate/time assertions failUse UTC, mock time
Parallel Test InterferenceTests pass alone, fail in parallelDon't share global state

UnfoldCI Helps​

Once UnfoldCI detects these patterns:

  1. Dashboard shows which tests are flaky with failure patterns
  2. AI Analysis identifies the root cause category
  3. Automatic PRs suggest fixes based on detected patterns

FAQ​

Q: Does UnfoldCI need any special GitHub permissions?

A: No! UnfoldCI only needs read access to your test result files. The GitHub App handles all repository interactions (creating fix PRs, etc.) using its own authentication.

Q: Why does my CI show green even when tests fail?

A: If you're using continue-on-error: true on your test step, CI will always pass. This is useful for UnfoldCI analysis but can hide failures.

For accurate CI status, either:

  1. Remove continue-on-error and use if: always() on the UnfoldCI step
  2. Add a final "check results" step (see Configuration Guide)

Q: How many test runs are needed for flake detection?

A: Minimum 5 runs. Detection improves with more data.

Q: Can I use UnfoldCI with private repositories?

A: Yes. The GitHub App supports private repositories.

Q: What happens if the API is unavailable?

A: The action logs a warning and completes successfully. Your CI pipeline is never blocked.

Q: Are my test results stored?

A: Yes, test outcomes are stored for flake analysis. Source code is not stored.

Q: Can I adjust flake detection sensitivity?

A: Detection thresholds are automatically calibrated based on your test patterns.

Q: How are CI savings calculated?

A: Based on average test duration, flaky test frequency, and estimated rerun costs.

Q: Can I use multiple notification platforms?

A: Yes. Configure Slack, Discord, and Teams simultaneously if desired.

Q: Are notifications sent for all repositories?

A: Yes, notification settings apply to all repositories in your GitHub App installation.

Q: Why do I see repos from multiple organizations?

A: The dashboard shows all repos from all your GitHub App installations. If you've installed UnfoldCI on your personal account AND your organization, you'll see repos from both.

Q: Do team members see the same data?

A: Yes. All members of an organization can see that organization's repository data when they sign in with their GitHub accounts.

Q: Do I need a separate API key for each organization?

A: Yes. Each GitHub App installation (personal or org) has its own API key. Generate keys in Settings for each installation.

Q: Why didn't I get a PR after AI analysis completed?

A: PRs are only created when certain conditions are met:

  • High-confidence categories (missing await, timeout issues, browser timing) always get PRs if confidence ≥ 70%
  • Medium-confidence categories (network issues, timing dependencies) need confidence ≥ 85%
  • Low-confidence categories (race conditions, complex state issues) receive analysis only—no automatic PR
  • Tests must have a pass rate between 5-95% (not consistently failing or stable)

Q: My test stopped getting PRs after 3 attempts. Why?

A: UnfoldCI limits automatic fix attempts to 3 PRs per test to prevent spam. If the first 3 fixes didn't resolve the flakiness, the issue likely requires manual investigation. You can:

  • Review the AI analysis in the dashboard for insights
  • Examine why previous PRs didn't work
  • Manually implement a fix based on the root cause analysis

Q: Why is there a 24-hour delay between PR creations?

A: After creating a PR for a flaky test, UnfoldCI waits 24 hours before generating another PR for the same test. This gives you time to:

  • Review and merge the PR
  • Run CI to verify the fix works
  • See if the test stabilizes

This prevents PR spam if tests remain flaky after an initial fix attempt.