Flagging

Learn how to let users flag interesting or unexpected outputs in your Gradio interfaces

The “Flag” button allows users to mark inputs with interesting outputs, such as erroneous or unexpected model behavior, for you to review later.

How flagging works

When a user flags data in your Interface:

1. A CSV file logs the flagged inputs and outputs
2. If the interface involves file data (images, audio, etc.), folders are created to store those files
3. All flagged data is saved in the directory specified by flagging_dir

Basic flagging setup

The flag button appears by default in your Interface. You can control its behavior:

import gradio as gr

def calculator(num1, operation, num2):
    if operation == "add":
        return num1 + num2
    elif operation == "subtract":
        return num1 - num2
    elif operation == "multiply":
        return num1 * num2
    elif operation == "divide":
        if num2 == 0:
            raise gr.Error("Cannot divide by zero!")
        return num1 / num2

demo = gr.Interface(
    calculator,
    inputs=["number", gr.Radio(["add", "subtract", "multiply", "divide"]), "number"],
    outputs="number",
    flagging_mode="manual"  # Users click flag button to flag
)

demo.launch()

Flagging modes

You can control when and how flagging occurs with the flagging_mode parameter:

manual (default)

Users see a flag button and must click it to flag data:

gr.Interface(
    fn=model,
    inputs=[...],
    outputs=[...],
    flagging_mode="manual"
)

auto

Every submission is automatically flagged along with its output:

gr.Interface(
    fn=model,
    inputs=[...],
    outputs=[...],
    flagging_mode="auto"
)

Users won’t see a flag button - all interactions are logged automatically.

never

Flagging is completely disabled:

gr.Interface(
    fn=model,
    inputs=[...],
    outputs=[...],
    flagging_mode="never"
)

You can also set the GRADIO_FLAGGING_MODE environment variable to control flagging mode globally.

Flagged data structure

For a simple calculator interface, flagged data is stored like this:

+-- calculator.py
+-- flagged/
|   +-- logs.csv

flagged/logs.csv:

num1,operation,num2,Output
5,add,7,12
6,subtract,1.5,4.5

For interfaces with file inputs/outputs (like images), the structure includes data folders:

+-- sepia.py
+-- flagged/
|   +-- logs.csv
|   +-- im/
|   |   +-- 0.png
|   |   +-- 1.png
|   +-- Output/
|   |   +-- 0.png
|   |   +-- 1.png

flagged/logs.csv:

im,Output
im/0.png,Output/0.png
im/1.png,Output/1.png

Custom flagging directory

Specify where flagged data is stored:

gr.Interface(
    fn=model,
    inputs=[...],
    outputs=[...],
    flagging_dir="./my_flagged_data"
)

The directory is created automatically if it doesn’t exist.

Flagging options

Allow users to provide a reason when flagging by passing a list of strings to flagging_options:

import gradio as gr

def image_classifier(img):
    return {"cat": 0.7, "dog": 0.3}

demo = gr.Interface(
    image_classifier,
    inputs="image",
    outputs="label",
    flagging_mode="manual",
    flagging_options=["Incorrect", "Offensive", "Other"]
)

demo.launch()

Users select one option when flagging, and it’s saved as an additional column in the CSV:

im,Output,flag_option
im/0.png,"{'cat': 0.7, 'dog': 0.3}",Incorrect
im/1.png,"{'cat': 0.2, 'dog': 0.8}",Other

Custom flag labels and values

You can customize both the button labels and the values stored in the CSV:

gr.Interface(
    fn=model,
    inputs=[...],
    outputs=[...],
    flagging_options=[
        ("Mark as incorrect", "incorrect"),
        ("Mark as offensive", "offensive"),
        ("Mark as problematic", "problematic")
    ]
)

The first element of each tuple is the button label shown to users, and the second is the value stored in the CSV.

Custom flagging callbacks

By default, Gradio uses CSVLogger to save flagged data. You can create custom flagging behavior by providing a FlaggingCallback subclass:

import gradio as gr
from gradio.flagging import FlaggingCallback

class SimpleFlaggingCallback(FlaggingCallback):
    def setup(self, components, flagging_dir):
        self.components = components
        self.flagging_dir = flagging_dir
        # Setup logic (create directories, initialize database, etc.)
    
    def flag(self, flag_data, flag_option=None, username=None):
        # Custom flagging logic (save to database, send notification, etc.)
        print(f"Flagged: {flag_data}")
        if flag_option:
            print(f"Reason: {flag_option}")
        return len(flag_data)  # Return count of flagged samples

demo = gr.Interface(
    fn=model,
    inputs="image",
    outputs="label",
    flagging_callback=SimpleFlaggingCallback()
)

demo.launch()

FlaggingCallback interface

Your custom callback must implement two methods:

setup(components, flagging_dir)

• Called once at the beginning of Interface.launch()
• components: List of input/output components
• flagging_dir: Directory path for storing flagged data

flag(flag_data, flag_option=None, username=None)

• Called every time the flag button is pressed
• flag_data: List of data to be flagged
• flag_option: Selected option if flagging_options provided
• username: Username if user is logged in
• Returns: Total number of flagged samples (int)

SimpleCSVLogger example

Gradio provides SimpleCSVLogger as a basic example:

import gradio as gr

def image_classifier(inp):
    return {'cat': 0.3, 'dog': 0.7}

demo = gr.Interface(
    fn=image_classifier,
    inputs="image",
    outputs="label",
    flagging_callback=gr.flagging.SimpleCSVLogger()
)

demo.launch()

This is a simplified implementation provided for illustrative purposes. The default CSVLogger has more features.

Loading flagged data as examples

You can use flagged data as examples by pointing to the flagged directory:

gr.Interface(
    fn=model,
    inputs=[...],
    outputs=[...],
    examples="./flagged"  # Load from flagged directory
)

This is helpful for reviewing and demonstrating edge cases.

Next steps

Interface class

Learn more about the Interface class

Examples

Working with examples in interfaces