CodeQL zero to hero part 4: Gradio framework case study

Motivation

CodeQL has models for the majority of most popular libraries and frameworks, and new ones are continuously being added to improve the detection of vulnerabilities.

One such example is Gradio. We will go through the process of analyzing it and modeling it today. Looking into a new framework or a library and modeling it with CodeQL is a perfect chance to do research on the projects using that framework, and potentially find multiple vulnerabilities at once.

Research and CodeQL modeling process

Frameworks have sources and sinks, and that’s what we are interested in identifying, and later modeling in CodeQL. A framework may also provide user input sanitizers, which we are also interested in.

The process consisted, among others, of:

• Reviewing the documentation and code for sources—any entry points to the application, which take user input.
• Reviewing the documentation and code for sinks—any functionality that is potentially dangerous, for example, a function that executes raw SQL queries and that may take user input.
• Dynamically testing interesting elements.
• Checking for previous security issues related to Gradio, and vulnerabilities in the applications that use Gradio.

Let me preface here that it’s natural that a framework like Gradio has sources and sinks, and there’s nothing inherently wrong about it. All frameworks have sources and sinks—Django, Flask, Tornado, and so on. The point is that if the classes and functions provided by the frameworks are not used in a secure way, they may lead to vulnerabilities. And that’s what we are interested in catching here, for applications that use the Gradio framework.

The issue was subsequently fixed in Gradio 5.0 and so, submitting values that are not valid choices in gr.Dropdown, gr.Radio, and gr.CheckboxGroup results in an error, namely:

gradio.exceptions.Error: "Value: turtle is not in the list of choices: ['cat', 'dog', 'bird']"

In this case, the vulnerabilities which may result from these sources, may not be exploitable in Gradio version 5.0 and later. There were also a number of other changes regarding security to the Gradio framework in version 5.0, which can be explored in the ToB report on Gradio security.

The change made me ponder whether to update the CodeQL models I have written and added to CodeQL. However, since the sources can still be misused in applications running Gradio versions below 5.0, I decided to leave them as they are.

Modeling Gradio with CodeQL

We have now identified a number of potential sources. We can then write CodeQL models for them, and later use these sources with existing sinks to find vulnerabilities in Gradio applications at scale. But let’s go back to the beginning: how do we model these Gradio sources with CodeQL?

import gradio as gr
import os

def execute_cmd(folder, logs):
    cmd = f"python caption.py --dir={folder} --logs={logs}"
    os.system(cmd)


with gr.Blocks() as demo:
    gr.Markdown("Create caption files for images in a directory")
    with gr.Row():
        folder = gr.Textbox(placeholder="Directory to caption")
        logs = gr.Checkbox(label="Add verbose logs")

    btn = gr.Button("Run")
    btn.click(fn=execute_cmd, inputs=[folder, logs])


if __name__ == "__main__":
    demo.launch(debug=True)

I also added two more code snippets which use positional arguments instead of keyword arguments. The database and the code snippets are available in the CodeQL zero to hero repository.

Now that we have the code, we can create a CodeQL database for it by using the CodeQL CLI. It can be installed either as an extension to the gh tool (recommended) or as a binary. Using the gh tool with the CodeQL CLI makes it much easier to update CodeQL.

After installing CodeQL CLI, we can create a CodeQL database. First, we move to the folder, where all our source code is located. Then, to create a CodeQL database called gradio-cmdi-db for the Python code in the folder gradio-tests, run:

codeql database create gradio-cmdi-db --language=python --source-root="./gradio-tests"

This command creates a new folder gradio-cmdi-db with the extracted source code elements. We will use this database to test and run CodeQL queries on.

I assume here you already have the VS Code CodeQL starter workspace set up. If not, follow the instructions to create a starter workspace and come back when you are finished.

To run queries on the database we created, we need to add it to the VS Code CodeQL extension. We can do it with the “Choose Database from Folder” button and pointing it to the gradio-cmdi-db folder.

Now that we are all set up, we can move on to actual CodeQL modeling.

Identifying code elements to query for

Let’s have another look at our intentionally vulnerable code.

import gradio as gr
import os

def execute_cmd(folder, logs):
    cmd = f"python caption.py --dir={folder} --logs={logs}"
    os.system(cmd)
    return f"Command: {cmd}"


folder = gr.Textbox(placeholder="Directory to caption")
logs = gr.Checkbox(label="Save verbose logs")
output = gr.Textbox()

demo = gr.Interface(
    fn=execute_cmd,
    inputs=[folder, logs],
    outputs=output)

if __name__ == "__main__":
    demo.launch(debug=True)

In the first example with the Interface class, we had several components on the website. In fact, on the left side we have one source component which takes user input – a textbox. On the right side, we have an output text component. So, it’s not enough that a component is, for example, of a gr.Textbox() type to be considered a source.

To be considered a source of an untrusted input, a component has to be passed to the inputs keyword argument, which takes an input component or a list of input components that will be used by the function passed to fn and processed by the logic of the application in a potentially vulnerable way. So, not all components are sources. In our case, any values passed to inputs in the gr.Interface class are sources. We could go a step further, and say that if gr.Interface is used, then anything passed to the execute_cmd function is a source – so here the folder and logs.

The same situation happens in the second example with the gr.Blocks class and the gr.Button.click event listener. Any arguments passed to inputs in the Button.onlick method, so, to the execute_cmd function, are sources.

We can get all references to gr.Interface calls with the following query. Note that:

• We set the query to be @kind problem, which will format the results of the select as an alert.
• In from, we define a node variable of the API::CallNode type, which gives us the set of all API::CallNodes in the program.
• In where, we filter the node to be a gr.Interface call.
• In select, we choose our output to be node and string “Call to gr.Interface”, which will be formatted as an alert due to setting @kind problem.

/**
 * @id codeql-zero-to-hero/4-1
 * @severity error
 * @kind problem
 */

import python
import semmle.python.ApiGraphs

from API::CallNode node
where node =
    API::moduleImport("gradio").getMember("Interface").getACall()
select node, "Call to gr.Interface"

Run the query by right-clicking and selecting “CodeQL: Run Query on Selected Database”. If you are using the same CodeQL database, you should see two results, which proves that the query is working as expected (recall that I added more code snippets to the test database. The database and the code snippets are available in the CodeQL zero to hero repository).

Next, we want to identify values passed to inputs in the gr.Interface class, which are passed to the execute_cmd function. We could do it in two ways—by identifying the values passed to inputs and then linking them to the function referenced in fn, or by looking at the parameters to the function referenced in fn directly. The latter is a bit easier, so let’s focus on that solution. If you’d be interested in the second solution, check out the Taint step section.

To sum up, we are interested in getting the folder and logs parameters.

import gradio as gr
import os

def execute_cmd(folder, logs):
    cmd = f"python caption.py --dir={folder} --logs={logs}"
    os.system(cmd)
    return f"Command: {cmd}"


folder = gr.Textbox(placeholder="Directory to caption")
logs = gr.Checkbox(label="Save verbose logs")
output = gr.Textbox()

demo = gr.Interface(
    fn=execute_cmd,
    inputs=[folder, logs],
    outputs=output)

if __name__ == "__main__":
    demo.launch(debug=True)

We can get folder and logs with the query below:

/**
 * @id codeql-zero-to-hero/4-2
 * @severity error
 * @kind problem
 */

 import python
 import semmle.python.ApiGraphs

 from API::CallNode node
 where node =
     API::moduleImport("gradio").getMember("Interface").getACall()

 select node.getParameter(0, "fn").getParameter(_), "Gradio sources"

To get the first the function reference in fn (or in the 1st positional argument) we use the getParameter(0, "fn") predicate – 0 refers to the 1st positional argument and "fn" refers to the fn keyword argument. Then, to get the parameters themselves, we use the getParameter(_) predicate. Note that an underscore here is a wildcard, meaning it will output all of the parameters to the function referenced in fn. Running the query results in 3 alerts.

We can also encapsulate the logic of the query into a class to make it more portable. This query will give us the same results. If you need a refresher on classes and using the exists mechanism, see CodeQL zero to hero part 2.

/**
 * @id codeql-zero-to-hero/4-3
 * @severity error
 * @kind problem
 */

 import python
 import semmle.python.ApiGraphs
 import semmle.python.dataflow.new.RemoteFlowSources

 class GradioInterface extends RemoteFlowSource::Range {
    GradioInterface() {
        exists(API::CallNode n |
        n = API::moduleImport("gradio").getMember("Interface").getACall() |
        this = n.getParameter(0, "fn").getParameter(_).asSource())
    }
    override string getSourceType() { result = "Gradio untrusted input" }

 }

from GradioInterface inp
select inp, "Gradio sources"

Note that for GradioInterface class we start with the RemoteFlowSource::Range supertype. This allows us to add the sources contained in the query to the RemoteFlowSource abstract class.

An abstract class is a union of all its subclasses, for example, the GradioInterface class we just modeled as well as the sources already added to CodeQL, for example, from Flask, Django or Tornado web frameworks. An abstract class is useful if you want to group multiple existing classes together under a common name.

Meaning, if we now query for all sources using the RemoteFlowSource class, the results will include the results produced from our class above. Try it!

/**
 * @id codeql-zero-to-hero/4-4
 * @severity error
 * @kind problem
 */

 import python
 import semmle.python.ApiGraphs
 import semmle.python.dataflow.new.RemoteFlowSources

class GradioInterface extends RemoteFlowSource::Range {
    GradioInterface() {
        exists(API::CallNode n |
        n = API::moduleImport("gradio").getMember("Interface").getACall() |
        this = n.getParameter(0, "fn").getParameter(_).asSource())
    }
    override string getSourceType() { result = "Gradio untrusted input" }

 }


from RemoteFlowSource rfs
select rfs, "All python sources"

For a refresher on RemoteFlowSource, and how to use it in a query, head to CodeQL zero to hero part 3.

Note that since we modeled the new sources using the RemoteFlowSource abstract class, all Python queries that already use RemoteFlowSource will automatically use our new sources if we add them to library files, like I did in this pull request to add Gradio models. Almost all CodeQL queries use RemoteFlowSource. For example, if you run the SQL injection query, it will also include vulnerabilities that use the sources we’ve modeled. See how to run prewritten queries in CodeQL zero to hero part 3.

Modeling gr.Button.click

We model gr.Button.click in a very similar way.

/**
 * @id codeql-zero-to-hero/4-5
 * @severity error
 * @kind problem
 */

 import python
 import semmle.python.ApiGraphs

 from API::CallNode node
 where node =
     API::moduleImport("gradio").getMember("Button").getReturn()
     .getMember("click").getACall()

select node.getParameter(0, "fn").getParameter(_), "Gradio sources"

Note that in the code we first create a Button object with gr.Button() and then call the click() event listener on it. Due to that, we need to use API::moduleImport("gradio").getMember("Button").getReturn() to first get the nodes representing the result of calling gr.Button() and then we continue with .getMember("click").getACall() to get all calls to gr.Button.click. Running the query results in 3 alerts.

We can also encapsulate the logic of this query into a class too.

/**
 * @id codeql-zero-to-hero/4-6
 * @severity error
 * @kind problem
 */


 import python
 import semmle.python.ApiGraphs
 import semmle.python.dataflow.new.RemoteFlowSources

class GradioButton extends RemoteFlowSource::Range {
    GradioButton() {
        exists(API::CallNode n |
        n = API::moduleImport("gradio").getMember("Button").getReturn()
        .getMember("click").getACall() |
        this = n.getParameter(0, "fn").getParameter(_).asSource())
    }

    override string getSourceType() { result = "Gradio untrusted input" }

 }

from GradioButton inp
select inp, "Gradio sources"

Vulnerabilities using Gradio sources

The os.system call is defined in the OsSystemSink class and the sink, that is the first argument to the os.system sink call, is defined in the isSink predicate.

/**
 * @id codeql-zero-to-hero/4-7
 * @severity error
 * @kind path-problem
 */


 import python
 import semmle.python.dataflow.new.DataFlow
 import semmle.python.dataflow.new.TaintTracking
 import semmle.python.ApiGraphs
 import semmle.python.dataflow.new.RemoteFlowSources
 import MyFlow::PathGraph

class GradioButton extends RemoteFlowSource::Range {
    GradioButton() {
        exists(API::CallNode n |
        n = API::moduleImport("gradio").getMember("Button").getReturn()
        .getMember("click").getACall() |
        this = n.getParameter(0, "fn").getParameter(_).asSource())
    }

    override string getSourceType() { result = "Gradio untrusted input" }

 }

 class GradioInterface extends RemoteFlowSource::Range {
    GradioInterface() {
        exists(API::CallNode n |
        n = API::moduleImport("gradio").getMember("Interface").getACall() |
        this = n.getParameter(0, "fn").getParameter(_).asSource())
    }
    override string getSourceType() { result = "Gradio untrusted input" }

 }


class OsSystemSink extends API::CallNode {
    OsSystemSink() {
        this = API::moduleImport("os").getMember("system").getACall()
    }
}


 private module MyConfig implements DataFlow::ConfigSig {
   predicate isSource(DataFlow::Node source) {
     source instanceof GradioButton
     or
     source instanceof GradioInterface
   }

   predicate isSink(DataFlow::Node sink) {
    exists(OsSystemSink call |
        sink = call.getArg(0)
        )
   }
 }

 module MyFlow = TaintTracking::Global;

 from MyFlow::PathNode source, MyFlow::PathNode sink
 where MyFlow::flowPath(source, sink)
 select sink.getNode(), source, sink, "Data Flow from a Gradio source to `os.system`"

Running the query results in 6 alerts, which show us the path from source to sink. Note that the os.system sink we used is already modeled in CodeQL, but we are using it here to illustrate the example.

Similarly to the GradioInterface class, since we have already written the sources, we can actually use them (as well as the query we’ve written above) on any Python project. We just have to add them to library files, like I did in this pull request to add Gradio models.

We can actually run any query on up to 1000 projects at once using a tool called Multi repository Variant Analysis (MRVA).

But before that.

Other sources in Gradio

Taint step

We’ve mentioned that there are two ways to model gr.Interface and other Gradio sources—by identifying the values passed to inputs and then linking them to the function referenced in fn, or by looking at the parameters to the function referenced in fn directly.

import gradio as gr
import os

def execute_cmd(folder, logs):
    cmd = f"python caption.py --dir={folder} --logs={logs}"
    os.system(cmd)
    return f"Command: {cmd}"


folder = gr.Textbox(placeholder="Directory to caption")
logs = gr.Checkbox(label="Save verbose logs")
output = gr.Textbox()

demo = gr.Interface(
    fn=execute_cmd,
    inputs=[folder, logs],
    outputs=output)

if __name__ == "__main__":
    demo.launch(debug=True)

As it turns out, machine learning applications written using Gradio often use a lot of input variables, which are later processed by the application. In this case, inputs argument gets a list of variables, which at times can be very long. I’ve found several cases which used a list with 10+ elements.

In these cases, it would be nice to be able to track the source all the way to the component that introduces it— in our case, gr.Textbox and gr.Checkbox.

To do that, we need to use a taint step. Taint steps are usually used in case taint analysis stops at a specific code element, and we want to make it propagate forward. In our case, however, we are going to write a taint step to track a variable in inputs, that is an element of a list, and track it back to the component.

We start by identifying the variables passed to inputs in, for example, gr.Interface:

class GradioInputList extends RemoteFlowSource::Range {
    GradioInputList() {
      exists(GradioInput call |
        // limit only to lists of parameters given to `inputs`.
        (
          (
            call.getKeywordParameter("inputs").asSink().asCfgNode() instanceof ListNode
            or
            call.getParameter(1).asSink().asCfgNode() instanceof ListNode
          ) and
          (
            this = call.getKeywordParameter("inputs").getASubscript().getAValueReachingSink()
            or
            this = call.getParameter(1).getASubscript().getAValueReachingSink()
          )
        )
      )
    }

    override string getSourceType() { result = "Gradio untrusted input" }
  }

Next, we identify the function in fn and link the elements of the list of variables in inputs to the parameters of the function referenced in fn.

class ListTaintStep extends TaintTracking::AdditionalTaintStep {
  override predicate step(DataFlow::Node nodeFrom, DataFlow::Node nodeTo) {
    exists(GradioInput node, ListNode inputList |
       inputList = node.getParameter(1, "inputs").asSink().asCfgNode() |
       exists(int i | 
         nodeTo = node.getParameter(0, "fn").getParameter(i).asSource() |
         nodeFrom.asCfgNode() =
         inputList.getElement(i))
       )
    }
  }

Let’s explain the taint step, step by step.

exists(GradioInput node, Listnode inputList |

In the taint step, we define two temporary variables in the exists mechanism—node of type GradioInput and inputList of type ListNode.

inputList = node.getParameter(1, "inputs").asSink().asCfgNode() |

Then, we set our inputList to the value of inputs. Note that because inputList has type ListNode, we are looking only for lists.

exists(int i | 
nodeTo = node.getParameter(0, "fn").getParameter(i).asSource() |
nodeFrom.asCfgNode() = inputList.getElement(i))

Next, we identify the function in fn and link the parameters of the function referenced in fn to the elements of the list of variables in inputs, by using a temporary variable i.

All in all, the taint step provides us with a nicer display of the paths, from the component used as a source to a potential sink.

Scaling the research to thousands of repositories with MRVA

In short,MRVA allows you to run a query on up to 1000 projects hosted on GitHub at once. It comes preconfigured with dynamic lists for most popular repositories 10, 100, and 1000 for each language. You can configure your own lists of repositories to run CodeQL queries on and potentially find more variants of vulnerabilities that use our new models. @maikypedia wrote a neat case study about using MRVA to find SSTI vulnerabilities in Ruby and Unsafe Deserialization vulnerabilities in Python.

MRVA is used together with the VS Code CodeQL extension and can be configured in the extension, in the “Variant Analysis” section. It uses GitHub Actions to run, so you need a repository, which will be used as a controller to run these actions. You can create a public repository, in which case running the queries will be free, but in this case, you can run MRVA only on public repositories. The docs contain more information about MRVA and its setup.

Using MRVA, I’ve found 11 vulnerabilities to date in several Gradio projects. Check out the vulnerability reports on GitHub Security Lab’s website.

Reach out!

Today, we learned how to model a new framework in CodeQL, using Gradio as an example, and how to use those models for finding vulnerabilities at scale. I hope that this post helps you with finding new cool vulnerabilities! 🔥