An LLM can Fool Itself:
A Prompt-Based Adversarial Attack

Let's Attack the LLM via PromptAttack!

1

Generating original input (OI).

2

Generating attack objective (AO).

3

Generating attack guidance (AG).

4

Generating adversarial sample via feeding the attack prompt composed of the OI, AO, and AG to the victim LLM.

5

Getting filtered adversarial sample.

6

Generating the prompt composed of the sample and the task description for solving the classifition task.

7

Obtaining predicted labels from the LLM.

We let $\mathcal{D}=\{(x_i,y_i)\}_{i=1}^N$ be the original test dataset consisting of $N \in \mathbb{N}$ data points.

For each data point $(x,y)\in \mathcal{D}$, $x = \{ t^i, c^i\}_{i=1}^n$ is the original sample where $n \in \mathbb{N}$ is the number of sentences, $t^i$ refers to the type of $i$-th sentence, and $c^i$ refers to the content of $i$-th sentence. For example, the original input in QQP and MNLI can have two types of sentences (i.e., $n=2$). We follow the types defined in their datasets, e.g., $t^1$ being question1 and $t^2$ being question2 for QQP, $t^1$ being premise and $t^2$ being hypothesis for MNLI.

Original input (OI)

The OI converts a data point composed of the original sample and ground-truth label sampled from a dataset into a sentence of an attack prompt. Given a data point $(x,y) \in \mathcal{D}$, we can formulate the OI as follows:

The original $t^1c^1$ and $t^2c^2$ and $\dots$ and $t^nc^n$ is classified as $y^k$.

Attack objective (AO)

The adversarial textual attack aims to generate an adversarial sample that should keep the same semantic meaning as its original version and can fool the LLM into doing incorrect classification. Here, we assume PromptAttack can perturb only one type of sentence for each data point. Therefore, given a data point $(x,y)\in \mathcal{D}$ and the type of the sentence that is targeted to be perturbed $t^a \in \{t^1,\dots,t^n \}$ where $a \in \mathbb{N}$, we formulate the AO as follows:

Your task is to generate a new $t^a$ which must satisfy the following conditions:
1.Keeping the semantic meaning of the new $t^a$ unchanged;
2.The new $t^a$ and the original $t^1$, $\dots$, $t^{a-1}$, $t^{a+1}$, $\dots$, $t^n$, should be classified as $y^1$ or $\dots$ or $y^{k-1}$ or $y^{k+1}$ or $\dots$ or $y^{C}$.

Attack guidance (AG)

AG contains the perturbation instruction to guide the LLM on how to perturb the original sample and specifies the format of the generated text. In the AG, we first ask the LLM to only perturb the type of the target sentence to finish the task. Then, we provide the perturbation instruction that guides the LLM on how to perturb the target sentence to generate the adversarial sample that fits the requirement of AO. Finally, we specify that the output of the LLM should only contain the newly generated sentence. Therefore, given a data point $(x,y)\in \mathcal{D}$ and the type of the target sentence $t^a$, we can formulate the AG as follows:

You can finish the task by modifying $t^a$ using the following guidance:
#perturbation_instruction
Only output the new $t^a$ without anything else.

Results

Attack success rate (ASR) evaluated on the GLUE dataset.
The ASR obtained by PromptAttack significantly outperforms AdvGLUE and AdvGLUE++.

The ASR w.r.t. BERTScore threshold evaluated in the GLUE dataset using GPT-3.5.
PromptAttack consistently yields a higher ASR compared to AdvGLUE and AdvGLUE++.

Adversarial Examples

BibTeX

        @article{xu2023promptattack,
          title={An LLM can Fool Itself: A Prompt-Based Adversarial Attack},
          author={Xilie Xu and Keyi Kong and Ning Liu and Lizhen Cui and Di Wang and Jingfeng Zhang and Mohan Kankanhalli},
          journal={arXiv preprint arXiv:2310.13345},
          year={2023}
        }