Introduction
Modern deep learing-based video classifiers achieve strong performance, yet their decision-making processes remain largely opaque. This black-box nature limits their reliability and hinders deployment in high-stakes domains. Consequently, developing effective explanation techniques has become a critical direction for improving model transparency and trustworthiness. Among these, counterfactual explanations (CFEs) provide an intuitive and informative paradigm for interpreting model behavior. Counterfactual explanations aim to answer a what-if question:
In this project, we propose Back To The Feature (BTTF), a video CFE method for generating counterfactual videos to explain video classifiers. Given an input video and a target classifier, BTTF produces alternative yet realistic video futures that start from the same first frame but lead to different predictions from the target classifier. By comparing these videos, we can gain insights into the spatiotemporal features that drive the classifier’s decision.
Method
BTTF is an optimization framework for generating video counterfactual explanations using an image-to-video (I2V) diffusion model (Wan-I2V series). Given an input video, BTTF leverages the property that I2V models can synthesize temporally coherent videos conditioned on a single input image (the first frame), where different initial noise realizations correspond to different plausible futures. Instead of sampling randomly, BTTF directly optimizes the initial latent noise via backpropagation through the diffusion process. The objective is to discover a counterfactual video that (i) changes the prediction of a target video classifier, (ii) remains close to the original input, and (iii) introduces realistic, actionable and temporally coherent modifications. The optimization is performed in a two-stage procedure entirely guided by the target classifier, ensuring faithful explanations.
Inversion. BTTF first anchors the search in the vicinity of the input video by optimizing the initial latent noise such that the diffusion model nearly reconstructs the original video. Concretely, the input video is encoded into a latent representation, and the initial noise is iteratively updated to minimize a reconstruction loss between the denoised latent and the encoded input. This stage effectively inverts the diffusion process, producing a latent initialization that corresponds to a video close to the original input, thereby enforcing proximity and reducing unnecessary deviations in subsequent optimization.
CFE Generation. The optimized latent is further refined to induce a target prediction from the classifier. The latent is decoded into a video and evaluated by the classifier, and gradients from the classification loss are backpropagated to update the initial noise. To preserve realism and stay on the data manifold, a video style regularization term is incorporated, encouraging consistency with the input video while allowing meaningful semantic changes. Additionally, a progressive optimization strategy gradually increases the number of denoising steps to stabilize training and improve convergence. This stage results in a counterfactual video that minimally yet effectively alters the classifier’s decision through coherent spatiotemporal edits.
Results
Citation
If you find BTTF useful for your research, please cite our paper:
@article{wang2025back,
title={Back to the Feature: Explaining Video Classifiers with Video Counterfactual Explanations},
author={Wang, Chao and Che, Chengan and Chen, Xinyue and Tsoka, Sophia and Garcia-Peraza-Herrera, Luis C},
journal={arXiv preprint arXiv:2511.20295},
year={2025}
}