Task2vec depends solely on the task, and ignores interactions with the model which may however play an important role. This can be particularly valuable when there is insufficient data to train or fine-tune a generic model, and transfer of knowledge is essential. As a motivating example, we study the problem of selecting the best pre-trained feature extractor to solve a new task. Our task embedding can be used to reason about the space of tasks and solve meta-tasks. We show this embedding encodes the “difficulty” of the task, characteristics of the input domain, and which features of the probe network are useful to solve it (Sect. ![]() Since the architecture and weights of the probe network are fixed, the FIM provides a fixed-dimensional representation of the task. Specifically, given a task defined by a dataset D = N i = 1 of labeled samples, we feed the data through a pre-trained reference convolutional neural network which we call “ probe network”, and compute the diagonal Fisher Information Matrix (FIM) of the network filter parameters to capture the structure of the task (Sect. 1 IntroductionĬomputation of the embedding leverages a duality between network parameters (weights) and outputs (activations) in a deep neural network (DNN): Just as the activations of a DNN trained on a complex visual recognition task are a rich representation of the input images, we show that the gradients of the weights relative to a task-specific loss are a rich representation of the task itself. In this case, the domain embeddings collapse to a region without recovering any sensible structure. However, the fashion attribute tasks on iMaterialist all share the same domain and only differ in their labels. INaturalist tasks from iMaterialist tasks due to differences in the two problemĭomains. ![]() (Right) T-SNE visualization of the domain embeddings (using mean feature activations) for the same tasks. For example, the tasks of jeans (clothing type), denim (material) and ripped (style) recognition are close in the task embedding. Notice that some tasks of similar type (such as color attributes) cluster together but attributes of different task types may also mix when the underlying visual semantics are correlated. iMaterialist is well separated from iNaturalist, as it entails very different tasks (clothing attributes). Notice that the bird classification tasks extracted from CUB-200 embed near the bird classification task from iNaturalist, even though the original datasets are different. Colors indicate ground-truth grouping of tasks based on taxonomic or semantic types. (Left) T-SNE visualization of the embedding of tasks extracted from the iNaturalist, CUB-200, iMaterialist datasets. Figure 1: Task embedding across a large library of tasks (best seen magnified). Selecting a feature extractor with task embedding obtains a performance close to the best available feature extractor, while costing substantially less than exhaustively training and evaluating on all available feature extractors. We present a simple meta-learning framework for learning a metric on embeddings that is capable of predicting which feature extractors will perform well. ![]() We also demonstrate the practical value of this framework for the meta-task of selecting a pre-trained feature extractor for a new task. We demonstrate that this embedding is capable of predicting task similarities that match our intuition about semantic and taxonomic relations between different visual tasks ( e.g., tasks based on classifying different types of plants are similar). ![]() This provides a fixed-dimensional embedding of the task that is independent of details such as the number of classes and does not require any understanding of the class label semantics. Given a dataset with ground-truth labels and a loss function defined over those labels, we process images through a “probe network” and compute an embedding based on estimates of the Fisher information matrix associated with the probe network parameters. We introduce a method to provide vectorial representations of visual classification tasks which can be used to reason about the nature of those tasks and their relations.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |