---
title: Blip API Usage Guide
slug: 6DCWoM_fbbp1XuSHH8vVc
createdAt: Thu Jul 18 2024 06:12:37 GMT+0000 (Coordinated Universal Time)
updatedAt: Thu Jul 18 2024 13:40:04 GMT+0000 (Coordinated Universal Time)
---

# Blip API Usage Guide

## Introduction

Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

## Prerequisites

- Node.js environment
- `aonweb` library installed
- Valid Aonet APPID

## Basic Usage

### 1. Import Required Modules

```js
import { AI, AIOptions } from 'aonweb';
```

### 2. Initialize AI Instance

```js
const ai_options = new AIOptions({
    appId: 'your_app_id_here',
    dev_mode: true
});

const aonweb = new AI(ai_options);
```

### 3. Prepare Input Data Example

```js
const data = {
  input: {
    "task": "image_captioning",
    "image": "https://replicate.delivery/mgxm/f4e50a7b-e8ca-432f-8e68-082034ebcc70/demo.jpg",
  }
};
```

```js
const data = {
  input: {
    "task": "image_text_matching",
    "image": "https://replicate.delivery/mgxm/2c0e8a0d-232d-4cfa-b97a-b3258be2a2e5/demo.jpg",
    "caption": "a dog and a cat are playing in the garden"
  }
};
```


```js
const data = {
  input: {
      "task": "visual_question_answering",
      "image": "https://replicate.delivery/mgxm/32e7126a-dd86-4e8c-8706-5855b8b69cf1/demo.jpg",
      "question": "where is the woman?"
    }
};
```

### 4. Call the AI Model

```js
const price = 8; // Cost of the AI call
try {
    const response = await aonweb.prediction("/predictions/ai/blip", data, price);
    // Handle response
    console.log("Blip result:", response);
} catch (error) {
    // Error handling
    console.error("Error generating :", error);
}
```


### Parameter Description

- `task`: String, The type of task the model needs to perform,choose[`image_captioning`,`image_text_matching`],
  - default: `image_captioning`
  - `image_captioning`: Identify the scene in the picture and give the result
  - `image_text_matching`: The result is obtained by matching the given caption parameters with the scene in the picture
  - `visual_question_answering`: Ask questions about the scene in the picture
- `image`: String, provide the URL of the image that needs to be recognized.
- `caption`:String, A text description of the scene in the image
- `question`:String,Provide a textual description of the questions

### Notes

- Ensure that the provided image URL is publicly accessible and of good quality to achieve the best recognition results.
- The API may take some time to process the input and generate the result, consider implementing appropriate wait or loading states.
- Handle possible errors, such as network issues, invalid input, or API limitations.
- Adhere to the terms of use and privacy regulations, especially when handling image samples of others.

### Example Response

The API response will contain the results of the image recognition or other relevant information. Parse and use the response data according to the actual API documentation.

## Advanced Usage

- Unimodal encoders, which separately encode image and text. The image encoder is a vision transformer. The text encoder is the same as BERT. A token is appended to the beginning of the text input to summarize the sentence.
- Image-grounded text encoder, which injects visual information by inserting a cross-attention layer between the self-attention layer and the feed forward network for each transformer block of the text encoder. A task-specific token is appended to the text, and the output embedding of  is used as the multimodal representation of the image-text pair.
- Image-grounded text decoder, which replaces the bi-directional self-attention layers in the text encoder with causal self-attention layers. A special token is used to signal the beginning of a sequence.
- Image-Text Contrastive Loss (ITC) activates the unimodal encoder. It aims to align the feature space of the visual transformer and the text transformer by encouraging positive image-text pairs to have similar representations in contrast to the negative pairs.
- Image-Text Matching Loss (ITM) activates the image-grounded text encoder. ITM is a binary classification task, where the model is asked to predict whether an image-text pair is positive (matched) or negative (unmatched) given their multimodal feature.
- Language Modeling Loss (LM) activates the image-grounded text decoder, which aims to generate textual descriptions conditioned on the images.

