Contact me

How I built a product visual search using react-native, Google Cloud Vision API , Algolia and Remix.js

by Ilias Haddad

As part of my journey learning react native, I was thinking about building something cool that is related to the thing I enjoy talking about which is Shopify.

So, I decided to add a feature to the Shop app. that will allow user search in Shopify stores product catalog using an image instead of the keywords.

The workflow:

  • A user scrolling through Instagram and the found a product that they like
  • Take a screenshot of it
  • Open the mobile app, click on the camera icon.
  • Select the image and let the magic happen
  • Get a list back of similar looking products across different Shopify store (in our case, it’ll be one Shopify Store)

Architectural Overview

The solution leverages a robust technology stack:

  • React Native for cross-platform mobile development
  • Google Cloud Vision API for image classification
  • Algolia Search for efficient product indexing and retrieval
  • Remix.js for backend API handling
  • Shopify Storefront API for product data management

Developer Note: Keeping it Simple

While building this demo, I intentionally simplified many aspects that would be crucial for a production system. Here's why:

The main goal was to demonstrate how to:

  • Connect Google Cloud Vision API with Shopify product search
  • Process product images to enable visual search
  • Create a basic mobile interface for image uploads

I deliberately omitted production concerns like error handling, security, and scalability because:

  1. This is a learning demo focused on the core visual search workflow
  2. Adding production-grade features would obscure the main technical concepts
  3. Each production requirement (auth, caching, monitoring etc.) deserves its own deep dive

Think of this implementation as a "proof of concept" that shows:

  • The basic architecture works
  • Visual search can enhance product discovery
  • The core APIs integrate successfully

For anyone looking to build on this demo, you'd want to address production concerns based on your specific needs - like authentication, error handling, and performance optimization. But those additions shouldn't overshadow understanding the core visual search mechanics demonstrated here.

The goal was to keep the code approachable and focused on demonstrating value, rather than building a production-ready system. This lets developers understand the core concepts first, then layer in production requirements as needed.

Technical Implementation

Step 1: Labeling the Shopify products images

In order to label the Shopify products images, I used the Google Cloud Vision API to label the product variant image. Here’s an example code

const vision = require("@google-cloud/vision");


// Instantiate Google Vision client
const client = new vision.ImageAnnotatorClient({
  keyFilename: ""// Path to your service account key file here
});


export const getImageLabels = async (imageURL, objectID) => {
  const [result] = await client.labelDetection(imageURL);
  const labels = result.labelAnnotations
    .filter((label) => label.score > 0.5)
    .map((label) => ({
      description: label.description,
      score: label.score,
    }));
  return { objectID, labels, imageURL };
};

In the getImageLabels function, I passed the imageURL from the Shopify query results for each product variant image on each Shopify product. Using Shopify Storefront API, here’s the example code of it.

const getAllProducts = async () => {
  const productQuery = `
    {
  products(first: 250) {
    edges {
      node {
        id
        variants(first: 250) {
          edges {
            node {
              id
              image {
                url
              }
            }
          }
        }
      }
    }
  }
}
    `;
  const { data, errors } = await ShopifyClient.request(
    productQuery
  );
  if (errors) {
    console.error(errors);
  }
  const products = data.products.edges;
  const productImages = products.map((product) => {
    const productNode = product.node;
    const productVariants = productNode.variants.edges;
    const productImages = productVariants.map((variant) => {
      const variantNode = variant.node;
      const image = variantNode.image;
      if (!image) return;
      return {
        imageURL: image.url,
        objectID: `${productNode.id}-${variantNode.id}`,
      };
    });
    return productImages;
  });
  const flatProductImages = productImages.flat();


  const allImageLabels = await Promise.all(
    flatProductImages.map(async (image) => {
      if (!image) return;
      return await getImageLabels( image.imageURL, image.objectID, image.imageURL);
    })
  );
 }

Key implementation highlights:

  • Filtering labels with confidence scores above 0.5
  • Extracting comprehensive image metadata
  • Preparing data for subsequent indexing

Step 2: Indexing the Shopify products images with their labels

After getting the image labels, I’ll be using Algolia Search API and each product is indexed with a unique identifier combining product and variant IDs, along with associated image labels.

Indexing strategy:

  • Unique object identification
  • Comprehensive label preservation
  • Efficient search optimization

Here’s an example code snippet:

 const { searchClient } = require("@algolia/client-search");
const algoliaClient = searchClient(
  "",
  ""
);


const getAllProducts = async () => {


 // ...previous code 


  const indexedImages = await Promise.all(
    allImageLabels.map(async (image) => {
      if (!image) {
        console.log(`warning: ${image} is not an image`);
        return;
      }
      return await algoliaClient.saveObject({
        indexName: "products",
        body: image,
      });
    })
  );
  
  }

Awesome, we made a great progress to index the product using Algolia and here’s a screenshot of the dashboard in my Algolia Dashboard.

Step 3: Visual Search API Endpoint

The Remix.js backend action function serves as the core of our visual search mechanism. It processes base64-encoded images, generates labels, and performs a sophisticated search across the product index.

Search workflow:

  • Image base64 decoding
  • Label generation
  • Algolia search with optional filters
  • Unique product ID extraction

Here’s a snippet code of my Remix.jsaction function

import type { LoaderFunctionArgs } from '@remix-run/cloudflare';
import { json } from '@remix-run/react';
import {
	getImageLabels,
	reduceLabelsToFilters,
} from '../services/vision.server';
import { algoliaClient } from '../services/algolia.server';


export async function action({ request }: LoaderFunctionArgs) {
	const formData = await request.formData();
	const image = formData.get('image') as string; // Image Base64
	if (!image) {
		return json({ error: 'No image provided' }, { status: 400 });
	}


	const classifiedImage = await getImageLabels(image);
	const labels = reduceLabelsToFilters(classifiedImage.labels);


	const indexName = 'products';
	const { results } = await algoliaClient.search({
		requests: [
			{
				indexName,
				optionalFilters: labels,
				query: classifiedImage.labels.map(label => label.description).join(' '),
			},
		],
	});
	const products = results[0].hits.map(
		product => product.objectID.split('-')[0],
	);
	// remove duplicates
	const uniqueProducts = [...new Set(products)];
	console.log(uniqueProducts);
	return json({ results: uniqueProducts });
}

Step 4: Build a mobile app to upload the image

The React Native mobile application provides an intuitive interface for visual product search. Key components include:

  • Image picker functionality
  • Seamless API integration

Here’s the code snippet responsible for handle image search.

// components/searchBar.tsx


import { useState } from "react";
import { TextInput, useColorScheme } from "react-native";
import {
  launchImageLibrary,
  type ImageLibraryOptions,
  type ImagePickerResponse,
} from "react-native-image-picker";
import Ionicons from "@expo/vector-icons/Ionicons";
import { Box } from "./Container";
import theme, { darkTheme } from "@/theme";


const ShopSearchBar = ({
  onSearch,
  placeholder = "Search products",
  setSelectedImage,
  setSearching,
}: {
  onSearch?: (query: string) => void;
  onFilterPress?: () => void;
  placeholder?: string;
  setSelectedImage: (image: string | null) => void;
  setSearching: (searching: boolean) => void;
}) => {
  const colorSchema = useColorScheme();
  const [searchQuery, setSearchQuery] = useState("");
  const openImagePicker = () => {
    const options: ImageLibraryOptions = {
      mediaType: "photo",
      includeBase64: true,
      maxHeight: 2000,
      maxWidth: 2000,
    };


    launchImageLibrary(options, (response: ImagePickerResponse) => {
      if (response.didCancel) {
        console.log("User cancelled image picker");
      } else if (response.errorMessage) {
        console.log("Image picker error: ", response.errorMessage);
      } else if (response.assets) {
        let imageUri = response.assets[0].base64;
        if (imageUri) {
          setSelectedImage(imageUri);
          setSearching(true);
        }
      }
    });
  };


  const handleSearch = () => {
    if (onSearch) {
      onSearch(searchQuery);
    }
  };


  return (
    <Box
      flexDirection="row"
      padding="s"
      marginRight="s"
      marginLeft="s"
      backgroundColor="cardPrimaryBackground"
      borderRadius="m"
    >
      <Box
        flexDirection="row"
        padding="s"
        flex={1}
        justifyContent="space-between"
        alignContent="center"
      >
        <Ionicons
          name="search"
          size={20}
          color={
            colorSchema === "dark"
              ? darkTheme.colors.iconColor
              : theme.colors.iconColor
          }
        />
        <TextInput
          placeholder={placeholder}
          placeholderTextColor={
            colorSchema === "dark"
              ? darkTheme.colors.placeholderText
              : theme.colors.placeholderText
          }
          value={searchQuery}
          onChangeText={setSearchQuery}
          onSubmitEditing={handleSearch}
          clearButtonMode="while-editing"
          autoCapitalize="none"
          autoCorrect={false}
          allowFontScaling={true}
        />
        <Ionicons
          name="camera"
          size={20}
          color={
            colorSchema === "dark"
              ? darkTheme.colors.iconColor
              : theme.colors.iconColor
          }
          onPress={openImagePicker}
        />
      </Box>
    </Box>
  );
};


export default ShopSearchBar;

And this is a screenshot of the app:

Step 5: Link the mobile app with the API

The final step connects the mobile application with the backend API, enabling users to upload images and receive relevant product recommendations dynamically.

// lib/search.ts
export const searchByImage = async (image: string) => {
  const formdata = new FormData();
  formdata.append("image", image);


  const requestOptions = {
    method: "POST",
    body: formdata,
  };


  const response = await fetch(
    `${process.env.EXPO_PUBLIC_APP_HOST}/search`,
    requestOptions
  );
  const data = await response.json();
  return data.results;
};

Server Setup

I used Remix.js Cloudflare Template

I made an API route for searching using an image in base64 format using Algolia Search API which I used before to index our Shopify product image with image labels we got from Google Cloud Vision API. I used the Shopify product variant id and product id as an index.

// app/routes/search.tsx
import type { LoaderFunctionArgs } from '@remix-run/cloudflare';
import { json } from '@remix-run/react';
import {
	getImageLabels,
	reduceLabelsToFilters,
} from '../services/vision.server';
import { algoliaClient } from '../services/algolia.server';


export async function action({ request }: LoaderFunctionArgs) {
	const formData = await request.formData();
	const image = formData.get('image') as string; // Image Base64
	if (!image) {
		return json({ error: 'No image provided' }, { status: 400 });
	}


	const classifiedImage = await getImageLabels(image);
	const labels = reduceLabelsToFilters(classifiedImage.labels);


	const indexName = 'products';
	const { results } = await algoliaClient.search({
		requests: [
			{
				indexName,
				optionalFilters: labels,
				query: classifiedImage.labels.map(label => label.description).join(' '),
			},
		],
	});


	// ObjectID is a unique identifier for each image in Algolia, Example: "1234-54355", first part is the product ID and the second part is the variant ID
	// We're indexing each variant image separately, and we're using the product ID to get parent product information
	const products = results[0].hits.map(
		product => product.objectID && product.objectID.split('-')[0],
	);




	return json({ results: products });
}

Here's our Algolia Search API client:

// app/services/algolia.server.ts
import { searchClient } from '@algolia/client-search';
import { configDotenv } from 'dotenv';
configDotenv({
	path: '.dev.vars',
});
if (!process.env.ALGOLIA_APP_ID) {
	throw new Error(
		'Missing Algolia App ID. Please provide it in the .env file.',
	);
}
if (!process.env.ALGOLIA_SEARCH_KEY) {
	throw new Error(
		'Missing Algolia Search Key. Please provide it in the .env file.',
	);
}
export const algoliaClient = searchClient(
	process.env.ALGOLIA_APP_ID,
	process.env.ALGOLIA_SEARCH_KEY,
);

Here's our Shopify Storefront API client:

// app/services/shopify.server.ts


import { createStorefrontApiClient } from '@shopify/storefront-api-client';
import { configDotenv } from 'dotenv';
configDotenv({
	path: '.dev.vars',
});
if (!process.env.STORE_DOMAIN) {
	throw new Error('Missing Store Domain. Please provide it in the .env file.');
}
if (!process.env.STOREFRONT_ACCESS_TOKEN) {
	throw new Error(
		'Missing Storefront Access Token. Please provide it in the .env file.',
	);
}


export const ShopifyClient = createStorefrontApiClient({
	storeDomain: process.env.STORE_DOMAIN,
	publicAccessToken: process.env.STOREFRONT_ACCESS_TOKEN,
	apiVersion: '2024-10',
});

And our Google Cloud Vision API to get image labels by passing an image in base64 format

// app/services/vision.server.ts


import vision from '@google-cloud/vision';
import { configDotenv } from 'dotenv';
configDotenv({
	path: '.dev.vars',
});
if (!process.env.GOOGLE_APPLICATION_CREDENTIALS_BASE64) {
	throw new Error(
		'Missing Google Application Credentials. Please provide it in the .env file.',
	);
}


const credentials = JSON.parse(
	Buffer.from(
		process.env.GOOGLE_APPLICATION_CREDENTIALS_BASE64,
		'base64',
	).toString('utf-8'),
);


const client = new vision.ImageAnnotatorClient({
	credentials,
});


type Label = {
	description: string | null | undefined;
	score: number | null | undefined;
};
export const getImageLabels = async (imageBase64: string) => {
	const [result] = await client.annotateImage({
		image: {
			content: imageBase64,
		},
		features: [
			{
				type: 'LABEL_DETECTION',
				maxResults: 10,
			},
		],
	});
	if (!result.labelAnnotations) {
		return { labels: [] };
	}


	const labels = result.labelAnnotations
		.filter(label => label.score && label.score > 0.5)
		.map(label => ({
			description: label.description,
			score: label.score,
		}));
	return { labels };
};


export const reduceLabelsToFilters = (labels: Label[]) => {
	const optionalFilters = labels.map(
		label => `labels.description:'${label.description}'`,
	);
	return optionalFilters;
};

Conclusion

This implementation demonstrates an approach to visual product search, showcasing the power of modern web and mobile technologies in creating innovative e-commerce experiences.

Source Code: https://github.com/IliasHad/visual-product-search-app

Notes:

What I mentioned above is one of the approaches to use for product image search, we can have other methods like:

  • React Native

Tell me about your project

Say Hei