The Center for Data Innovation spoke with Sandra Mau, chief executive officer and founder of TrademarkVision, an image recognition company based in Brisbane, Australia. Mau discussed the problems with how regulators classify intellectual property (IP), as well as how machine learning could help content owners better protect their IP.
Joshua New: TrademarkVision uses machine learning to make it easier for regulators and companies to compare logos. Can you explain the implications of this seemingly simple concept?
Sandra Mau: Before TrademarkVision, the only way to search for image-based marks was to use a laborious system of hierarchical codes. You would pick a code or codes you thought might apply to the mark you were interested in searching and hope that any relevant marks had also been tagged with the same codes. This presents quite the challenge when you consider the U.S. system alone has over 3,800 different design codes. To make matters even more complicated, different countries have different coding systems, meaning you have to essentially start your search over every time you want to search in a new jurisdiction.
It can take years to become skilled at identifying potentially relevant codes and accurately searching for marks in the various government databases. When you consider all of the logos companies use to identify their goods and services, it is no surprise that over 40 percent of all trademarks have an image component. As you can imagine, that means there is a huge amount of time being spent identifying codes and hoping other professionals think the same codes are relevant to what you are searching.
TrademarkVision’s breakthrough is to allow both intellectual property experts and people with no background in trademark searching to conduct a comprehensive search in less time with greater accuracy. You simply upload an image and in seconds the search returns visually similar marks, automatically ranked with the most similar marks at the top of the page.
Machine learning becomes vital to achieving this solution when you consider the two primary ways in which humans recognize visual similarity. On one level, people identify objects as similar based on their shape. For example, most depictions of a horse are going to have a similar exterior shape. Using the shape of a horse to find other horses is the most basic way to tackle image based searching, and an important part of the overall result.
The second way humans recognize visual similarity is where machine learning represents a huge breakthrough. To the human brain, what the object actually represents is equally important to the shape of the object. This is known as the semantic meaning of the image. To go back to our example, a horse and a zebra will have a very similar exterior shape, but the human brain is quickly able to identify the difference between the two. A search based only on shape won’t be able to account for semantic meaning.
We’ve focused on machine learning techniques so the system can recognize objects in trademarks and logos more like humans do, using both elements of identifying similarity. Despite the wide variety of ways humans pictorially depict objects in logos, deep learning has helped to provide a robust solution to the semantic meaning challenge.
New: TrademarkVision now works with Australia’s and the EU’s intellectual property regulators. How is implementation going? I imagine preparing a database of an entire country’s registered logos for this kind of system is pretty laborious.
Mau: Onboarding a new dataset can involve indexing millions of images. Each image has to be formatted, segmented, and run through different pattern recognition algorithm routines. While processing all this data was a challenge in the beginning, we now have experience processing dozens of countries’ datasets, so our data pipeline has become very efficient. We are also very fortunate that both the Australian office (IPA) and the European Intellectual Property Office (EUIPO) have been fantastic partners in this process.
Both the EUIPO and IPA are passionate about being on the cutting edge and offering the highest quality services to the citizens they serve. As a result they are fantastic about working with us to ensure that we get the data and support we need.
New: What’s preventing the U.S. from adopting a system like this?
Mau: The U.S. Patent and Trademark Office has done a great job in digitization and, in particular, focusing on converting process such as trademark applications to use e-filing. Every government has their own list of priorities and timeline. We are ready to support them if they decide to look into updating their trademark search engine.
New: This approach is obviously useful for IP regulators, but could you describe the benefits for the private sector in a country that has adopted your technology?
Mau: Everybody wins when countries invest in lowering the barriers to obtain and protect intellectual property. Both private trademark attorneys and government trademark examiners have a much easier time clearing and registering marks and this leads to many benefits for the private sector.
First off, the private sector can benefit from better tools for the trademark screening and registration process in countries that adopt our technology. Before paying the registration fees, companies can now do a quick and simple search to see if any similar trademarks already exist. We’ve tried to make the system simple and understandable even for non-trademark attorneys as while large corporations may have dedicated in-house counsels for brand protection, most small to medium sized businesses do not.
Additionally, doing international searches are much easier now. If a brand owner wants to do a pre-registration screening for a new logo in Australia, for example, they can now go to the government website, upload the image of its logo, and do a visual search for similarity within seconds. Unless you were a trademark attorney and well versed in the intricacies of the design code systems around the world, this was not easy to do before TrademarkVision.
On the government side, examiners can potentially increase both their efficiency and accuracy with image search, similar to private sector users. But perhaps more importantly, they will be incentivizing innovation within their borders. Without intellectual property protection, inventors and innovators have no incentive to create anything of value since they will not benefit from their creations. Therefore, every step a country takes to increase intellectual property protection within their borders will directly incentivize innovation in their country. We believe implementing our technology is a big step in the right direction for protecting intellectual property and incentivizing innovation around the world.
New: Are you working on expanding the kinds of intellectual property TrademarkVision can analyze?
Mau: While we have focused on trademarks early on, our vision has always been to be the world leader in image recognition. Part of that vision includes revolutionizing brand protection for all types of visual intellectual property.
One of the best parts about working with government partners is that they can provide background on what areas of IP are pain points for industry, which enables us to develop new solutions. For instance, design patents are a different form of IP right that protects the form of a product, rather than a patent that protects the function of a product, ranging from 2D designs, graphical user interfaces, to 3D models. In working with our government partners, we have recently released a prototype of DesignsVision, our design patents image recognition search system.