Bad guys with bad intentions are messing with crucial high-tech systems that control national security, transportation and more.
Great news -- the good guys are on it.
Specifically, researchers from the Luddy School of Informatics, Computing and Engineering are at the forefront of this never-ending battle, and have the first-place honors to prove it.
The Luddy School won two categories in the recent $50,000 Trojan Detection Challenge. It was announced during December’s prestigious NeurlPS 2022 conference. The Neural Information Processing Systems is an international organization focused on machine learning, computational neuroscience, artificial intelligence and statistics.
Teams from all over the world participated in the Challenge, including the Qatar Computing Research Institute and a combination group from Purdue and Rutgers.
The victories were worth $8,000 to computer science postdoc Di Tang and computer science doctoral student Ziaho Wang. Advisors were XiaoFeng Wang, James H Rudy professor of computer science, and Haixu Tang, professor of informatics and computing.
Haixu Tang says Di Tang and Zihao Wang, “Put a lot of effort to implement an efficient and effective technology so we can be more competitive and, in some cases, become the best.”
Neural trojans attack and disrupt networks. Given the ever-increasing, world-wide use of artificial intelligence and machine models, this is a major problem.
“It’s always a battle between two armies,” XiaoFeng Wang says. “We want to capture the bad guy. The bad guy wants to avoid detection.”
The competition began in mid-August and ran through mid-October. There were 18 teams in the detection round and seven in the evasion round. The finals consisted of 13 teams.
Teams had to create a hard-to-detect trojan and then develop a system to find it.
“We were asked to find the most effective evading technique,” XiaoFeng Wang says, “and also the most effective technique for finding trojans inside machine-learning models.”
Mission accomplished.
“We had to come up with the nastiest bad machine-learning model,” XiaoFeng Wang says. “We demonstrated we could do that. Then we had to come up with a new detector to capture all the trojans. We did that.”
The result, XiaoFeng Wang adds, “Shows we have the capability. We had the best performer in the detection round, the best in the evasion round. In the final round, we had the best of the best.
“We competed against the best research groups. This demonstrates the technique we’ve developed is powerful. It works.”
Are they satisfied?
Not even close.
“I think we can do better,” Di Tang says with a laugh. “It sounds ambitious, but if we attend other (conferences) we can make more money and get better results.”
Adversaries hide trojans deep in neural networks of companies, universities and even governments. They can be inserted through infected data or components. The networks continue to behave normally, until an attack is triggered by a sign or pattern.
For example, XiaoFeng Wang says it could happen by using special glasses with a unique pattern through a facial recognition system.
Trojans can also create problems with self-driving systems. Normally vehicles would recognize speed limits and stop signs. A trojan could change that, causing a vehicle to drive through a stop sign or to speed.
“This research is very important because this country is using machine-learning systems in critical infrastructure,” XiaoFeng Wang says.
The Challenge success reflects ongoing research in a project titled, Statistical Methods for Backdoor Detection. The four-year, $1.76 million grant (Haixu Tang and XiaoFeng Wang are the primary investigators) is part of Intelligence Advanced Research Projects Activity’s Trojan AI research.
“We aim to develop statistical and computational methods for detecting backdoor in deep neural network models,” Haixu Tang says.
The Challenge fit that research.
“We are in this field working on technology to detect trojans and develop backdoor attacks to evade current detection techniques,” Haixu Tang says. “We were prepared for this challenge because we are doing this research. We are developing insightful novel ideas to get better performance.”
The Challenge success, XiaoFeng Wang says, showcased three main priorities:
First, the importance of their research.
Second, devising exceptional AI security techniques that will have lasting impact.
Third, demonstrate the value and impact of Luddy’s AI education in the real world.
Winning was important, Haixu Tang says, but was not the end goal.
“There are still many challenges. Even though we achieved 100 percent accuracy, it doesn’t mean we completely solved the problem. There are new research directions we can pursue.”
They will.
“We will continue to develop our technique,” Haixu Tang says. “Hopefully we can have even better solutions to tackle these problems.”