Getting (and keeping) NLP models safely in production

Introducing a natural language processing (NLP) model into a production environment is a lot like buying a car. In either case, you may set the parameters to suit your desired results, test some approaches, and retest them. And as soon as you drive away the lot, the value begins to plummet. Just like having a car, having an NLP or AI-enabled product has many advantages, but it doesn’t stop maintenance. Do not stop, at least for it to function properly over time.

AI production is difficult enough, but ensuring model accuracy in a real environment can present even greater governance challenges. The accuracy of the model drops the moment the model hits the market. This is because the predictable research environment in which the model is trained behaves differently in real life. As the highway is a different scenario than the dealer’s parcel.

This is called concept drift. In other words, as variables change, the concepts learned can become inaccurate. It’s nothing new in the areas of AI and machine learning (ML), but it continues to challenge users. This is also the reason why only about 13% of data science projects actually move to production, despite huge investments in AI and NLP in recent years ()VentureBeat).

So what does it take to safely move a product from research to production? It’s definitely just as important, but what do you need to keep producing accurately as the tide changes? There are some considerations that companies should keep in mind in order for their AI investment to actually see the light of day.

Introduce AI model to production environment

Model governance is an important factor in producing NLP initiatives, and is a common reason why many products remain projects. Model governance targets how a company tracks model activity, access, and behavior in a particular production environment. Monitoring this is important to mitigate risk, troubleshoot, and maintain compliance. This concept is well understood by the AI ​​global community, but it’s also a thorn on their side.

Data from 2021 NLP Industry Survey Respondents have shown that precision tools that are easy to adjust and customize are a top priority. Technology leaders repeated this, noting that accuracy, followed by production readiness, and scalability are essential when evaluating NLP solutions. Constant adjustment is the key to a model functioning correctly over time, but it is also the biggest challenge faced by practitioners.

The NLP project contains a pipeline that uses the results of previous tasks and pre-trained models downstream. Models often need to be tailored and customized for a particular domain or application. For example, health care models trained in academic papers and medical journals do not work the same when used by media companies to identify fake news.

Improving searchability and collaboration between AI communities plays an important role in standardizing model governance practices. This includes storing modeling assets such as notebooks, datasets, resulting measurements, hyperparameters, and other metadata in searchable catalogs. Allowing experiment reproducibility and sharing among members of the data science team is another area of ​​benefit for those seeking to move a project into production.

More tactically, rigorous testing and retesting is the best way to ensure that your model behaves like a study in a production environment. Two very different environments. All practitioners should exercise model versioning that goes beyond experimentation to release candidates, testing their accuracy, bias, and stability, and validating pre-release models in new regions and populations. The element.

When you release software, you need to incorporate security and compliance into your strategy from the beginning. AI projects are no exception. Role-based access control and an approval workflow for storing and delivering all the metadata needed for a model’s release and full audit trail are the security required for the model to be considered production-ready. It is a part of the measures.

These practices can greatly increase the likelihood that an AI project will move from concept to production. More importantly, they help lay the foundation for practices that need to be applied once the product is customer-ready.

Maintain production of AI models

Returning to the car analogy: In a production environment, there is no clear “check engine” light for AI, so the data team must constantly monitor the model. Unlike traditional software projects, it’s important to keep data scientists and engineers in the project even after the model is deployed.

From an operational perspective, this requires more resources in terms of both human capital and cost. This may be the reason why many organizations cannot do this. The pressure to keep up with the pace of the business and move on to the “next thing” is also taken into account, but perhaps the biggest oversight is that even IT leaders don’t expect model degradation to be a problem. That is.

For example, in medical care, the model analyzes electronic medical records (EMRs) and a patient may develop an emergency C section based on risk factors such as obesity, smoking, drug use, and other health determinants. Can be predicted. If patients are dubbed high risk, their practitioners may ask them to come earlier or more often to reduce pregnancy complications.

These risk factors remain constant over time, and as with many of them, patients are expected to be difficult to predict. Did they quit smoking? Have they been diagnosed with gestational diabetes? There are also subtle differences in how clinicians ask questions and record their answers in hospital records, which can have different consequences.

This can be even more difficult given the NLP tools most practitioners use. The majority of respondents in the previous survey (83%) said they used at least one of the following NLP cloud services: AWS Comprehend, Azure Text Analytics, Google Cloud Natural Language AI, or IBM Watson NLU. While the popularity and accessibility of cloud services is clear, technology leaders cite the difficulty and cost of adjusting models as a major challenge. Basically, even professionals are committed to maintaining the accuracy of the model in production.

Another problem is that it takes time to see if something is wrong. The period can vary significantly. Amazon is updating its fraud detection algorithm, which may inadvertently block customers in the process. Within hours, and possibly minutes, a customer service email will point out the problem. In health care, it can take several months to get enough data under certain conditions to confirm that the model has deteriorated.

Basically, to keep your model accurate, you need to apply the same rigorous testing, retraining pipeline automation, and measurements that were performed before the model was deployed. When working with AI and ML models in a production environment, it’s better to anticipate problems than to expect optimal performance in the coming months.

Considering all the work required to move a model to production and keep it safe, it’s understandable why 87% of data projects aren’t on the market. Nonetheless, 93% of tech leaders have shown that their NLP budget has increased by 10-30% compared to last year (Radiant flow). It’s encouraging to see more investment in NLP technology, but it’s all in vain unless the company knows the expertise, time, and continuous updates needed to deploy a successful NLP project. ..