AP006: Can your AI/ML Stack do this? Predict, Monitor and Retrieve....

Fred and Melody

27 Nov 2024

welcome to continuous TV where we

transform Life Sciences with Dialogue on

AI and GXP today's episode is focused on

L large action model in GXP your host is

nagesh Nama our CEO today's guests are

sham patadia and priyan

Sinha welcome to continuous TV uh this

is our first episode in the series our

focus is on AI and ml in GXP that's our

focus and and today we'll be talking

about lamb large action models that's

and its implications or its impact on

the life sciences GXP environments my

guests are sham patadia who is a senior

AIML uh lead at xlm and prian Sina who

is also an AIML engineer at

xlm uh hello sham hello prian how are

you hello hello

I'm good sir um sham and prian you've

been working on AI ML and also um the

lamb models for some time now and it's

part of our continuous Labs

correct

yes um so just for the audience sham can

you briefly describe what exactly is

large action model and how is it

different from our uh uh uh our typical

AI llms you know how is it different

from llms can you please uh uh describe

that for our audience um yes sure uh so

large action model as you know as we see

its word itself it is action model right

so it is basically going to perform the

action that we are going to tell so what

large uh how it differs from large

language model is basically language

models are capable of mostly all the

task which are related to te text

generation which are related to natural

language processing now what large

action model is basically it is

leveraging the uh Power of large

language models and Performing the

actions uh as well so what it's

basically it's using large language

models in order to understand the prompt

that is given by the user and it will uh

also perform that action so

ites is it what Opa is talking about now

they call it the the agent know that

they're going to be deploying an agent

framework uh is it something

similar uh uh yes I mean uh it's on the

similar lines but large action models I

think uh was first coined by uh the

rabbit team so they developed uh they

developed the first commercial tool

based on top of lamp so they invented

the words uh the large action models so

uh open AI is definitely I mean coming

into the race but they were the first

ones to you know uh coin the word large

action models which they have developed

good thing you mentioned about rabbit

when I watched rabbit R1 uh at the CES

in January I was amazed um and because

they have the hardware right uh with all

the different devices built in and also

they have the software working um nicely

uh and it very fast and it looks like it

understood what the the user wanted can

you just describe how rabbit is designed

I know and how how did it really work

what were the different uh layers within

the rabbit

stack yes so uh a large action model

comprises it is built on top of large

language models plus it uses

uh one of the concepts in a which is

reinforcement learning so they combined

those two things and created uh created

an you know environment or basically a

model uh which uses both the things so

you know once a user is perform so that

large action models requires a user to

you know perform the workflow once with

a certain application let's say you know

uh a user is one time going through the

workflow of booking an Uber right

right now next time at the at the large

action model will be looking at its

action and it will be keeping in uh you

know learning from those actions via

using reinforcement learning and then

once you know next time a user asks the

action model you know to book the ride

it uses that data and combines it with

uh its own intelligence which is

generated via large language models for

different scenarios so both it can learn

on it is that true over time it can

learn on its own yep yep so that's the

whole idea so uh once a user describes a

single flow of the application it will

know that you know it will capture each

and every essence of that application

and it next time it will use the power

and reasoning of large language models

in order to perform different actions on

top of it so it's almost like a human

right it can think it can actually

perform the action leveraging llms on

own learning and do it so is it

something similar to what Nvidia did

remember the one where the they designed

it could actually play the game on its

own uh and it went on for hours you know

is it something similar correct correct

something similar so reinforcement

learning is the concept behind it so uh

Nvidia open a both of the both of them

have done it uh previously so you know

NV Nvidia people and the people from

open AI they both have have been

champions in the game uh I think for

open AI I can share surely that you know

they have defeated the world's champions

uh of the game for d 2 and their agent

their agent uh you know defeated the uh

top people in the I mean who used to

play datu like for a long time and they

that agent defeated them uh I mean um uh

but this is a very nent uh this is at a

very nent stage right it's kind of

developing now right it's not as mature

as the llm um world correct correct

large action models are developing I

mean u a lot of you know people from

open AI itself they are seeing that you

know this year is going to last year was

about large language models this year

it's going to be about multimodel models

which can you know interact with images

and it can also interact with the

language so large action models falls

into that uh please so that's a next

sequence right once we can have an

interaction via text with Ani model now

the next is how can an agent leverage

that and actually perform actions

correct yep very nice you know so what

are some of the major projects in uh

lamb can you just open source or

otherwise can you just we discussed a

couple of them can you just quickly uh

let us know what are the major projects

currently underway correct so uh

one of the top one that comes to my mind

is open interpreter so that is exactly

not lamb but it is leveraging a lot of

Concepts uh from lamb so it is using

multimodel models for uh and also

language models for performing the

actions that we tell it to uh perform so

that is exactly because the

reinforcement learning part is missing

uh from the open interpreter but that is

you know very close to lamb

uh and also uh sorry before before we

move ahead open interpreter can actually

interact with the browser as well as a

computer applications like desktop

applications basically take over your

computer uh anything you know a human

can do uh on recently I saw one thing

called what is that U uh light or

something like that one light one or

something you know remember where yeah

which is released by open interpreter

they are lightweight device yeah right

right so that's very similar right yes

yes give voice commands and it actually

can take over your computer check your

emails respond all that is happening

correct correct yes yes uh so basically

open source uh alternative of rabbit

kind of correct correct correct or Devon

maybe right yeah yes yeah go ahead and

uh before we move on just list what

other major initia so recently uh

recently you know recently

after open interpreter there came one of

the projects which is you know in

current development which is in very

early stage which is La uh we are also

exploring that so that is also U that is

also on the lines of lamb but it is

using uh large language models per se

but it is you know uh able to perform

the actions from the large language

models so it is not combining the

reinforcement learning part as of now

but uh going ahead you know U they might

plan to integrate that as well with this

it's open source correct yeah that is

open source yes and um so is it similar

to open interpreter

then uh it that is laig is specifically

designed for web uh automation tasks

okay so that takes control over your

browser and open interpreter basically

takes control over your computer got it

got it got it got it got it got it uh

any other project that you like to

mention uh a lot of them are there but

at top of my mind there are only two

coming which open interpret and I know

you have mentioned in offline

discussions of like for example zero

step you know uh working with right yes

all of those are uh are basically tools

uh which can you know help us uh I mean

those are very useful AI Tools in terms

of you know automating the web

automation tasks so zero step is also

basically leveraging the power of llms

in order to automate you know the some

of the clicks some of the uh tasks on

the let's say you want to fill a form uh

we can just write that you know fill a

form on this page it will do everything

on its own it like a natural language in

English you can write it basically yes

so there's a function Ai and you put

that you know uh fill the form for me on

the given page uh then it will fill the

whole form for you so that is you know a

tool for test automation Developers for

for them you know not to find the

experts not to find the selectors and

all of that so basically they just need

to give the natural language and it will

do everything on its own so it can

actually read the HTML and interpret and

figure out the elements and do it right

so it's very different from the way we

use selenium today correct where we have

to tell selenium the exact X paath and

then it can perform some action so

that's kind of the Legacy coding right

uh the way leg yeah yes yes okay so

coming back to our continuous validation

uh how are we how is our uh Foundation

or framework how does it work today

today not the lamb or AI based tools the

current foundation so our audience can

understand how is our Legacy framework

or current framework and then we'll talk

about how will it look maybe a six

months or year down the road so can you

just let our audience know how how our

stack works today the key uh component

of our stack correct so the key

components of our stack is you know we

are using Behavior driven development uh

model for our test automation framework

so here you know we will have a uh uh

feature file per se that you know that

is being designed as a test case and

then we integrate that with you know

selenium in order to automate that whole

feature file uh and we have our internal

reporting tool which will generate the

you know out the generate the output

which is captured during the uh test run

of that feature file and we have a

specific uh you know uh design back end

which is you know xlm core which will

generate and capture all the relevant

details whenever the test case is

running uh so you know so that we can uh

create a perfect output uh for it we can

generate a basically report of the test

which has been executed so in other

words you know um typically uh a quality

engineer will understand the what needs

to be tested right for a software

application once the workflow is

understood they will put together a a

feature file which is based on bdd and

that kind of Quasi English right the

feature file and that's it and the

developer takes over understands what

each step in the feature file has to do

and then generates the c or in our case

it's C and selenium code right and then

you know obviously there are other

modules which capture the evidence and

the PDF is generated is is that correct

yep yep uh that is accurate So based on

the work we are doing at xlm continuous

validation how how will this change uh

let's say 6 months 3 months 6 months or

2024 so uh as as I mentioned about you

know as we talked about the Lambs right

how they operate is basically based on

natural language so go going ahead we

also want to do the same we don't want

to you know write the code uh and you

know write the whole code for the whole

feature file we basically want to

automate each and everything so you know

we will be automating you know the

creation of feature file we'll be

automating the you know writing the code

of the for the same feature file and the

execution part as well so end to end you

know basically and L which is focused on

uh GXP validation got it so in other

words will take uh the the various uh

modules within L stack you know and you

know so that way it can completely

automate perform end automation based on

like a natural language prompt right

like a test description or something

like that you know that you can create

the feature files figure it out you know

then generate the selenium code that

goes with it you know then execute the

test and give us a report right so it's

a totally different uh way of doing

testing correct so let's say you just

have your requirements defined you can

just put in the requirements and you

know you will have the executed test

cases basically you just gave in the

requirements uh feature file is created

automatically it is tested automatically

and you have the output PDF of it and

Sham I can't wait to see that um yeah

sham you know before I move on to prian

you know can you just show our audience

you know maybe a one or a small demo as

to how a lamb really

works uh yes so for that you know I'm

going to demonstrate a tool which is LA

just to get give you an idea of you know

because it will show you you know what

is going on in the back end as well okay

okay so let's go uh I'm going to share

my screen

now

uh yep my screen is visible

yep so uh this is the you know the

interface uh of laig what we you can see

here so now I'll be typing uh a

prompt uh and it will

execute so now it is processing that

prompt it will uh basically you know

right now what you see here this is the

browser right so this is the actual

browser window where you know the whole

code is uh present right so now as you

can see we have logged into here right

so it says uh failure open the debug tab

but we have been able to log into to

that because in the previous window

window you saw that you know

the credential login page was there now

we have already logged in here right now

I'll just say you know click on xlm qms

clone project button and if you can see

here in the recent projects right there

is a project that we want to navigate to

so I'll just type in this particular

prompt and it will uh do its own action

there basically so this is basically in

English right what you're saying is in

click on the correct this is you know

very very simple language click on XM

qms clone project button that's it

that's all I'm giving so what in the

back end will happen is it will uh it

will take the whole HTML uh it will take

my prompt and it will uh you know

automatically identify the element which

is present uh in the HTML source and it

will uh execute that particular it will

it will generate the selenium code in

order to click on that element we can

see that right as it's doing it we can

see the code come up here correct so if

you can see here so till now whatever we

have processed the code right in order

to log in and all of that all of that is

written here so you can you can get the

whole history of it basically it is

generating the code for you yeah end to

end right

so in going future we we even don't have

to give the instructions like this so

again this thing the whole thing will be

automated end to end yeah it can even

actually navigate some in in 100% and

reinforc learning it can even

investigate a software on its own right

so it can learn the buttons learn the

architecture like a human does you know

and uh then you don't have to teach it

everything correct correct yes so that

is a part where reinforcement learning

comes into play where you know we give

uh we given a URL we given the login

credentials to the application and it

will explore the whole application on

its own so if you see right now it

generated the code itself so if you see

it created the xath and it clicked on it

right now what I'll just say is wait for

one second in order to uh get the page

loaded so I had not given the command

that you know uh wait until the page

gets loaded so now I'm giving it so it

will uh again you know generate the code

for it and again execute that particular

code in Python for us got it got it so

uh right now it is processing it is

taking a little bit of time because I'm

running the whole large language model

on my local machine on my GPU uh so that

that is why a Computing is taking a

little bit of time that is the only

reason and uh you can basically see that

you know whatever instruction we can

give it can automate it from you know

from any of the instruction that we give

for any web automation task that you

want and this can be given in a sequence

of commands also right that way it can

execute uh based on the English input

correct yes

so right now we are still waiting for it

to execute it yes we are still waiting

for it to execute just because you know

uh if you see uh the whole GPU memory is

stocked 92% is used so when you load a

particular model on a local machine it

takes a lot of time in order to execute

this it did it right yeah it did so if

you see the first two lines import time

and time do sleep 1 second so it

generated

yeah yeah got it correct and it is also

returning us the HTML so if you see the

this particular window is open right now

correct can we just show one command

then I'll move on to prian yep

definitely so now we are going to click

the here if you see there is a create

button on top in order to create an

issue yeah so right now I have given the

command click on the create button on

the top and wait for two seconds so I've

combined the waiting time here so it

will WR again the code for creating uh

first it will uh get the HTML extract

the element U then it will again give

the that particular element to the llm

and say that you know generate an

equivalent selenium code for it and then

once the code is generated it will get

executed here and also you know you can

see whatever is executed as of now got

it so you if you see uh we clicked on

xlm qms button right that is again

mentioned here so now what you can do is

you can copy the whole thing and your

test case is ready right right right

right so that's amazing what is doing uh

correct so are we going to issue this

command the new command correct so we

already are processing that particular

command so it is uh uh navigating that

particular thing uh sorry extracting

that thing and now uh it will you know

once it finds the element it will uh

basically update so it is a little bit

of time uh once we you know train this

particular model for these kind of task

uh you know with a lot of data this time

should be reduced so if you can see and

also can they may need more compute

power also right yes so if you see here

it clicked on the create button right

correct

so uh wait for one second now if I give

this it was not able to wait for the uh

it was not able to wait for the two

seconds so that second command was he

was not able to execute it right so now

uh now I gave it another command so

basically it will again wait for one

second and then again so but the screen

came up though right the create uh

screen did come up so if you see yeah it

clicked on the create issue button right

right what and why this issue is

happening is because you know I'm

running it on my local machine so if I

want to change the back end uh if I if I

can change the back end uh of the the LM

right so I'm on my local machine I'm

using 7 billion model uh parameter model

so that is again generating you know a

little bit of here and there code it's

not proper code uh but it is able to

extract the things from HTML that is one

of the main things that we need to take

a look at because uh executions we can

modify it uh afterwards we can integrate

that into our back end but the it is

able to you know really easily perform

the application I mean the really easily

perform the extraction from the HTML for

whatever prom that we

give sounds good sounds good so I'm

looking at the clock also um so prian

you know sham you can stop sharing it

you know um prian you know um uh I know

you're working on a with Sham on a major

initiative that where we want to move

from our current framework to L or what

I call the V right validation automation

model so can you quick quickly tell the

audience about uh couple of areas that

you're focusing on right now I know

you're working on open interpreter as

well as integrating tools like zero step

AI uh can you just talk about it sure so

uh what sham mentioned the whole novelty

of L comes from including various blocks

blocks like computer vision blocks like

RL as sh mentioned uh similarly uh the

action part the obviously the engine

base engine is based on the llm models

and but the action part is being being

handled by those llm models which are

generating codes for in selenium or

playrite which actually interacts with

our browser or our desktop so the That

Base part is what I'm working on right

now which is uh based on playright and

there's a tool called zero step AI which

is also doing the exact same thing it

combines the extraction of HTML along

with the computer vision algorithms it

takes the screenshot the browser it

locates where the stuff are where the

elements are and gives uh Compares it

with with the HTML and then gives me

what to click and where to click based

on just my natural language uh process

can you show them show the audience can

you please share your screen and can you

show an example please

sure yeah we can see it yeah so here you

can see uh JB notebook where in I have

uh written the code on based on a

feature file to automate a uh asset

feature class uh based on our uh this is

based on play right and it includes some

functions of play Zero step AI the

function AI is what is calling the Zer

step AI so in the and can you just show

the full command and see how that zero

step AI

looks yes so uh the whole thing

is it's just a natural language thing uh

scroll down the page it's very like

natural language similarly here type

comment as ready to start the progress

nice just like how sham was doing uh in

LA right yeah the base is the same which

handles all in all all the LM can we run

this do you think yeah

sure

so right now we using playright and zero

step AI correct it's a combination of

both of them so the commands are mixed

with playright and Z Zer step AI so here

the scroll down was done using zero step

AI the clicking part of this link was

done using zero step AI got it got it uh

is speed an issue or is it working good

it works uh good with uh because spite

has this ability that it waits for the

uh page to load without us giving any

explicit command to wait for some time

wait for delay so it that was used to be

a problem for us in selenium right where

used to actually put thread sleep and

weight and we couldn't estimate that so

our testing times were longer than it

should be correct yes yes correct

comparatively uh little bit longer yeah

yeah long

yeah yeah so it's going to what it's

going to do what will the code do now so

yeah it tried to find the issue number

1731 it opened it and it was checking to

change the action but it seems that it

doesn't have the permission as of now uh

as I'm using a different

account no problem but the bottom line

is you know you just can automate a

workflow based using uh your player with

zero step AI which will simplifies

development correct yes true uh you can

use English prompts to generate uh uh

this one yeah very good and

similarly yeah there are to fill the

whole form entirely so that can that the

entire filling of the form can be done

using just one AI command in playright

or selum you can you you would have had

to write various uh text boxes fill in

the text boxes and the text that we want

to fill but in here in AI zero step AI

we can just write it in one single call

so I've written uh test summary in One

summary then it's description there's

other so you're you're filling multiple

Fields right with very simple uh line of

code which is actually not a line of

code it's like English right I mean

natural language

yes U very nice you know um yep uh sham

and priyan uh thank you very much you

know uh we'll have a follow-up session

on this maybe a month or two from now so

that you know we can share more of our

findings and how our development uh is

taking shape over time because

ultimately our goal is to develop vam uh

in 2024 where it'll be a totally

different foundational model compared to

our current foundational model which is

based on selenium C and bdd and that'll

be replaced with hopefully with a AI

based model and what we call Bam correct

yes yes n g the idea is to automate

everything from scratch and Sham thank

you uh prian thank you good luck thank

you sir thank you

thank you for watching continuous TV if

you have any further questions please

shoot an email at info@ continuous

validation.

About

This podcast is a lively discussion between Fred and Melody as they take a deep dive into the transformative potential of a modern AI stack in manufacturing and technology.

It focuses on three key technologies: Continuous Predictive Maintenance (cPdM), Continuous Temperature Monitoring (cTM), and ContinuousGPT.

cPdM uses AI and machine learning to predict and prevent equipment failures, reducing downtime and costs.
cTM automates temperature monitoring in regulated environments, ensuring compliance with stringent standards like FDA regulations.
ContinuousGPT enables conversational interactions with data, improving accessibility and decision-making by simplifying data retrieval and integration.

These technologies are presented as a departure from traditional practices, showcasing the benefits of embracing a modern AI stack for greater efficiency and proactive decision-making in various industries.

==**==

Click here for the related Newsletter: #032: Can your AI/ML Stack do this? Predict, Monitor and Retrieve.....

==**==

Please follow us on