[D] I just realised: GPT-4 with image input can interpret any computer screen, any userinterface and any combination of them.

Submitted by Balance- t3_120guce in MachineLearning

GPT-4 is a multimodal model, which specifically accepts image and text inputs, and emits text outputs. And I just realised: You can layer this over any application, or even combinations of them. You can make a screenshot tool in which you can ask question.

This makes literally any current software with an GUI machine-interpretable. A multimodal language model could look at the exact same interface that you are. And thus you don't need advanced integrations anymore.

Of course, a custom integration will almost always be better, since you have better acces to underlying data and commands, but the fact that it can immediately work on any program will be just insane.

Just a thought I wanted to share, curious what everybody thinks.

432

Comments

You must log in or register to comment.

dlrace t1_jdh8ra8 wrote

The new plugins can be/ are created by just documenting the api and feeding it to gpt4 aren't they? no actual coding . So it seems at least plausible that the other approach would be as you say, let it interpret the ui visually.

16

BinarySplit t1_jdh9zu6 wrote

GPT-4 is potentially missing a vital feature to take this one step further: Visual Grounding - the ability to say where inside an image a specific element is, e.g. if the model wants to click a button, what X,Y position on the screen does that translate to?

Other MLLMs have it though, e.g. One-For-All. I guess it's only a matter of time before we can get MLLMs to provide a layer of automation over desktop applications...

204

banmeyoucoward t1_jdhg7kt wrote

I'd bet that screen recordings + mouse clicks + keyboard inputs made their way into the training data too.

0

Deep-Station-1746 t1_jdhhbbg wrote

Nope. Ability to input something doesn't mean being able to use it reliably. For example, take this post - your eyes have an ability to input all the info on the screen, but as a contribution, this post is pretty worthless. And, you are a lot smarter than GPT-4, I think.

Edit: spelling

−19

SkinnyJoshPeck t1_jdhis65 wrote

Replying to BinarySplit (#2,333,436)

i imagine you could interpolate, given access to more info about the image post-GPT analysis. i.e. i’d like to think it has some boundary defined for the objects it identifies in the image as part of metadata or something in the API.

9

harharveryfunny t1_jdhkn99 wrote

> GPT-4 with image input can interpret any computer screen

Not necessarily - it depends how they've implemented it. If it's just dense object and text detection, then that's all you're going to get.

For the model to be able to actually "see" the image they would need to feed it into the model at the level of neural net representation, not post-detection object description.

For example, if you wanted the model to guage whether two photos of someone not in it's training set are the same person, then it'd need face embeddings to do that (to gauge distance). They could special case all sorts of cases like this in addition to object detection, but you could always find something they missed.

The back-of-a-napkin hand-drawn website sketch demo is promising, but could have been done via object detection.

In the announcement of GPT-4, OpenAI said they're working with another company on the image/vision tech, and gave a link to an assistive vision company... for that type of use maybe dense labelling is enough.

30

dankaiv t1_jdhn2at wrote

... and computer interfaces (i.e. GUIs) have extremely low noise to signal ratio compared to image data from the real world. I believe soon AI will be better at using computers than most humans.

63

loopuleasa t1_jdhrwkv wrote

GPT4 is not publicly multimodal though

9

farmingvillein t1_jdhua51 wrote

Replying to loopuleasa (#2,335,582)

Hmm, what do you mean by "publicly"? OpenAI has publicly stated that GPT-4 is multi-modal, and that they simply haven't exposed the image API yet.

The image API isn't publicly available yet, but it is clearly coming.

9

ThirdMover t1_jdhvx8i wrote

Replying to BinarySplit (#2,333,436)

>GPT-4 is potentially missing a vital feature to take this one step further: Visual Grounding - the ability to say where inside an image a specific element is, e.g. if the model wants to click a button, what X,Y position on the screen does that translate to?

You could just ask it to move a cursor around until it's on the specified element. I'd be shocked if GPT-4 couldn't do that.

45

mycall t1_jdi3cko wrote

Can it detect object in the photo? Maybe drive an RC car with it? :)

1

TikiTDO t1_jdi8ims wrote

Replying to harharveryfunny (#2,334,775)

The embeddings are still just a representation of information. They are extremely dense, effectively continuous representations, true, but in theory you could represent that information using other formats. It would just take far more space and require more processing.

Obviously having the visual system provide data that the model can use directly is going to be far more effective, but nothing about dense object detection and description is going to be fundamentally incompatible with any level of detail you could extract into an embedding vectror. I'm not saying it would be a smart or effective solution, but it could be done.

In fact, going to another level, LLMs aren't restricted to working with just words. You could train an LLM to receive a serialized embedding as text input, and then train it to interpret those. After all, it's effectively just a list of numbers. I'm not sure why you'd do that if you could just feed it in directly, but maybe it's more convenient to not have to train in on different types of inputs or something.

3

ginger_beer_m t1_jdi9e5j wrote

Carry this to the conclusion. Maybe not GPT4, but future LLM could interpret what's on the screen and drive the interaction with the computer themselves. This would potentially displace millions of human out of job as they get automated by the model.

50

harharveryfunny t1_jdic1s3 wrote

Replying to TikiTDO (#2,337,473)

>Obviously having the visual system provide data that the model can use directly is going to be far more effective, but nothing about dense object detection and description is going to be fundamentally incompatible with any level of detail you could extract into an embedding vectror. I'm not saying it would be a smart or effective solution, but it could be done.

I can't see how that could work for something like my face example. You could individually detect facial features, subclassified into hundreds of different eye/mouth/hair/etc/etc variants, and still fail to capture the subtle differences that differentiate one individual from another.

4

BullockHouse t1_jdidje0 wrote

I'm curious if it can be instructed to play minecraft in a keyboard only mode simply by connecting a sequence of images to key stroke outputs.

3

nixed9 t1_jdifhni wrote

Replying to ginger_beer_m (#2,337,560)

This is quite literally what we hope for/deeply fear at /r/singularity. It's going to be able to interact with computer systems itself. Give it read/write memory access and access to it's own API, or the ability to just simply visually process the screen output... and then.... what?

Several years ago, as recently as 2017 or so, this seemed extremely far-fetched and the "estimation" of a technological singularity of 2045 seemed wildly optimistic.

Right now it seems like it's more like than not to happen by 2030.

46

MjrK t1_jdiflsw wrote

Replying to ThirdMover (#2,336,051)

I'm confident that someone can fine-tune an end-to-end vision-tranformer that can extract user interface elements from photos and enumerate interaction options.

Seems like such an obviously-useful tool and Vit-22B should be able to handle it, or many other Computer Vision tools on Hugging Face... I would've assumed some grad student somewhere is already hacking away at that.

But then also, compute costs are a b**** but generating training data set should be somewhat easy.

Free research paper idea, I guess.

20

TikiTDO t1_jdiirji wrote

Replying to harharveryfunny (#2,337,846)

For a computer words are just bits of information. If you wanted a system that used text to communicate this info, it would just assign some values to particular words, and you'd probably end up with ultra long strings of descriptions relating things to each other using god knows what terminology. It probably wouldn't really make sense to you if you were reading it because it would just be a text-encoded representation of an embedding vector describing finer relations that would only make sense to AIs.

5

yashdes t1_jdij1tl wrote

Replying to loopuleasa (#2,335,890)

these models are very sparse, meaning very few of the actual calculations actually effect the output. My guess is trimming the model is how they got gpt3.5-turbo and I wouldn't be surprised if gpt4-turbo is coming.

0

wind_dude t1_jdikwc4 wrote

I'm also curious about this, I reached out for developer access to try and test this on web screenshots for information extraction.

1

itsnotlupus t1_jdj2xpr wrote

Meh. We see a few demos and all of the demos work all of the time, but that could easily be an optical illusion.

Yes, GPT-4 is probably hooked to subsystems that can parse an image, be it some revision of CLIP or whatever else, and yes it's going to work well enough some of the time, maybe even most of the time.

But maybe wait until actual non-corpo people have their hands on it and can assess how well it actually works, how often it fails, and whether anyone can actually trust it to do those things consistently.

1

harharveryfunny t1_jdj5mom wrote

Replying to TikiTDO (#2,338,558)

>it would just be a text-encoded representation of an embedding vector

One you've decided to input image embeddings into the model, you may as well enter them directly, not converted into text.

In any case, embeddings, whether represented as text or not, are not the same as object recognition labels.

3

TikiTDO t1_jdj6dum wrote

Replying to harharveryfunny (#2,341,067)

I'm not saying it's a good solution, I'm just saying if you want to hack it together for whatever reason, I see no reason why it couldn't work. It's sort of like the idea of building a computer using the game of life. It's probably not something you'd want to run your code on... But you could.

2

MysteryInc152 t1_jdj8x5e wrote

Replying to loopuleasa (#2,335,890)

>they mentioned an image takes 30 seconds to "comprehend" by the model...

wait really ? Cn you link source or something. There's no reason a native implementation should take that long.

Now i'm wondering if they're just doing something like this -https://github.com/microsoft/MM-REACT

3

harharveryfunny t1_jdj9if0 wrote

Replying to TikiTDO (#2,341,153)

I'm not sure what your point is.

I started by pointing out that there are some use cases (giving face comparison as an example) where you need access to the neural representation of the image (e.g. embeddings), not just object recognition labels.

You seem to want to argue and say that text labels are all you need, but now you've come full circle back to agree with me and say that the model needs that neural representation (embeddings)!

As I said, embeddings are not the same as object labels. An embedding is a point in n-dimensional space. A label is an object name like "cat" or "nose". Encoding an embedding as text (simple enough - just a vector of numbers) doesn't turn it into an object label.

5

farmingvillein t1_jdj9w98 wrote

Replying to yashdes (#2,338,597)

> these models are very sparse

Hmm, do you have any sources for this assertion?

It isn't entirely unreasonable, but 1) GPU speed-ups for sparsity aren't that high (unless OpenAI is doing something crazy secret/special...possible?), so this isn't actually that big of an upswing (unless we're including MoE?) and 2) openai hasn't released architecture details (beyond the original gpt3 paper--which did not indicate that the model was "very" sparse).

1

Spziokles t1_jdjc4wn wrote

So when playing League of Legends, it could tell you which enemy champion disappeared from their lane, and in how many seconds you should retreat to stay safe?

Curious how this will impact E-Sports and wether it will be treated like doping in some form.

2

TikiTDO t1_jdjibnv wrote

Replying to harharveryfunny (#2,341,464)

My point was that you could pass all the information contained in an embedding as a text prompts into a model, rather than using it directly as an input vector, and an LLM could probably figure out how to use it even if the way you chose to deliver those embeddings was doing a numpy.savetxt and then sending the resulting string is as a prompt. I also pointed out that you could if your really wanted to write a network to convert an embedding to some sort of semantically meaningful word soup that stores the same amount of information. It's basically a pointless bit of trivia which illustrates a fun idea.

I'm not particularly interested in arguing whatever you think I want to argue. I made a pedantic aside that technically you can represent the same information in different formats, including representing embedding as text, and that a transformer based architecture would be able to find patterns it it all the same. I don't see anything to argue here, it's just a "you could also do it this way, isn't that neat." It's sort of the nature of a public forum; you made a post that made me think something, so I hit reply and wrote down my thoughts, nothing more.

2

H0lzm1ch3l t1_jdjntxm wrote

Yes but why let the AI use a GUI when we can just give it an API …

3

simmol t1_jdjq815 wrote

I think for this to be truly effective, the LLM would need to take in huge amounts of computer screen images in its training set, and I am not sure if that was done for the pre-trained model for GPT-4. But once this is done for all possible computer screen image combinations that one can think of, then it would probably be akin to the self-driving car type of algorithm where you can navigate accordingly based on the images.

But this type of multi-modality would be useful if you have the person actually sitting in front of the computer working side-by-side with the AI, right? Because if you want to eliminate the human from the loop, then I am not sure if this is an efficient way of training the LLM since these type of computer screen images are what helps a human navigate the computer, and not necessarily optimal for the LLM.

1

Puzzleheaded_Acadia1 t1_jdjx3oq wrote

I have questions can I fine-tune the gpt-neo-x 125m parameters on chat dataset to give me a decent answer like human because when I run it give me random characters

1

plocco-tocco t1_jdjx7qz wrote

Replying to ThirdMover (#2,342,042)

The complexity of the input wouldn't change in this case since it's just a screen grab of the display. Just that you'd need to do inference at a certain frame rate to be able to detect the cursor, which isn't that cheap with GPT-4. Now, I'm not sure what the latency or cost would be, I'd need to get access to the API to answer it.

1

DisasterEquivalent t1_jdk10wf wrote

Replying to BinarySplit (#2,333,436)

I mean, most apps have accessibility tags for all objects you can interact with (it is standard in UIKit) - The accessibility tags have hooks in them you can use for automation. so you should be able just have it find the correct element there without much searching.

2

simmol t1_jdkd4pf wrote

Replying to MyPetGoat (#2,345,494)

Seems quite inefficient though. Can't GPT just access the HTML or other type of codes associated with the website and just access the websites via the text as opposed to image?

1

rePAN6517 t1_jdkinrg wrote

Replying to nixed9 (#2,338,219)

> This is quite literally what we hope for/deeply fear at /r/singularity

That sub is a cesspool of unthinking starry-eyed singularity fanbois that worship it like a religion.

12

MassiveIndependence8 t1_jdl9oq9 wrote

Replying to ThirdMover (#2,336,051)

You’re actually suggesting putting every single frame into gpt-4? It’ll cost you a fortune after 5 seconds of running it. Plus the latency is super high, it might takes you an hour to process a “5 seconds” worth of images.

1

skaag t1_jdli80o wrote

I have not seen a way in the GPT-4 UI by OpenAI to submit an image? How do you do it?

1

fiftyfourseventeen t1_jdlm1n7 wrote

Replying to rePAN6517 (#2,346,513)

Lmao it seems everyone used chatGPT for a grand total of 20 minutes and threw their hands up saying "this is the end!". I have always wondered how the public would react once this tech finally became good enough for the public to notice, can't say this was too far from what I envisioned. "What if it's conscious and we don't even know it!" Cmon give me a break

0

thePaddyMK t1_jdlqyng wrote

Replying to dankaiv (#2,335,052)

I think so, too. IMO this will open new ways for software development. There has already been work looking towards RL to find bugs in games. Like climbing walls that you should not. With a multimodal model there might be interesting new ways to debug and develop UIs.

6

SeymourBits t1_jdlwrgi wrote

Replying to itsnotlupus (#2,340,814)

This is the most accurate comment I've come across. The entire system is only as good and granular as the CLIP text description that's passed into GPT-4 which then has to "imagine" the described image, often with varying degrees of hallucinations. I've used it and can confirm it is currently not possible to operate anything close to a GUI with the current approach.

1

MjrK t1_jdm4ola wrote

Replying to modcowboy (#2,348,085)

For many (perhaps these days, most) use cases, absolutely! The advantage of vision in some others might be interacting more directly with the browser itself, as well as other applications, and multi-tasking... perhaps similar to the way we use PCs and mobile devices to accomplish more complex tasks

2

alexmin93 t1_jdmocbw wrote

The problem is that LLMs aren't capable to make decisions. While GPT-4 can chat almost like a sentient being, it's not sentient at all. It's not able to coprehend the limitations of it's knowledge and capabilities. It's extremely hard to make it call an API to ask for more context. There's no way it will be good at using a computer like a user. It can predict what wappens if you do something but it won't be able to take some action. It's a dataset limitation mostly, it's relatively easy to train language models as there's almost infinite ammount of text on the Internet. But are there any condition-action kind of datasets? You'd need to observe human behavior for millenias (or install some tracker software on thousands of workstations and observe users behavior for years)

2

fiftyfourseventeen t1_jdnqlqc wrote

Replying to nixed9 (#2,357,576)

That's really cool, but I mean, it's published by Microsoft which is working with openAI, and it's a commerical closed source product. It's in their best interest to brag about it's capabilities as much as possible.

There are maybe sparks of AGI, but there are a lot of problems that are going to be very difficult to solve that people have been trying to solve for decades.

0

Suspicious-Box- t1_jdzj7wr wrote

Replying to BinarySplit (#2,333,436)

Just need training for that. Its amazing but what could it do with camera vision into the world and a robot body. Would it need specific training or could it brute force its way to moving a limb. The model would need to be able to improve itself real time though.

0

emissaryo t1_je4jvzt wrote

Now I'm even more concerned about privacy. Governments will use it for surveillance and the more modalities we add, the more surveillance there will be.

1