It’s weird that you say they are failing. If you asked a human to highlight the face In that picture they would do the exact same thing!

Your application might need something different but don’t call this “failing.” It’s succeeding at what it was designed to do, which is find faces.

What is your application by the way?

Pytorch / Keras / Tensorflow for deep learning

And any basic ML library you want, scitkit leaen etc.

Deep learning is all about GPU usage and running long experiments in production. I’m confused what you even want

Is the question basically asking, what skills would someone specialized in DL have vs someone specializing in non-DL ML have?

I'm using with torch.no_grad(): when calculating the output of the teacher model.

How do you manage the GPU issue in building an api?

The question is about development and tech stack

How do we assess the GPU need in production?

Right so when it comes to computing if I am building a DL api for someone. How should I address the computing issue?

Quite different tech stack for APIs. DL requires some kind of model server with GPU. For traditional ML use Lambda or FastAPI on a server.

For batch processing it's more similar, depending on your data size, you might not need a GPU even for Deep learning.

Also deep learning is usually unstructured data, which requires different storage and training infrastructure.

You can read books about that topic however at the core that's the difference a that's why a lot of companies still don't utilize DL.

Will do!

Looking forward to seeing it when done.. Please do share the pictures

From the top of my head, DL requires much more data preprocessing and research. ML is more like - fit an XGBoost model, and if it doesn't work well, see why, fix that in data and try again. If XGBoost can't solve it, your data is bad or you need DL.

[deleted]

Showing you can create a smaller model with the same performance means SWIN is overparameterized for that given task. Give it datasets with varying complexity, not just one single one.

Google difference between ML and Deep Learning.

Haven't purchased the motherboard yet, but the case would be a fractal design torrent. In order to get 2 4090s to fit you would need to go custom liquid cooling to get rid of the massive heatsinks on the 4090.

Not that cringe to be honest.

That makes no sense. Are you sure you're not doing backprop on the teacher model? It should be a lot less resource intensive.

Furthermore, check how you're distilling the model, i.e. what layers and what weights. Generally, for transformer architectures, you distill the first, embedding layer, the attention and hidden layers, and the final, prediction layer. Distilling only the prediction layer works poorly.

Because they weren't trained against animated characters, try adding some image processing tools for converting animated to normal characters or normal and animated characters to some common filter

I remember reading it, I'll read it again and discuss it. Thanks.

I'm already using a pretrained model as the teacher model. But the distillation part itself has nearly the cost of training a model. I'm not insisting but I feel like I'm doing something wrong and needed some advices (note that I've only had theoritical experience in such areas of research, this is the first time I'm doing it practically).

Thanks for you comments.

The result you observe is intentional. The training objective of face detection models is usually to detect faces in any kind of pictures: drawings, 3d renders, photos.

A foto of a face isn't a face either. That's why Apple's Face ID uses a 3D scanner in addition.

This is a great insight. Thanks for the help.

What mobo and case do you have to put 2 4090s in?

Due to lack of details of cartoon faces, the model should not detect these cartoon faces as faces with a high confidence score (about 86.8 %). If this is the case then these models are as bad as haar cascade based face detectors.